Repository: brooklyn-docs Updated Branches: refs/heads/master a41e0d785 -> bf43bd004
improve location docs, including AWS VPC acct type Project: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/repo Commit: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/commit/9d4ab40e Tree: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/tree/9d4ab40e Diff: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/diff/9d4ab40e Branch: refs/heads/master Commit: 9d4ab40e3220af84468e8d7a0c33f103d15a5119 Parents: ee148da Author: Alex Heneveld <[email protected]> Authored: Tue Feb 23 10:46:37 2016 -0800 Committer: Alex Heneveld <[email protected]> Committed: Tue Feb 23 10:46:37 2016 -0800 ---------------------------------------------------------------------- guide/ops/locations/cloud-credentials.md | 83 +---------- guide/ops/locations/index.md | 14 +- guide/ops/locations/location-customizers.md | 2 +- guide/ops/locations/more-clouds.md | 171 +++++++++++++++++++++++ 4 files changed, 183 insertions(+), 87 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/9d4ab40e/guide/ops/locations/cloud-credentials.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/cloud-credentials.md b/guide/ops/locations/cloud-credentials.md index 40cc108..1594012 100644 --- a/guide/ops/locations/cloud-credentials.md +++ b/guide/ops/locations/cloud-credentials.md @@ -1,85 +1,6 @@ --- -title: Cloud Setup +title: More Clouds layout: website-normal --- -To connect to a Cloud, Brooklyn requires appropriate credentials. These comprise the "identity" and -"credential" in Brooklyn terminology. - -For private clouds (and for some clouds being targeted using a standard API), the "endpoint" -must also be specified, which is the cloud's URL. - -The [jclouds guides](https://jclouds.apache.org/guides) includes documentation on configuring -different clouds. - - -## AWS - -### Credentials - -AWS has an "access key" and a "secret key", which correspond to Brooklyn's identity and credential -respectively. - -These keys are the way for any programmatic mechanism to access the AWS API. - -To generate an access key and a secret key, see [jclouds instructions](http://jclouds.apache.org/guides/aws) -and [AWS IAM instructions](http://docs.aws.amazon.com/IAM/latest/UserGuide/ManagingCredentials.html). - -An example of the expected format is shown below: - - brooklyn.location.jclouds.aws-ec2.identity=ABCDEFGHIJKLMNOPQRST - brooklyn.location.jclouds.aws-ec2.credential=abcdefghijklmnopqrstu+vwxyzabcdefghijklm - - -### Tidying up after jclouds - -Security groups are not always deleted by jclouds. This is due to a limitation in AWS (see -https://issues.apache.org/jira/browse/JCLOUDS-207). In brief, AWS prevents the security group -being deleted until there are no VMs using it. However, there is eventual consistency for -recording which VMs still reference those security groups. After deleting the VM, it can sometimes -take several minutes before the security group can be deleted. jclouds retries for 3 seconds, but -does not block for longer. - -There is utility written by Cloudsoft for deleting these unused resources: -http://www.cloudsoftcorp.com/blog/2013/03/tidying-up-after-jclouds. - - -## Google Compute Engine - -### Credentials - -Google Compute Engine (GCE) uses a service account e-mail address for the identity, and a private key -as the credential. - -To obtain these from GCE, see the [jclouds instructions](https://jclouds.apache.org/guides/google). - -An example of the expected format is shown below (note the credential is one long line, -with `\n` to represent the new line characters): - - brooklyn.location.jclouds.google-compute-engine.identity=123456789...@developer.gserviceaccount.com - brooklyn.location.jclouds.google-compute-engine.credential=-----BEGIN RSA PRIVATE KEY-----\nabcdefghijklmnopqrstuvwxyznabcdefghijk/lmnopqrstuvwxyzabcdefghij\nabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghij+lm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxy\nzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk\nlmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\nxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghi\njklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstu\nvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefg\nhijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrs\ntuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcde\nfghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\n-----END RSA PRIVATE KEY----- - - -### Quotas - -GCE accounts can have low default [quotas](https://cloud.google.com/compute/docs/resource-quotas). - -It is easy to requesta quota increase by submitting a [quota increase form](https://support.google.com/cloud/answer/6075746?hl=en). - - -### Networks - -GCE accounts often have a limit to the number of networks that can be created. One work around -is to manually create a network with the required open ports, and to refer to that named network -in Brooklyn's location configuration. - -To create a network, see [GCE network instructions](https://cloud.google.com/compute/docs/networking#networks_1). - -For example, for dev/demo purposes an "everything" network could be created that opens all ports. - -| Name | everything | -| Description | opens all tcp ports | -| Source IP Ranges | 0.0.0.0/0 | -| Allowed protocols and ports | tcp:0-65535 and udp:0-65535 | - - +This page has moved. See [More Clouds](more-clouds.html) instead. http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/9d4ab40e/guide/ops/locations/index.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/index.md b/guide/ops/locations/index.md index 5ab7737..92b3ad8 100644 --- a/guide/ops/locations/index.md +++ b/guide/ops/locations/index.md @@ -6,7 +6,7 @@ children: - { section: Inheritance and Named Locations, title: Named Locations } - { section: Localhost } - { section: BYON } -- cloud-credentials.md +- { title: More on Clouds, path: more-clouds.md } - more-locations.md - location-customizers.md - ssh-keys.md @@ -62,8 +62,12 @@ brooklyn.location.jclouds.aws-ec2.credential=s3cr3tsq1rr3ls3cr3tsq1rr3ls3cr3tsq1 And in this case you can reference the location in YAML with `location: jclouds:aws-ec2`. -The Getting Started [template brooklyn.properties]({{ site.path.guide }}/start/brooklyn.properties) contains more examples -of configuring cloud endpoints, including information on credential types used in different clouds. +Brooklyn irons out many of the differences between clouds so that blueprints run similarly +in a wide range of locations, including setting up access and configuring images and machine specs. +The configuration options are described in more detail below. + +In some cases, a cloud provider may offer special features or unexpected quirks; +known issues are described a the **[More Details for Specific Clouds](more-clouds.html)** page. #### OS Initial Login and Setup @@ -216,7 +220,7 @@ For more keys and more detail on the keys below, see This setting prevents scripts executed on the VMs from being deleted on completion. Note that some scripts run periodically so this can eventually fill a disk; it should only be used for dev/test. -###### Custom template options +###### Custom Template Options jclouds supports many additional options for configuring how a virtual machine is created and deployed, many of which are for cloud-specific features and enhancements. Brooklyn supports some of these, but if what you are looking for is @@ -445,7 +449,7 @@ provisioned. See `FixedListMachineProvisioningLocation.MACHINE_CHOOSER`. ### Other Location Topics -* [Cloud Credentials](cloud-credentials.html) +* [More Details on Specific Clouds](more-clouds.html) * [More Locations](more-locations.html) * [Location Customizers](location-customizers.html) * [SSH Keys](ssh-keys.html) http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/9d4ab40e/guide/ops/locations/location-customizers.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/location-customizers.md b/guide/ops/locations/location-customizers.md index ad1f17e..a0f21ed 100644 --- a/guide/ops/locations/location-customizers.md +++ b/guide/ops/locations/location-customizers.md @@ -1,5 +1,5 @@ --- -title: Location customizers +title: Location Customizers layout: website-normal --- http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/9d4ab40e/guide/ops/locations/more-clouds.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/more-clouds.md b/guide/ops/locations/more-clouds.md new file mode 100644 index 0000000..5b776df --- /dev/null +++ b/guide/ops/locations/more-clouds.md @@ -0,0 +1,171 @@ +--- +title: More Details on Specific Clouds +layout: website-normal +--- + +To connect to a Cloud, Brooklyn requires appropriate credentials. These comprise the `identity` and +`credential` in Brooklyn terminology. + +For private clouds (and for some clouds being targeted using a standard API), the `endpoint` +must also be specified, which is the cloud's URL. For public clouds, Brooklyn comes preconfigured +with the endpoints, but many offer different choices of the `region` where you might want to deploy. + +Clouds vary in the format of the identity, credential, endpoint, and region. +Some also have their own idiosyncracies. More details for configuring some common clouds +is included below. You may also find these sources helpful: + +* The **[template brooklyn.properties]({{ site.path.guide }}/start/brooklyn.properties)** file + in the Getting Started guide + contains numerous examples of configuring specific clouds, + including the format of credentials and options for sometimes-fiddly private clouds. +* The **[jclouds guides](https://jclouds.apache.org/guides)** describes low-level configuration + sometimes required for various clouds. + + + +## Amazon Web Services (AWS) + +### Credentials + +AWS has an "access key" and a "secret key", which correspond to Brooklyn's identity and credential +respectively. + +These keys are the way for any programmatic mechanism to access the AWS API. + +To generate an access key and a secret key, see [jclouds instructions](http://jclouds.apache.org/guides/aws) +and [AWS IAM instructions](http://docs.aws.amazon.com/IAM/latest/UserGuide/ManagingCredentials.html). + +An example of the expected format is shown below: + + brooklyn.location.jclouds.aws-ec2.identity=ABCDEFGHIJKLMNOPQRST + brooklyn.location.jclouds.aws-ec2.credential=abcdefghijklmnopqrstu+vwxyzabcdefghijklm + + +### Tidying up after jclouds + +Security groups are not always deleted by jclouds. This is due to a limitation in AWS (see +https://issues.apache.org/jira/browse/JCLOUDS-207). In brief, AWS prevents the security group +being deleted until there are no VMs using it. However, there is eventual consistency for +recording which VMs still reference those security groups: after deleting the VM, it can sometimes +take several minutes before the security group can be deleted. jclouds retries for 3 seconds, but +does not block for longer. + +There is utility written by Cloudsoft for deleting these unused resources: +http://www.cloudsoftcorp.com/blog/2013/03/tidying-up-after-jclouds. + + +### Using Subnets and Security Groups + +Apache Brooklyn can run with AWS VPC and both public and private subnets. +Simply provide the `subnet-a1b2c3d4` as the `networkName` when deploying: + + location: + jclouds:aws-ec2: + region: us-west-1 + networkName: subnet-a1b2c3d4 # use your subnet ID + +Subnets are typically used in conjunction with security groups. +Brooklyn does *not* attempt to open additional ports +when private subnets or security groups are supplied, +so the subnet and ports must be configured appropriately for the blueprints being deployed. +You can configure a default security group with appropriate (or all) ports opened for +access from the appropriate (or all) CIDRs and security groups, +or you can define specific `securityGroups` on the location +or as `provisioning.properties` on the entities. + +Make sure that Brooklyn has access to the machines under management. +This includes SSH, which might be done with a public IP created with inbound access +on port 22 permitted for a CIDR range including the IP from which Brooklyn contacts it. +Alternatively you can run Brooklyn on a machine in that same subnet, or +set up a VPN or jumphost which Brooklyn will use. + + +### EC2 "Classic" Problems with VPC-only Hardware Instance Types + +If you have a pre-2014 Amazon account, it is likely configured in some regions to run in "EC2 Classic" mode +by default, instead of the more modern "VPC" default mode. This can cause failures when requesting certain hardware +configurations because many of the more recent hardware "instance types" only run in "VPC" mode. +For instance when requesting an instance with `minRam: 8gb`, Brooklyn may opt for an `m4.large`, +which is a VPC-only instance type. If you are in a region configured to use "EC2 Classic" mode, +you may see a message such as this: + + 400 VPCResourceNotSpecified: The specified instance type can only be used in a VPC. + A subnet ID or network interface ID is required to carry out the request. + +This is a limitation of "legacy" accounts. The easiest fixes are either: + +* specify an instance type which is supported in classic, such as `m3.xlarge` (see below) +* move to a different region where VPC is the default + (`eu-central-1` should work as it *only* offers VPC mode, + irrespective of the age of your AWS account) +* get a new AWS account -- "VPC" will be the default mode + (Amazon recommend this and if you want to migrate existing deployments + they provide [detailed instructions](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/vpc-migrate.html)) + +To understand the situation, the following resources may be useful: + +* Background on VPC vs Classic: [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-vpc.html](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-vpc.html) +* Good succinct answers to FAQs: [http://aws.amazon.com/vpc/faqs/#Default_VPCs]() +* Check if a region in your account is "VPC" or "Classic": [http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/default-vpc.html#default-vpc-availability]() +* Regarding instance types: + * All instance types: [https://aws.amazon.com/ec2/instance-types]() + * Those which require VPC: [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-vpc.html#vpc-only-instance-types]() + +If you want to solve this problem with your existing account, +you can create a VPC and instruct Brooklyn to use it: + +1. Use the "Start VPC Wizard" option in [the VPC dashboard](https://console.aws.amazon.com/vpc), + making sure it is for the right region, and selecting a "Single Public Subnet". + (More information is in [these AWS instructions](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/get-set-up-for-amazon-ec2.html#create-a-vpc).) +2. Once the VPC is created, open the "Subnets" view and modify the "Public subnet" + so that it will "Auto-assign Public IP". +3. Next click on the "Security Groups" and find the `default` security group for that VPC. + Modify its "Inbound Rules" to allow "All traffic" from "Anywhere". + (Or for more secure options, see the instructions in the previous section, + "Using Subnets".) +4. Finally make a note of the subnet ID (e.g. `subnet-a1b2c3d4`) for use in Brooklyn. + +You can then deploy blueprints to the subnet, allowing VPC hardware instance types, +by specifying the subnet ID as the `networkName` in your YAML blueprint. +This is covered in the previous section, "Using Subnets". + + +## Google Compute Engine (GCE) + +### Credentials + +GCE uses a service account e-mail address for the identity and a private key as the credential. + +To obtain these from GCE, see the [jclouds instructions](https://jclouds.apache.org/guides/google). + +An example of the expected format is shown below. +Note that when supplying the credential in a properties file, it should be one long line +with `\n` representing the new line characters: + + brooklyn.location.jclouds.google-compute-engine.identity=123456789...@developer.gserviceaccount.com + brooklyn.location.jclouds.google-compute-engine.credential=-----BEGIN RSA PRIVATE KEY-----\nabcdefghijklmnopqrstuvwxyznabcdefghijk/lmnopqrstuvwxyzabcdefghij\nabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghij+lm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxy\nzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk\nlmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\nxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghi\njklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstu\nvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefg\nhijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrs\ntuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcde\nfghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\n-----END RSA PRIVATE KEY----- + + +### Quotas + +GCE accounts can have low default [quotas](https://cloud.google.com/compute/docs/resource-quotas). + +It is easy to requesta quota increase by submitting a [quota increase form](https://support.google.com/cloud/answer/6075746?hl=en). + + +### Networks + +GCE accounts often have a limit to the number of networks that can be created. One work around +is to manually create a network with the required open ports, and to refer to that named network +in Brooklyn's location configuration. + +To create a network, see [GCE network instructions](https://cloud.google.com/compute/docs/networking#networks_1). + +For example, for dev/demo purposes an "everything" network could be created that opens all ports. + +|| Name || everything | +|| Description || opens all tcp ports | +|| Source IP Ranges || 0.0.0.0/0 | +|| Allowed protocols and ports || tcp:0-65535 and udp:0-65535 | + +
