Updated Branches: refs/heads/ACS101 59a58bcf9 -> c7243d4da
more docs on knife and whirr Project: http://git-wip-us.apache.org/repos/asf/cloudstack/repo Commit: http://git-wip-us.apache.org/repos/asf/cloudstack/commit/c7243d4d Tree: http://git-wip-us.apache.org/repos/asf/cloudstack/tree/c7243d4d Diff: http://git-wip-us.apache.org/repos/asf/cloudstack/diff/c7243d4d Branch: refs/heads/ACS101 Commit: c7243d4da23b1cf01326305386b40e1da88a509a Parents: 59a58bc Author: Sebastien Goasguen <[email protected]> Authored: Mon Jun 10 08:25:14 2013 -0400 Committer: Sebastien Goasguen <[email protected]> Committed: Mon Jun 10 08:25:14 2013 -0400 ---------------------------------------------------------------------- docs/acs101/en-US/The_Little__Book.xml | 1 - docs/acs101/en-US/knife-cloudstack.xml | 109 ++++++++++++++--- docs/acs101/en-US/whirr.xml | 176 ++++++++++++++++++++++++++-- 3 files changed, 262 insertions(+), 24 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cloudstack/blob/c7243d4d/docs/acs101/en-US/The_Little__Book.xml ---------------------------------------------------------------------- diff --git a/docs/acs101/en-US/The_Little__Book.xml b/docs/acs101/en-US/The_Little__Book.xml index c2dda0f..6f1a08b 100644 --- a/docs/acs101/en-US/The_Little__Book.xml +++ b/docs/acs101/en-US/The_Little__Book.xml @@ -25,7 +25,6 @@ <book> <xi:include href="Book_Info.xml" xmlns:xi="http://www.w3.org/2001/XInclude"; /> - <xi:include href="Overview.xml" xmlns:xi="http://www.w3.org/2001/XInclude"; /> <xi:include href="Gettinggoing.xml" xmlns:xi="http://www.w3.org/2001/XInclude"; /> <xi:include href="Clientsandshells.xml" xmlns:xi="http://www.w3.org/2001/XInclude"; /> <xi:include href="Wrappers.xml" xmlns:xi="http://www.w3.org/2001/XInclude"; /> http://git-wip-us.apache.org/repos/asf/cloudstack/blob/c7243d4d/docs/acs101/en-US/knife-cloudstack.xml ---------------------------------------------------------------------- diff --git a/docs/acs101/en-US/knife-cloudstack.xml b/docs/acs101/en-US/knife-cloudstack.xml index 4096792..a859311 100644 --- a/docs/acs101/en-US/knife-cloudstack.xml +++ b/docs/acs101/en-US/knife-cloudstack.xml @@ -211,35 +211,114 @@ foobar 178.170.XX.XX m1.small CentOS 6.4 - Minimal - 64bits Running N/A </section> <section id="bootstrapping-chef"> - <title>Bootstrapping Chef</title> + <title>Bootstrapping Instances with Hosted-Chef</title> <para> - Knife is taking it's full potential when used to bootstrap Chef and use it for configuration management of the instances. Below is an example that does so: + Knife is taking it's full potential when used to bootstrap Chef and use it for configuration management of the instances. To get started with Chef, the easiest is to use <ulink url="http://www.opscode.com/hosted-chef/";>Hosted Chef</ulink>. There is some great documentation on <ulink url="https://learnchef.opscode.com/quickstart/chef-repo/";>how</ulink> to do it. The basics concept is that you will download or create cookbooks locally and publish them to your own hosted Chef server. + </para> + </section> + <section id="boostrapping-knife"> + <title>Using Knife with Hosted-Chef</title> + <para> + With your <emphasis>hosted Chef</emphasis> account created and your local <emphasis>chef-repo</emphasis> setup, you can start instances on your Cloud and specify the <emphasis>cookbooks</emphasis> to use to configure those instances. The boostrapping process will fetch those cookbooks and configure the node. Below is an example that does so, it uses the <ulink url="http://www.exoscale.ch";>exoscale</ulink> cloud which runs on CloudStack. This cloud is enabled as a Basic zone and uses ssh keypairs and security groups for access. </para> <para> <programlisting> -$ knife cs server create --service m1.small --template "CentOS 6.4 - Minimal - 64bits" --ipfwd-rules 22 --ssh-user sebgoa --ssh-password totoestcon foobar4 +$ knife cs server create --service Tiny --template "Linux CentOS 6.4 64-bit" --ssh-user root --identity ~/.ssh/id_rsa --run-list "recipe[apache2]" --ssh-keypair foobar --security-group www --no-public-ip foobar -Waiting for Server to be created....... +Waiting for Server to be created.... +Name: foobar +Public IP: 185.19.XX.XX + + +Waiting for sshd..... + +Name: foobar13 +Public IP: 185.19.XX.XX +Environment: _default +Run List: recipe[apache2] + +Bootstrapping Chef on 185.19.XX.XX +185.19.XX.XX --2013-06-10 11:47:54-- http://opscode.com/chef/install.sh +185.19.XX.XX Resolving opscode.com... +185.19.XX.XX 184.ZZ.YY.YY +185.19.XX.XX Connecting to opscode.com|184.ZZ.XX.XX|:80... +185.19.XX.XX connected. +185.19.XX.XX HTTP request sent, awaiting response... +185.19.XX.XX 301 Moved Permanently +185.19.XX.XX Location: http://www.opscode.com/chef/install.sh [following] +185.19.XX.XX --2013-06-10 11:47:55-- http://www.opscode.com/chef/install.sh +185.19.XX.XX Resolving www.opscode.com... +185.19.XX.XX 184.ZZ.YY.YY +185.19.XX.XX Reusing existing connection to opscode.com:80. +185.19.XX.XX HTTP request sent, awaiting response... +185.19.XX.XX 200 OK +185.19.XX.XX Length: 6509 (6.4K) [application/x-sh] +185.19.XX.XX Saving to: âSTDOUTâ +185.19.XX.XX + 0% [ ] 0 --.-K/s +100%[======================================>] 6,509 --.-K/s in 0.1s +185.19.XX.XX +185.19.XX.XX 2013-06-10 11:47:55 (60.8 KB/s) - written to stdout [6509/6509] +185.19.XX.XX +185.19.XX.XX Downloading Chef 11.4.4 for el... +185.19.XX.XX Installing Chef 11.4.4 + </programlisting> + </para> + <para> + Chef will then configure the machine based on the cookbook passed in the --run-list option, here I setup a simple webserver. Note the keypair that I used and the security group. I also specify <emphasis>--no-public-ip</emphasis> which disables the IP address allocation and association. This is specific to the setup of <emphasis>exoscale</emphasis> which automatically uses a public IP address for the instances. + </para> + <note> + <para> + The latest version of knife-cloudstack allows you to manage keypairs and securitygroups. For instance listing, creation and deletion of keypairs is possible, as well as listing of securitygroups: + <programlisting> +$ knife cs securitygroup list +Name Description Account +default Default Security Group [email protected] +www apache server [email protected] +$ knife cs keypair list +Name Fingerprint +exoscale xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx + </programlisting> + </para> + </note> + <para> + When using a &PRODUCT; based cloud in an Advanced zone setting, <emphasis>knife</emphasis> can automatically allocate and associate an IP address. To illustrate this slightly different example I use <ulink url="http://www.ikoula.com";>iKoula</ulink> a french Cloud Provider which uses &PRODUCT;. I edit my <emphasis>knife.rb</emphasis> file to setup a different endpoint and the different API and secret keys. I remove the keypair, security group and public ip option and I do not specify an identity file as I will retrieve the ssh password with the <emphasis>--cloudstack-password</emphasis> option. The example is as follows: + </para> + <para> + <programlisting> +$ knife cs server create --service m1.small --template "CentOS 6.4 - Minimal - 64bits" --ssh-user root --cloudstack-password --run-list "recipe[apache2]" foobar + +Waiting for Server to be created........ Allocate ip address, create forwarding rules params: {"command"=>"associateIpAddress", "zoneId"=>"a41b82a0-78d8-4a8f-bb79-303a791bb8a7", "networkId"=>"df2288bb-26d7-4b2f-bf41-e0fae1c6d198"}. -Allocated IP Address: 178.170.71.134 -.... -Name: foobar4 -Public IP: 178.170.71.134 +Allocated IP Address: 178.170.71.148 +... +Name: foobar +Password: $%@#$%#$%#$ +Public IP: 178.xx.yy.zz -Waiting for sshd..... +Waiting for sshd...... -Name: foobar4 -Public IP: 178.170.71.134 -Environment: _default -Run List: +Name: foobar +Public IP: 178.xx.yy.zz +Environment: _default +Run List: recipe[apache2] -Bootstrapping Chef on 178.170.71.134 -ERROR: Errno::ENOENT: No such file or directory - /etc/chef/validation.pem +Bootstrapping Chef on 178.xx.yy.zz +178.xx.yy.zz --2013-06-10 13:24:29-- http://opscode.com/chef/install.sh +178.xx.yy.zz Resolving opscode.com... </programlisting> </para> + <warning> + <para> + You will want to review the security implications of doing the boostrap as root and using the default password to do so. + </para> + </warning> + <para> + With the basics of Chef configuration and usage covered, as well as the basic examples of using <emphasis>knife cloudstack</emphasis> to provision and configure instances we can now move on to the interesting case of configuring a group of instances and handling basic dependencies between those machines. The typical use case is when machines need to be provisioned according to a schedule so that information about one instance can be passed to another one. With <emphasis>knife cloudstack</emphasis> this is possible by using <emphasis>stacks</emphasis>. + </para> </section> http://git-wip-us.apache.org/repos/asf/cloudstack/blob/c7243d4d/docs/acs101/en-US/whirr.xml ---------------------------------------------------------------------- diff --git a/docs/acs101/en-US/whirr.xml b/docs/acs101/en-US/whirr.xml index c3f3123..4b14a1a 100644 --- a/docs/acs101/en-US/whirr.xml +++ b/docs/acs101/en-US/whirr.xml @@ -24,12 +24,172 @@ <section id="whirr"> <title>Apache Whirr</title> - <para>Whirr is ta stream processing analysis framework provides a <emphasis>Cloud in a box</emphasis>.</para> - <note> - <para>DevCloud is provided as a convenience by community members. It is not an official &PRODUCT; release artifact.</para> - <para>The &PRODUCT; source code however, contains tools to build your own DevCloud.</para> - </note> - <warning> - <para>Storm is </para> - </warning> + <para><ulink url="http://whirr.apache.org";>Apache Whirr</ulink> is a set of libraries to run cloud services, internally it uses <ulink url="http://jclouds.incubator.apache.org";>jclouds</ulink> that we introduced earlier via the jclouds-cli interface to &PRODUCT;, it is java based and of interest to provision clusters of virtual machines on cloud providers. Historically it started as a set of scripts to deploy <ulink url="http://hadoop.apache.org";>Hadoop</ulink> clusters on Amazon EC2. We introduce Whirr has a potential &PRODUCT; tool to provision Hadoop cluster on &PRODUCT; based clouds.</para> + + <section id="whirr-install"> + <title>Installing Apache Whirr</title> + <para> + To install Whirr you can follow the <ulink url="http://whirr.apache.org/docs/0.8.1/quick-start-guide.html";>Quick Start Guide</ulink>, download a tarball or clone the git repository. In the spirit of this document we clone the repo: + </para> + <programlisting> +git clone git://git.apache.org/whirr.git + </programlisting> + <para> + And build the source with maven that we now know and love...: + </para> + <programlisting> +mvn install + </programlisting> + <para> + The whirr binary will be available in the <emphasis>bin</emphasis> directory that we can add to our path + </para> + <programlisting> +export PATH=$PATH:/Users/sebgoa/Documents/whirr/bin + </programlisting> + <para> + If all went well you should now be able to get the usage of <emphasis>whirr</emphasis>: + </para> + <programlisting> +$ whirr --help +Unrecognized command '--help' + +Usage: whirr COMMAND [ARGS] +where COMMAND may be one of: + + launch-cluster Launch a new cluster running a service. + start-services Start the cluster services. + stop-services Stop the cluster services. +restart-services Restart the cluster services. + destroy-cluster Terminate and cleanup resources for a running cluster. +destroy-instance Terminate and cleanup resources for a single instance. + list-cluster List the nodes in a cluster. + list-providers Show a list of the supported providers + run-script Run a script on a specific instance or a group of instances matching a role name + version Print the version number and exit. + help Show help about an action + +Available roles for instances: + cassandra + elasticsearch + ganglia-metad + ganglia-monitor + hadoop-datanode + hadoop-jobtracker + hadoop-namenode + hadoop-tasktracker + hama-groomserver + hama-master + hbase-avroserver + hbase-master + hbase-regionserver + hbase-restserver + hbase-thriftserver + kerberosclient + kerberosserver + mahout-client + mapreduce-historyserver + noop + pig-client + puppet-install + solr + yarn-nodemanager + yarn-resourcemanager + zookeeper + </programlisting> + <para> + From the look of the usage you clearly see that <emphasis>whirr</emphasis> is about more than just <emphasis>hadoop</emphasis> and that it can be used to configure <emphasis>elasticsearch</emphasis> clusters, <emphasis>cassandra</emphasis> databases as well as the entire <emphasis>hadoop</emphasis> ecosystem with <emphasis>mahout</emphasis>, <emphasis>pig</emphasis>, <emphasis>hbase</emphasis>, <emphasis>hama</emphasis>, <emphasis>mapreduce</emphasis> and <emphasis>yarn</emphasis>. + </para> + </section> + + <section id="whirr-use"> + <title>Using Apache Whirr</title> + <para> + To get started with Whirr you need to setup the credentials and endpoint of your &PRODUCT; based cloud that you will be using. Edit the <emphasis>~/.whirr/credentials</emphasis> file to include a PROVIDER, IDENTITY, CREDENTIAL and ENDPOINT. The PROVIDER needs to be set to <emphasis>cloudstack</emphasis>, the IDENTITY is your API key, the CREDENTIAL is your secret key and the ENDPPOINT is the endpoint url. For instance: + </para> + <para> + <programlisting> +PROVIDER=cloudstack +IDENTITY=mnH5EbKc4534592347523486724389673248AZW4kYV5gdsfgdfsgdsfg87sdfohrjktn5Q +CREDENTIAL=Hv97W58iby5PWL1ylC4oJls46456435634564537sdfgdfhrteydfg87sdf89gysdfjhlicg +ENDPOINT=https://api.exoscale.ch/compute + </programlisting> + </para> + <para> + With the credentials and endpoint defined you can create a <emphasis>properties</emphasis> file that describes the cluster you want to launch on your cloud. The file contains information such as the cluster name, the number of instances and their type, the distribution of hadoop you want to use, the service offering id and the template id of the instances. It also defines the ssh keys to be used for accessing the virtual machines. In the case of a cloud that uses security groups, you may also need to specify it. A tricky point is the handling of DNS name resolution. You might have to use the <emphasis>whirr.store-cluster-in-etc-hosts</emphasis> key to bypass any DNS issues. For a full description of the whirr property keys, see the <ulink url="http://whirr.apache.org/docs/0.8.1/configuration-guide.html";>documentation</ulink>. + </para> + <para> + <programlisting> +$ more whirr.properties + +# +# Setup an Apache Hadoop Cluster +# + +# Change the cluster name here +whirr.cluster-name=hadoop + +whirr.store-cluster-in-etc-hosts=true + +whirr.use-cloudstack-security-group=true + +# Change the name of cluster admin user +whirr.cluster-user=${sys:user.name} + +# Change the number of machines in the cluster here +whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,3 hadoop-datanode+hadoop-tasktracker + +# Uncomment out the following two lines to run CDH +whirr.env.repo=cdh4 +whirr.hadoop.install-function=install_cdh_hadoop +whirr.hadoop.configure-function=configure_cdh_hadoop + +whirr.hardware-id=b6cd1ff5-3a2f-4e9d-a4d1-8988c1191fe8 + +whirr.private-key-file=/path/to/ssh/key/ +whirr.public-key-file=/path/to/ssh/public/key/ + +whirr.provider=cloudstack +whirr.endpoint=https://the/endpoint/url +whirr.image-id=1d16c78d-268f-47d0-be0c-b80d31e765d2 + </programlisting> + </para> + <para> + You are now ready to launch an hadoop cluster: + </para> + <para> + <programlisting> +<![CDATA[ +$ whirr launch-cluster --config hadoop.properties +Running on provider cloudstack using identity mnH5EbKcKeJd456456345634563456345654634563456345 +Bootstrapping cluster +Configuring template for bootstrap-hadoop-datanode_hadoop-tasktracker +Configuring template for bootstrap-hadoop-namenode_hadoop-jobtracker +Starting 3 node(s) with roles [hadoop-datanode, hadoop-tasktracker] +Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker] +>> running InitScript{INSTANCE_NAME=bootstrap-hadoop-datanode_hadoop-tasktracker} on node(b9457a87-5890-4b6f-9cf3-1ebd1581f725) +>> running InitScript{INSTANCE_NAME=bootstrap-hadoop-datanode_hadoop-tasktracker} on node(9d5c46f8-003d-4368-aabf-9402af7f8321) +>> running InitScript{INSTANCE_NAME=bootstrap-hadoop-datanode_hadoop-tasktracker} on node(6727950e-ea43-488d-8d5a-6f3ef3018b0f) +>> running InitScript{INSTANCE_NAME=bootstrap-hadoop-namenode_hadoop-jobtracker} on node(6a643851-2034-4e82-b735-2de3f125c437) +<< success executing InitScript{INSTANCE_NAME=bootstrap-hadoop-datanode_hadoop-tasktracker} on node(b9457a87-5890-4b6f-9cf3-1ebd1581f725): {output=This function does nothing. It just needs to exist so Statements.call("retry_helpers") doesn't call something which doesn't exist +Get:1 http://security.ubuntu.com precise-security Release.gpg [198 B] +Get:2 http://security.ubuntu.com precise-security Release [49.6 kB] +Hit http://ch.archive.ubuntu.com precise Release.gpg +Get:3 http://ch.archive.ubuntu.com precise-updates Release.gpg [198 B] +Get:4 http://ch.archive.ubuntu.com precise-backports Release.gpg [198 B] +Hit http://ch.archive.ubuntu.com precise Release +..../snip/..... +You can log into instances using the following ssh commands: +[hadoop-datanode+hadoop-tasktracker]: ssh -i /Users/sebastiengoasguen/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no [email protected] +[hadoop-datanode+hadoop-tasktracker]: ssh -i /Users/sebastiengoasguen/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no [email protected] +[hadoop-datanode+hadoop-tasktracker]: ssh -i /Users/sebastiengoasguen/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no [email protected] +[hadoop-namenode+hadoop-jobtracker]: ssh -i /Users/sebastiengoasguen/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no [email protected] +To destroy cluster, run 'whirr destroy-cluster' with the same options used to launch it. +]]> + </programlisting> + </para> + <para> + After the boostrapping process finishes you should be able to login to your instances and use <emphasis>hadoop</emphasis> or if you are running a proxy on your machine, you will be able to access your hadoop cluster locally. Testing of Whirr for &PRODUCT; is still under <ulink url="https://issues.apache.org/jira/browse/WHIRR-725";>investigation</ulink> and the subject of a Google Summer of Code 2013 project. More information will be added as we learn them. + </para> + </section> + </section>
