mikewalch closed pull request #242: Fixes #239 - Several updates URL: https://github.com/apache/fluo-muchos/pull/242
This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/README.md b/README.md index cd7544a..75b9f5a 100644 --- a/README.md +++ b/README.md @@ -21,22 +21,50 @@ Muchos is structured into two high level components: Checkout [Uno] for setting up Accumulo or Fluo on a single machine. -## Installation +## Requirements -First clone the Muchos repo: +Muchos requires the following: - git clone https://github.com/apache/fluo-muchos.git +* Python 2 +* [awscli] & [boto3] libraries - Install using `pip install awscli boto3 --upgrade --user` +* `ssh-agent` installed and running +* An AWS account with your SSH public key uploaded. When you configure [muchos.props], set `key.name` + to name of your key pair in AWS. +* `~/.aws` [configured][aws-config] on your machine. Can be created manually or using [aws configure][awscli-config]. -Now, create and modify your [muchos.props] configuration file for Muchos: +## Quickstart - cd muchos/ - cp conf/muchos.props.example conf/muchos.props +The following commands will install Muchos, launch an EC2 cluster, and setup/run Accumulo: -In order to run the `muchos` command, you will need to create [AWS configuration and credential files][aws-config] -in your home directory. These files can be created by hand or by running `aws configure` using the [AWS CLI][aws-cli]. +```bash +git clone https://github.com/apache/fluo-muchos.git +cd fluo-muchos/ +cp conf/muchos.props.example conf/muchos.props +vim conf/muchos.props # Edit to configure Muchos cluster +./bin/muchos launch -c mycluster # Launches Muchos cluster in EC2 +./bin/muchos setup # Set up cluster and start Accumulo +``` + +The `setup` command can be run repeatedly to fix any failures and will not repeat successful operations. + +After your cluster is launched, SSH to it using the following command: + +```bash +./bin/muchos ssh +``` + +Run the following command to terminate your cluster. WARNING: All cluster data will be lost. -You will need to upload your public key to the AWS management console and set `key.name` in -[muchos.props] to the name of your key pair. If you want to give others access to your cluster, add +```bash +./bin/muchos terminate +``` + +Please continue reading for more detailed Muchos instructions. + +## Launching an EC2 cluster + +Before launching a cluster, you will need to complete the requirements above, clone the Muchos repo, and +create [muchos.props]. If you want to give others access to your cluster, add their public keys to a file named `keys` in your `conf/` directory. During the setup of your cluster, this file will be appended on each node to the `~/.ssh/authorized_keys` file for the user set by the `cluster.username` property. @@ -49,26 +77,24 @@ error when you try to launch your cluster. By default, the `aws_ami` property is You will need to changes this value if a newer image has been released or if you are running in different region than `us-east-1`. -## Launching an EC2 cluster - -After following the installation steps above, run the following command to launch an EC2 cluster called `mycluster`: +After following the steps above, run the following command to launch an EC2 cluster called `mycluster`: - muchos launch -c mycluster + ./bin/muchos launch -c mycluster After your cluster has launched, you do not have to specify a cluster anymore using `-c` (unless you have multiple clusters running). Run the following command to confirm that you can ssh to the leader node: - muchos ssh + ./bin/muchos ssh You can check the status of the nodes using the EC2 Dashboard or by running the following command: - muchos status + ./bin/muchos status ## Set up the cluster -The `muchos setup` command will set up your cluster and start Hadoop, Zookeeper, & Accumulo. It +The `./bin/muchos setup` command will set up your cluster and start Hadoop, Zookeeper, & Accumulo. It will download release tarballs of Fluo, Accumulo, Hadoop, etc. The versions of these tarballs are specified in [muchos.props] and can be changed if desired. @@ -78,7 +104,7 @@ version of Fluo or Accumulo. Before running the `muchos setup` command, you shou version and SHA-256 hash of your tarball matches what is set in [muchos.props]. Run the command `shasum -a 256 /path/to/tarball` on your tarball to determine its hash. -The `muchos setup` command will install and start Accumulo, Hadoop, and Zookeeper. The optional +The `muchos setup` command will install and start Accumulo, Hadoop, and Zookeeper. The optional services below will only be set up if configured in the `[nodes]` section of [muchos.props]: 1. `fluo` - Fluo only needs to be installed and configured on a single node in your cluster as Fluo @@ -114,10 +140,10 @@ take over a minute, use `ctrl-c` to stop setup if it hangs for a long time. Just ## Manage the cluster The `setup` command is idempotent. It can be run again on a working cluster. It will not change the -cluster if everything is configured and running correctly. If a process has stopped, the `setup` +cluster if everything is configured and running correctly. If a process has stopped, the `setup` command will restart the process. -The `muchos wipe` command can be used to wipe all data from the cluster and kill any running +The `./bin/muchos wipe` command can be used to wipe all data from the cluster and kill any running processes. After running the `wipe` command, run the `setup` command to start a fresh cluster. If you set `proxy_socks_port` in your [muchos.props], a SOCKS proxy will be created on that port @@ -142,7 +168,7 @@ with Muchos as it configures your shell with common environment variables. To ru application, SSH to a node on cluster where Fluo is installed and clone the example repo: ```bash -muchos ssh # SSH to cluster proxy node +./bin/muchos ssh # SSH to cluster proxy node ssh <node where Fluo is installed> # Nodes with Fluo installed is determined by Muchos config hub clone apache/fluo-examples # Clone repo of Fluo example applications. Press enter for user/password. ``` @@ -165,7 +191,7 @@ mimic the scripts of example Fluo applications above. ## Customize your cluster -After `muchos setup` is run, users can install additional software on the cluster using their own +After `./bin/muchos setup` is run, users can install additional software on the cluster using their own Ansible playbooks. In their own playbooks, users can reference any configuration in the Ansible inventory file at `/etc/ansible/hosts` which is set up by Muchos on the proxy node. The inventory file lists the hosts for services on the cluster such as the Zookeeper nodes, Namenode, Accumulo @@ -178,7 +204,7 @@ managed in their own git repository (see [mikewalch/muchos-custom][mc] for an ex If you launched your cluster on EC2, run the following command terminate your cluster. WARNING - All data on your cluster will be lost: - muchos terminate + ./bin/muchos terminate ## Automatic shutdown of EC2 clusters @@ -198,7 +224,7 @@ If you decide later to cancel the shutdown, run `muchos cancel_shutdown`. The `config` command allows you to retrieve cluster configuration for your own scripts: ```bash -$ muchos config -p leader.public.ip +$ ./bin/muchos config -p leader.public.ip 10.10.10.10 ``` @@ -222,12 +248,14 @@ The following command runs the unit tests: [centos7]: https://aws.amazon.com/marketplace/pp/B00O7WM7QW [aws-config]: http://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html -[aws-cli]: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-quick-configuration +[awscli]: https://docs.aws.amazon.com/cli/latest/userguide/installing.html +[awscli-config]: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-quick-configuration [fluo-app]: https://github.com/apache/fluo/blob/master/docs/applications.md [WebIndex]: https://github.com/apache/fluo-examples/tree/master/webindex [Phrasecount]: https://github.com/apache/fluo-examples/tree/master/phrasecount [Stresso]: https://github.com/apache/fluo-examples/tree/master/stresso [boto]: http://boto.cloudhackers.com/en/latest/ +[boto3]: https://github.com/boto/boto3 [Ansible]: https://www.ansible.com/ [ti]: https://travis-ci.org/apache/fluo-muchos.svg?branch=master [tl]: https://travis-ci.org/apache/fluo-muchos diff --git a/bin/muchos b/bin/muchos index 65f6c78..26cd0f4 100755 --- a/bin/muchos +++ b/bin/muchos @@ -16,48 +16,11 @@ bin="$(dirname $0)" basedir="$( cd -P ${bin}/.. && pwd )" -lib="$basedir/lib" export MUCHOS_HOME=$basedir if [ ! -f "$basedir/conf/muchos.props" ]; then echo "Please create a configuration file at $basedir/conf/muchos.props" exit 1 fi -if [[ "$OSTYPE" == "darwin"* ]]; then - export MD5="md5 -q" -else - export MD5="md5sum" -fi - -function fetch_lib() { - version=$1 - md5=$2 - url=$3 - tarball="$version.tar.gz" - if [ ! -d "$lib/$version" ]; then - - wget -cq -P "$lib" "$url" - - actual_md5=$($MD5 "$lib/$tarball" | awk '{print $1}') - if [[ "$actual_md5" != "$md5" ]]; then - echo "The md5 checksum ($actual_md5) of $tarball does not match the expected checksum ($md5)" - exit 1 - fi - tar xzf "$lib/$tarball" -C "$lib" - rm "$lib/$tarball" - fi - export PYTHONPATH="$lib/$version:$PYTHONPATH" -} - -boto3_version="boto3-1.4.7" -boto3_md5="eb95e700fa9e06db6e9ff4ee159adbd9" -boto3_url="https://pypi.python.org/packages/b6/7c/b9caa157b514e0a456286172862428a83814e4eda114ea1e80267de85378/boto3-1.4.7.tar.gz#md5=eb95e700fa9e06db6e9ff4ee159adbd9"; - -bc_version="botocore-1.7.8" -bc_md5="d7e3179b619b936048d763b2f579a896" -bc_url="https://pypi.python.org/packages/d5/a1/499da39780f26bca1d2e652bd01f115addf57c6509d92f0d35334cbcacea/botocore-1.7.8.tar.gz#md5=d7e3179b619b936048d763b2f579a896"; - -fetch_lib $boto3_version $boto3_md5 $boto3_url -fetch_lib $bc_version $bc_md5 $bc_url -python -Wdefault "$lib/muchos/main.py" "$@" +python "$basedir/lib/muchos/main.py" "$@" diff --git a/lib/muchos/main.py b/lib/muchos/main.py index 930030c..ba2061b 100644 --- a/lib/muchos/main.py +++ b/lib/muchos/main.py @@ -1,4 +1,4 @@ -#!/usr/bin/python2 +#!/usr/bin/env python2 # Copyright 2014 Muchos authors (see AUTHORS) # ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
