Repository: bigtop Updated Branches: refs/heads/master f7fc0fe78 -> d8a506577
BIGTOP-1603. Rename docker-puppet and vagrant-puppet to better indicate its function and difference Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/d8a50657 Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/d8a50657 Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/d8a50657 Branch: refs/heads/master Commit: d8a50657734dd087472f72296d5152f28bb4f899 Parents: f7fc0fe Author: Evans Ye <[email protected]> Authored: Tue Jan 20 13:44:28 2015 +0000 Committer: Evans Ye <[email protected]> Committed: Tue Jan 27 13:55:20 2015 +0000 ---------------------------------------------------------------------- README.md | 2 +- bigtop-deploy/puppet/README.md | 2 +- bigtop-deploy/vm/docker-puppet/.gitignore | 3 - bigtop-deploy/vm/docker-puppet/Dockerfile | 33 ---- bigtop-deploy/vm/docker-puppet/README.md | 179 ------------------- bigtop-deploy/vm/docker-puppet/Vagrantfile | 74 -------- .../vm/docker-puppet/boot2docker/Vagrantfile | 24 --- bigtop-deploy/vm/docker-puppet/docker-hadoop.sh | 117 ------------ bigtop-deploy/vm/docker-puppet/provision.sh | 36 ---- .../vm/docker-puppet/vagrantconfig.yaml | 13 -- .../vm/vagrant-puppet-docker/.gitignore | 3 + .../vm/vagrant-puppet-docker/Dockerfile | 33 ++++ .../vm/vagrant-puppet-docker/README.md | 179 +++++++++++++++++++ .../vm/vagrant-puppet-docker/Vagrantfile | 74 ++++++++ .../boot2docker/Vagrantfile | 24 +++ .../vm/vagrant-puppet-docker/docker-hadoop.sh | 117 ++++++++++++ .../vm/vagrant-puppet-docker/provision.sh | 36 ++++ .../vm/vagrant-puppet-docker/vagrantconfig.yaml | 13 ++ bigtop-deploy/vm/vagrant-puppet-vm/.gitignore | 1 + bigtop-deploy/vm/vagrant-puppet-vm/README.md | 92 ++++++++++ bigtop-deploy/vm/vagrant-puppet-vm/Vagrantfile | 123 +++++++++++++ .../vm/vagrant-puppet-vm/vagrantconfig.yaml | 8 + bigtop-deploy/vm/vagrant-puppet/.gitignore | 1 - bigtop-deploy/vm/vagrant-puppet/README.md | 92 ---------- bigtop-deploy/vm/vagrant-puppet/Vagrantfile | 123 ------------- .../vm/vagrant-puppet/vagrantconfig.yaml | 8 - 26 files changed, 705 insertions(+), 705 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index de51707..cdf4c4c 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ The primary goal of Apache Bigtop is to build a community around the packaging a Immediately Get Started with Deployment and Smoke Testing of BigTop =================================================================== -The simplest way to get a feel for how bigtop works, is to just cd into `bigtop-deploy/vm` and try out the recipes under vagrant-puppet, docker-puppet, and so on. Each one rapidly spins up, and runs the bigtop smoke tests on, a local bigtop based big data distribution. Once you get the gist, you can hack around with the recipes to learn how the puppet/rpm/smoke-tests all work together, going deeper into the components you are interested in as described below. +The simplest way to get a feel for how bigtop works, is to just cd into `bigtop-deploy/vm` and try out the recipes under vagrant-puppet-vm, vagrant-puppet-docker, and so on. Each one rapidly spins up, and runs the bigtop smoke tests on, a local bigtop based big data distribution. Once you get the gist, you can hack around with the recipes to learn how the puppet/rpm/smoke-tests all work together, going deeper into the components you are interested in as described below. Quick overview of source code directories ========================================= http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/puppet/README.md ---------------------------------------------------------------------- diff --git a/bigtop-deploy/puppet/README.md b/bigtop-deploy/puppet/README.md index c76185a..b2a8b6a 100644 --- a/bigtop-deploy/puppet/README.md +++ b/bigtop-deploy/puppet/README.md @@ -30,7 +30,7 @@ If you have site.csv in /etc/puppet/config, Then you should use --confdir=/etc/puppet , and puppet finds the config dir underneath. -As an end to end example, you can follow the vagrant-puppet recipes to see how to set up +As an end to end example, you can follow the vagrant-puppet-vm recipes to see how to set up a puppet managed bigtop hadoop installation. Those examples are gauranteed to work and http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/.gitignore ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/docker-puppet/.gitignore b/bigtop-deploy/vm/docker-puppet/.gitignore deleted file mode 100644 index 2cd389d..0000000 --- a/bigtop-deploy/vm/docker-puppet/.gitignore +++ /dev/null @@ -1,3 +0,0 @@ -config.rb -config/ -hosts http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/Dockerfile ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/docker-puppet/Dockerfile b/bigtop-deploy/vm/docker-puppet/Dockerfile deleted file mode 100644 index 8e65126..0000000 --- a/bigtop-deploy/vm/docker-puppet/Dockerfile +++ /dev/null @@ -1,33 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -FROM bigtop/seed:centos-6.4 - -# enable ssh -RUN yum -y install openssh-server openssh-clients sudo -RUN sed -i.bak s/UsePAM\ yes/UsePAM\ no/ /etc/ssh/sshd_config -RUN ssh-keygen -q -N "" -t dsa -f /etc/ssh/ssh_host_dsa_key -RUN ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key - -# requiretty off -RUN sed -i.bak 's/requiretty/!requiretty/' /etc/sudoers - -# setup vagrant account -RUN mkdir /root/.ssh/ -RUN chmod 0755 /root/.ssh -RUN wget http://github.com/mitchellh/vagrant/raw/master/keys/vagrant.pub --no-check-certificate -O /root/.ssh/authorized_keys -RUN chmod 0644 /root/.ssh/authorized_keys - -CMD /usr/sbin/sshd -D http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/README.md ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/docker-puppet/README.md b/bigtop-deploy/vm/docker-puppet/README.md deleted file mode 100644 index 7848571..0000000 --- a/bigtop-deploy/vm/docker-puppet/README.md +++ /dev/null @@ -1,179 +0,0 @@ - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - ------------------------------------------------------------------------------------------------------------------------------------------------------- - -#BigTop Docker provisioner - -## Overview - -The Vagrantfile definition and wrapper script that creates Bigtop virtual Hadoop cluster on top of Docker containers for you, by pulling from existing publishing bigtop repositories. -This cluster can be used: - -- to test bigtop smoke tests -- to test bigtop puppet recipes - -These containers start sshd daemons, which vagrant uses to provision and install the hadoop cluster. - -This has been verified on docker client 1.2.0, with api version 1.15, and vagrant 1.6.5 on Fedora 20 as well as Centos 6. - -## Prerequisites - -### OS X and Windows - -* Install [VirtualBox](https://www.virtualbox.org/wiki/Downloads) - -* Install [Vagrant](http://www.vagrantup.com/downloads.html). Need version 1.6.5 or higher. - -### Linux - -* [Kernel Requirements](http://docker.readthedocs.org/en/v0.5.3/installation/kernel/) - -* Install [Docker](https://docs.docker.com/installation/) - -* Install [Vagrant](http://www.vagrantup.com/downloads.html) - -## Getting Started - -* Create a 3 node Bigtop Hadoop cluster from scratch - -NOTE : SELinux can PREVENT you from ssh'ing into your docker container. -As a brute force way to disable it - remove it from vi /etc/sysconfig/docker arguments -(fedora and centos may by default launch docker daemon with the --selinux-enabled option)! -In the future, lets update this README with the RIGHT way to allow selinux without breaking -ssh into a docker container! - -``` -service docker restart -docker pull bigtop/seed:centos-6.4 -``` - -Now, you can start your cluster: - -``` -cd bigtop/bigtop-deploy/vm/docker-puppet -./docker-hadoop.sh --build-image --create 3 -``` -In case of errors you can attempt running as root, or else, ping the mailing list. - -## USAGE - -1) Build up the base Docker image that supports Vagrant. - -``` -./docker-hadoop.sh --build-image -``` - -2) Create a Bigtop Hadoop cluster by given # of node. (will place a file called config.rb) - -``` -./docker-hadoop.sh --create 3 -``` - -3) Destroy the cluster. - -``` -./docker-hadoop.sh --destroy -``` - -4) Update your cluster after doing configuration changes. (re-run puppet apply) - -``` -./docker-hadoop.sh --provision -``` - -5) Run Bigtop smoke tests - -``` -./docker-hadoop.sh --smoke-tests -``` - -6) Chain your operations with-in one command. - -``` -./docker-hadoop.sh --build-image --create 5 --smoke-tests --destroy -``` - -Commands will be executed by following order: - -``` -build-image => create 5 node cluster => destroy the cluster -``` - -7) Run hbase-test.sh to evaluate the deployment. - -``` -../utils/hbase-test.sh -``` - -8) See helper message: - -``` -./docker-hadoop.sh -h -usage: docker-hadoop.sh [options] - -b, --build-image Build base Docker image for Bigtop Hadoop - (must be exectued at least once before creating cluster) - -c NUM_INSTANCES, --create=NUM_INSTANCES Create a docker based Bigtop Hadoop cluster - -p, --provision Deploy configuration changes - -s, --smoke-tests Run Bigtop smoke tests - -d, --destroy Destroy the cluster - -h, --help - -``` - -##Configurations - -* There are several parameters can be configured in the vagrantconfig.yaml: - -1) Modify memory limit for Docker containers - -``` -docker: - memory_size: "2048" - -``` - -2) If you're running Docker provisioner on OS X or Windows, you can customize the boot2docker VM settings - -``` -boot2docker: - memory_size: "4196" - number_cpus: "2" -``` - -3) Use different host ports mapping for web UIs - -``` -namenode_ui_port: "50070" -yarn_ui_port: "8088" -hbase_ui_port: "60010" - -``` -Note: If running on OS X or Windows, the boot2docker VM should be reloaded after ports changed - - -##Configure Apache Hadoop ecosystem components -* Choose the ecosystem you want to be deployed by modifying components in vagrantconfig.yaml - -``` -components: "hadoop,hbase,yarn,..." -``` - -By default, Apache Hadoop, YARN, and Apache HBase will be installed. -See `bigtop-deploy/puppet/config/site.csv.example` for more details. - -##Notes - -* Users currently using vagrant 1.6+ is strongly recommanded to upgrade to 1.6.4+, otherwise you will encounter the [issue](https://github.com/mitchellh/vagrant/issues/3769) when installing plguins http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/Vagrantfile ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/docker-puppet/Vagrantfile b/bigtop-deploy/vm/docker-puppet/Vagrantfile deleted file mode 100755 index 3ec023b..0000000 --- a/bigtop-deploy/vm/docker-puppet/Vagrantfile +++ /dev/null @@ -1,74 +0,0 @@ -# -*- mode: ruby -*- -# vi: set ft=ruby : - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -require 'fileutils' - -# number of instances -$num_instances = 3 - -# load configuration from config.rb -CONFIG = File.join(File.dirname(__FILE__), "config.rb") -if File.exist?(CONFIG) - require CONFIG -end - -require "yaml" - -_config = YAML.load(File.open(File.join(File.dirname(__FILE__), "vagrantconfig.yaml"), File::RDONLY).read) -CONF = _config - -ENV['VAGRANT_DEFAULT_PROVIDER'] ||= 'docker' - -Vagrant.require_version ">= 1.6.0" - -VAGRANTFILE_API_VERSION = "2" -Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| - - # image definition - config.vm.define "image", autostart: false do |image| - image.vm.provider "docker" do |d| - d.build_dir = "." - d.build_args = ["-t",CONF["docker"]["image"]] - d.vagrant_machine = "boot2docker" - d.vagrant_vagrantfile = "./boot2docker/Vagrantfile" - end - image.ssh.username = "root" - image.ssh.port = 22 - end - - # nodes definition - (1..$num_instances).each do |i| - config.vm.define "bigtop#{i}" do |bigtop| - # docker container settings - bigtop.vm.provider "docker" do |d| - d.image = CONF["docker"]["image"] - d.create_args = ["--privileged=true", "-m", CONF["docker"]['memory_size'] + "m"] - if "#{i}" == "1" - d.ports = [CONF["namenode_ui_port"] + ":50070", CONF["yarn_ui_port"] + ":8088", CONF["hbase_ui_port"] + ":60010"] - end - d.vagrant_machine = "boot2docker" - d.vagrant_vagrantfile = "./boot2docker/Vagrantfile" - end - bigtop.ssh.username = "root" - bigtop.ssh.port = 22 - bigtop.vm.hostname = "bigtop#{i}.docker" - bigtop.vm.synced_folder "../../../", "/bigtop-home" - end - end - -end http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/boot2docker/Vagrantfile ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/docker-puppet/boot2docker/Vagrantfile b/bigtop-deploy/vm/docker-puppet/boot2docker/Vagrantfile deleted file mode 100644 index 5913fec..0000000 --- a/bigtop-deploy/vm/docker-puppet/boot2docker/Vagrantfile +++ /dev/null @@ -1,24 +0,0 @@ -# -*- mode: ruby -*- -# vi: set ft=ruby : - -require "yaml" - -_config = YAML.load(File.open(File.join(File.dirname(__FILE__), "../vagrantconfig.yaml"), File::RDONLY).read) -CONF = _config - -VAGRANTFILE_API_VERSION = "2" - -Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| - - config.vm.define "boot2docker" do |node| - node.vm.box = "yungsang/boot2docker" - node.vm.provider "virtualbox" do |vb| - vb.customize ["modifyvm", :id, "--memory", CONF["boot2docker"]["memory_size"]] - vb.customize ['modifyvm', :id, '--cpus', CONF["boot2docker"]['number_cpus']] - end - node.vm.network "forwarded_port", host: CONF["namenode_ui_port"], guest: 50070 - node.vm.network "forwarded_port", host: CONF["yarn_ui_port"], guest: 8088 - node.vm.network "forwarded_port", host: CONF["hbase_ui_port"], guest: 60010 - end - -end http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/docker-hadoop.sh ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/docker-puppet/docker-hadoop.sh b/bigtop-deploy/vm/docker-puppet/docker-hadoop.sh deleted file mode 100755 index 4b504b3..0000000 --- a/bigtop-deploy/vm/docker-puppet/docker-hadoop.sh +++ /dev/null @@ -1,117 +0,0 @@ -#!/bin/bash - -usage() { - echo "usage: $PROG [options]" - echo " -b, --build-image Build base Docker image for Bigtop Hadoop" - echo " (must be exectued at least once before creating cluster)" - echo " -c NUM_INSTANCES, --create=NUM_INSTANCES Create a Docker based Bigtop Hadoop cluster" - echo " -p, --provision Deploy configuration changes" - echo " -s, --smoke-tests Run Bigtop smoke tests" - echo " -d, --destroy Destroy the cluster" - echo " -h, --help" - exit 1 -} - -build-image() { - vagrant up image --provider docker - { - echo "echo -e '\nBUILD IMAGE SUCCESS.\n'" |vagrant ssh image - } || { - >&2 echo -e "\nBUILD IMAGE FAILED!\n" - exit 2 - } -} - -create() { - echo "\$num_instances = $1" > config.rb - vagrant up --no-parallel - nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`) - hadoop_head_node=(`echo "hostname -f" |vagrant ssh ${nodes[0]} |tail -n 1`) - repo=$(get-yaml-config repo) - components=$(get-yaml-config components) - echo "/bigtop-home/bigtop-deploy/vm/utils/setup-env.sh" |vagrant ssh ${nodes[0]} - echo "/vagrant/provision.sh $hadoop_head_node $repo $components" |vagrant ssh ${nodes[0]} - bigtop-puppet ${nodes[0]} - for ((i=1 ; i<${#nodes[*]} ; i++)); do - ( - echo "/bigtop-home/bigtop-deploy/vm/utils/setup-env.sh" |vagrant ssh ${nodes[$i]} - echo "/vagrant/provision.sh $hadoop_head_node $repo $components" |vagrant ssh ${nodes[$i]} - bigtop-puppet ${nodes[$i]} - ) & - done - wait -} - -provision() { - nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`) - for node in $nodes; do - bigtop-puppet $node & - done - wait -} - -smoke-tests() { - nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`) - echo "/bigtop-home/bigtop-deploy/vm/utils/smoke-tests.sh" |vagrant ssh ${nodes[0]} -} - - -destroy() { - rm -rf ./hosts ./config ./config.rb - nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`) - for node in $nodes; do - vagrant destroy -f $node - done - wait -} - -bigtop-puppet() { - echo "puppet apply -d --confdir=/vagrant --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules /bigtop-home/bigtop-deploy/puppet/manifests/site.pp" |vagrant ssh $1 -} - -get-yaml-config() { - RUBY_EXE=ruby - which ruby > /dev/null 2>&1 - if [ $? -ne 0 ]; then - # use vagrant embedded ruby on Windows - RUBY_EXE=$(dirname $(which vagrant))/../embedded/bin/ruby - fi - RUBY_SCRIPT="data = YAML::load(STDIN.read); puts data['$1'];" - cat vagrantconfig.yaml | $RUBY_EXE -ryaml -e "$RUBY_SCRIPT" | tr -d '\r' -} - -PROG=`basename $0` - -if [ $# -eq 0 ]; then - usage -fi - -while [ $# -gt 0 ]; do - case "$1" in - -b|--build-image) - build-image - shift;; - -c|--create) - if [ $# -lt 2 ]; then - echo "Create requires a number" 1>&2 - usage - fi - create $2 - shift 2;; - -p|--provision) - provision - shift;; - -s|--smoke-tests) - smoke-tests - shift;; - -d|--destroy) - destroy - shift;; - -h|--help) - usage - shift;; - *) - echo "Unknown argument: '$1'" 1>&2 - usage;; - esac -done http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/provision.sh ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/docker-puppet/provision.sh b/bigtop-deploy/vm/docker-puppet/provision.sh deleted file mode 100755 index da087c6..0000000 --- a/bigtop-deploy/vm/docker-puppet/provision.sh +++ /dev/null @@ -1,36 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -sysctl kernel.hostname=`hostname -f` - -# Unmount device /etc/hosts and replace it by a shared hosts file -echo -e "`hostname -i`\t`hostname -f`" >> /vagrant/hosts -umount /etc/hosts -mv /etc/hosts /etc/hosts.bak -ln -s /vagrant/hosts /etc/hosts - -# Prepare puppet configuration file -cd /etc/puppet/modules && puppet module install puppetlabs/stdlib - -mkdir /vagrant/config -cat > /vagrant/config/site.csv << EOF -hadoop_head_node,$1 -hadoop_storage_dirs,/data/1,/data/2 -bigtop_yumrepo_uri,$2 -jdk_package_name,java-1.7.0-openjdk-devel.x86_64 -components,$3 -EOF http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/docker-puppet/vagrantconfig.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/docker-puppet/vagrantconfig.yaml b/bigtop-deploy/vm/docker-puppet/vagrantconfig.yaml deleted file mode 100644 index 6a53d02..0000000 --- a/bigtop-deploy/vm/docker-puppet/vagrantconfig.yaml +++ /dev/null @@ -1,13 +0,0 @@ -docker: - memory_size: "2048" - image: "bigtop/ssh:centos-6.4" - -boot2docker: - memory_size: "4196" - number_cpus: "2" - -repo: "http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.8.0/label=centos6/6/artifact/output/"; -components: "hadoop,yarn" -namenode_ui_port: "50070" -yarn_ui_port: "8088" -hbase_ui_port: "60010" http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/.gitignore ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/.gitignore b/bigtop-deploy/vm/vagrant-puppet-docker/.gitignore new file mode 100644 index 0000000..2cd389d --- /dev/null +++ b/bigtop-deploy/vm/vagrant-puppet-docker/.gitignore @@ -0,0 +1,3 @@ +config.rb +config/ +hosts http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/Dockerfile ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/Dockerfile b/bigtop-deploy/vm/vagrant-puppet-docker/Dockerfile new file mode 100644 index 0000000..8e65126 --- /dev/null +++ b/bigtop-deploy/vm/vagrant-puppet-docker/Dockerfile @@ -0,0 +1,33 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +FROM bigtop/seed:centos-6.4 + +# enable ssh +RUN yum -y install openssh-server openssh-clients sudo +RUN sed -i.bak s/UsePAM\ yes/UsePAM\ no/ /etc/ssh/sshd_config +RUN ssh-keygen -q -N "" -t dsa -f /etc/ssh/ssh_host_dsa_key +RUN ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key + +# requiretty off +RUN sed -i.bak 's/requiretty/!requiretty/' /etc/sudoers + +# setup vagrant account +RUN mkdir /root/.ssh/ +RUN chmod 0755 /root/.ssh +RUN wget http://github.com/mitchellh/vagrant/raw/master/keys/vagrant.pub --no-check-certificate -O /root/.ssh/authorized_keys +RUN chmod 0644 /root/.ssh/authorized_keys + +CMD /usr/sbin/sshd -D http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/README.md ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/README.md b/bigtop-deploy/vm/vagrant-puppet-docker/README.md new file mode 100644 index 0000000..6d609b6 --- /dev/null +++ b/bigtop-deploy/vm/vagrant-puppet-docker/README.md @@ -0,0 +1,179 @@ + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +------------------------------------------------------------------------------------------------------------------------------------------------------ + +#BigTop Docker provisioner + +## Overview + +The Vagrantfile definition and wrapper script that creates Bigtop virtual Hadoop cluster on top of Docker containers for you, by pulling from existing publishing bigtop repositories. +This cluster can be used: + +- to test bigtop smoke tests +- to test bigtop puppet recipes + +These containers start sshd daemons, which vagrant uses to provision and install the hadoop cluster. + +This has been verified on docker client 1.2.0, with api version 1.15, and vagrant 1.6.5 on Fedora 20 as well as Centos 6. + +## Prerequisites + +### OS X and Windows + +* Install [VirtualBox](https://www.virtualbox.org/wiki/Downloads) + +* Install [Vagrant](http://www.vagrantup.com/downloads.html). Need version 1.6.5 or higher. + +### Linux + +* [Kernel Requirements](http://docker.readthedocs.org/en/v0.5.3/installation/kernel/) + +* Install [Docker](https://docs.docker.com/installation/) + +* Install [Vagrant](http://www.vagrantup.com/downloads.html) + +## Getting Started + +* Create a 3 node Bigtop Hadoop cluster from scratch + +NOTE : SELinux can PREVENT you from ssh'ing into your docker container. +As a brute force way to disable it - remove it from vi /etc/sysconfig/docker arguments +(fedora and centos may by default launch docker daemon with the --selinux-enabled option)! +In the future, lets update this README with the RIGHT way to allow selinux without breaking +ssh into a docker container! + +``` +service docker restart +docker pull bigtop/seed:centos-6.4 +``` + +Now, you can start your cluster: + +``` +cd bigtop/bigtop-deploy/vm/vagrant-puppet-docker +./docker-hadoop.sh --build-image --create 3 +``` +In case of errors you can attempt running as root, or else, ping the mailing list. + +## USAGE + +1) Build up the base Docker image that supports Vagrant. + +``` +./docker-hadoop.sh --build-image +``` + +2) Create a Bigtop Hadoop cluster by given # of node. (will place a file called config.rb) + +``` +./docker-hadoop.sh --create 3 +``` + +3) Destroy the cluster. + +``` +./docker-hadoop.sh --destroy +``` + +4) Update your cluster after doing configuration changes. (re-run puppet apply) + +``` +./docker-hadoop.sh --provision +``` + +5) Run Bigtop smoke tests + +``` +./docker-hadoop.sh --smoke-tests +``` + +6) Chain your operations with-in one command. + +``` +./docker-hadoop.sh --build-image --create 5 --smoke-tests --destroy +``` + +Commands will be executed by following order: + +``` +build-image => create 5 node cluster => destroy the cluster +``` + +7) Run hbase-test.sh to evaluate the deployment. + +``` +../utils/hbase-test.sh +``` + +8) See helper message: + +``` +./docker-hadoop.sh -h +usage: docker-hadoop.sh [options] + -b, --build-image Build base Docker image for Bigtop Hadoop + (must be exectued at least once before creating cluster) + -c NUM_INSTANCES, --create=NUM_INSTANCES Create a docker based Bigtop Hadoop cluster + -p, --provision Deploy configuration changes + -s, --smoke-tests Run Bigtop smoke tests + -d, --destroy Destroy the cluster + -h, --help + +``` + +##Configurations + +* There are several parameters can be configured in the vagrantconfig.yaml: + +1) Modify memory limit for Docker containers + +``` +docker: + memory_size: "2048" + +``` + +2) If you're running Docker provisioner on OS X or Windows, you can customize the boot2docker VM settings + +``` +boot2docker: + memory_size: "4196" + number_cpus: "2" +``` + +3) Use different host ports mapping for web UIs + +``` +namenode_ui_port: "50070" +yarn_ui_port: "8088" +hbase_ui_port: "60010" + +``` +Note: If running on OS X or Windows, the boot2docker VM should be reloaded after ports changed + + +##Configure Apache Hadoop ecosystem components +* Choose the ecosystem you want to be deployed by modifying components in vagrantconfig.yaml + +``` +components: "hadoop,hbase,yarn,..." +``` + +By default, Apache Hadoop, YARN, and Apache HBase will be installed. +See `bigtop-deploy/puppet/config/site.csv.example` for more details. + +##Notes + +* Users currently using vagrant 1.6+ is strongly recommanded to upgrade to 1.6.4+, otherwise you will encounter the [issue](https://github.com/mitchellh/vagrant/issues/3769) when installing plguins http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile b/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile new file mode 100755 index 0000000..3ec023b --- /dev/null +++ b/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile @@ -0,0 +1,74 @@ +# -*- mode: ruby -*- +# vi: set ft=ruby : + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +require 'fileutils' + +# number of instances +$num_instances = 3 + +# load configuration from config.rb +CONFIG = File.join(File.dirname(__FILE__), "config.rb") +if File.exist?(CONFIG) + require CONFIG +end + +require "yaml" + +_config = YAML.load(File.open(File.join(File.dirname(__FILE__), "vagrantconfig.yaml"), File::RDONLY).read) +CONF = _config + +ENV['VAGRANT_DEFAULT_PROVIDER'] ||= 'docker' + +Vagrant.require_version ">= 1.6.0" + +VAGRANTFILE_API_VERSION = "2" +Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| + + # image definition + config.vm.define "image", autostart: false do |image| + image.vm.provider "docker" do |d| + d.build_dir = "." + d.build_args = ["-t",CONF["docker"]["image"]] + d.vagrant_machine = "boot2docker" + d.vagrant_vagrantfile = "./boot2docker/Vagrantfile" + end + image.ssh.username = "root" + image.ssh.port = 22 + end + + # nodes definition + (1..$num_instances).each do |i| + config.vm.define "bigtop#{i}" do |bigtop| + # docker container settings + bigtop.vm.provider "docker" do |d| + d.image = CONF["docker"]["image"] + d.create_args = ["--privileged=true", "-m", CONF["docker"]['memory_size'] + "m"] + if "#{i}" == "1" + d.ports = [CONF["namenode_ui_port"] + ":50070", CONF["yarn_ui_port"] + ":8088", CONF["hbase_ui_port"] + ":60010"] + end + d.vagrant_machine = "boot2docker" + d.vagrant_vagrantfile = "./boot2docker/Vagrantfile" + end + bigtop.ssh.username = "root" + bigtop.ssh.port = 22 + bigtop.vm.hostname = "bigtop#{i}.docker" + bigtop.vm.synced_folder "../../../", "/bigtop-home" + end + end + +end http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/boot2docker/Vagrantfile ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/boot2docker/Vagrantfile b/bigtop-deploy/vm/vagrant-puppet-docker/boot2docker/Vagrantfile new file mode 100644 index 0000000..5913fec --- /dev/null +++ b/bigtop-deploy/vm/vagrant-puppet-docker/boot2docker/Vagrantfile @@ -0,0 +1,24 @@ +# -*- mode: ruby -*- +# vi: set ft=ruby : + +require "yaml" + +_config = YAML.load(File.open(File.join(File.dirname(__FILE__), "../vagrantconfig.yaml"), File::RDONLY).read) +CONF = _config + +VAGRANTFILE_API_VERSION = "2" + +Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| + + config.vm.define "boot2docker" do |node| + node.vm.box = "yungsang/boot2docker" + node.vm.provider "virtualbox" do |vb| + vb.customize ["modifyvm", :id, "--memory", CONF["boot2docker"]["memory_size"]] + vb.customize ['modifyvm', :id, '--cpus', CONF["boot2docker"]['number_cpus']] + end + node.vm.network "forwarded_port", host: CONF["namenode_ui_port"], guest: 50070 + node.vm.network "forwarded_port", host: CONF["yarn_ui_port"], guest: 8088 + node.vm.network "forwarded_port", host: CONF["hbase_ui_port"], guest: 60010 + end + +end http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh b/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh new file mode 100755 index 0000000..4b504b3 --- /dev/null +++ b/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh @@ -0,0 +1,117 @@ +#!/bin/bash + +usage() { + echo "usage: $PROG [options]" + echo " -b, --build-image Build base Docker image for Bigtop Hadoop" + echo " (must be exectued at least once before creating cluster)" + echo " -c NUM_INSTANCES, --create=NUM_INSTANCES Create a Docker based Bigtop Hadoop cluster" + echo " -p, --provision Deploy configuration changes" + echo " -s, --smoke-tests Run Bigtop smoke tests" + echo " -d, --destroy Destroy the cluster" + echo " -h, --help" + exit 1 +} + +build-image() { + vagrant up image --provider docker + { + echo "echo -e '\nBUILD IMAGE SUCCESS.\n'" |vagrant ssh image + } || { + >&2 echo -e "\nBUILD IMAGE FAILED!\n" + exit 2 + } +} + +create() { + echo "\$num_instances = $1" > config.rb + vagrant up --no-parallel + nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`) + hadoop_head_node=(`echo "hostname -f" |vagrant ssh ${nodes[0]} |tail -n 1`) + repo=$(get-yaml-config repo) + components=$(get-yaml-config components) + echo "/bigtop-home/bigtop-deploy/vm/utils/setup-env.sh" |vagrant ssh ${nodes[0]} + echo "/vagrant/provision.sh $hadoop_head_node $repo $components" |vagrant ssh ${nodes[0]} + bigtop-puppet ${nodes[0]} + for ((i=1 ; i<${#nodes[*]} ; i++)); do + ( + echo "/bigtop-home/bigtop-deploy/vm/utils/setup-env.sh" |vagrant ssh ${nodes[$i]} + echo "/vagrant/provision.sh $hadoop_head_node $repo $components" |vagrant ssh ${nodes[$i]} + bigtop-puppet ${nodes[$i]} + ) & + done + wait +} + +provision() { + nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`) + for node in $nodes; do + bigtop-puppet $node & + done + wait +} + +smoke-tests() { + nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`) + echo "/bigtop-home/bigtop-deploy/vm/utils/smoke-tests.sh" |vagrant ssh ${nodes[0]} +} + + +destroy() { + rm -rf ./hosts ./config ./config.rb + nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`) + for node in $nodes; do + vagrant destroy -f $node + done + wait +} + +bigtop-puppet() { + echo "puppet apply -d --confdir=/vagrant --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules /bigtop-home/bigtop-deploy/puppet/manifests/site.pp" |vagrant ssh $1 +} + +get-yaml-config() { + RUBY_EXE=ruby + which ruby > /dev/null 2>&1 + if [ $? -ne 0 ]; then + # use vagrant embedded ruby on Windows + RUBY_EXE=$(dirname $(which vagrant))/../embedded/bin/ruby + fi + RUBY_SCRIPT="data = YAML::load(STDIN.read); puts data['$1'];" + cat vagrantconfig.yaml | $RUBY_EXE -ryaml -e "$RUBY_SCRIPT" | tr -d '\r' +} + +PROG=`basename $0` + +if [ $# -eq 0 ]; then + usage +fi + +while [ $# -gt 0 ]; do + case "$1" in + -b|--build-image) + build-image + shift;; + -c|--create) + if [ $# -lt 2 ]; then + echo "Create requires a number" 1>&2 + usage + fi + create $2 + shift 2;; + -p|--provision) + provision + shift;; + -s|--smoke-tests) + smoke-tests + shift;; + -d|--destroy) + destroy + shift;; + -h|--help) + usage + shift;; + *) + echo "Unknown argument: '$1'" 1>&2 + usage;; + esac +done http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/provision.sh ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/provision.sh b/bigtop-deploy/vm/vagrant-puppet-docker/provision.sh new file mode 100755 index 0000000..da087c6 --- /dev/null +++ b/bigtop-deploy/vm/vagrant-puppet-docker/provision.sh @@ -0,0 +1,36 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +sysctl kernel.hostname=`hostname -f` + +# Unmount device /etc/hosts and replace it by a shared hosts file +echo -e "`hostname -i`\t`hostname -f`" >> /vagrant/hosts +umount /etc/hosts +mv /etc/hosts /etc/hosts.bak +ln -s /vagrant/hosts /etc/hosts + +# Prepare puppet configuration file +cd /etc/puppet/modules && puppet module install puppetlabs/stdlib + +mkdir /vagrant/config +cat > /vagrant/config/site.csv << EOF +hadoop_head_node,$1 +hadoop_storage_dirs,/data/1,/data/2 +bigtop_yumrepo_uri,$2 +jdk_package_name,java-1.7.0-openjdk-devel.x86_64 +components,$3 +EOF http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-docker/vagrantconfig.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/vagrantconfig.yaml b/bigtop-deploy/vm/vagrant-puppet-docker/vagrantconfig.yaml new file mode 100644 index 0000000..6a53d02 --- /dev/null +++ b/bigtop-deploy/vm/vagrant-puppet-docker/vagrantconfig.yaml @@ -0,0 +1,13 @@ +docker: + memory_size: "2048" + image: "bigtop/ssh:centos-6.4" + +boot2docker: + memory_size: "4196" + number_cpus: "2" + +repo: "http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.8.0/label=centos6/6/artifact/output/"; +components: "hadoop,yarn" +namenode_ui_port: "50070" +yarn_ui_port: "8088" +hbase_ui_port: "60010" http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-vm/.gitignore ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-vm/.gitignore b/bigtop-deploy/vm/vagrant-puppet-vm/.gitignore new file mode 100644 index 0000000..04204c7 --- /dev/null +++ b/bigtop-deploy/vm/vagrant-puppet-vm/.gitignore @@ -0,0 +1 @@ +config http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-vm/README.md ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-vm/README.md b/bigtop-deploy/vm/vagrant-puppet-vm/README.md new file mode 100644 index 0000000..e1e3d48 --- /dev/null +++ b/bigtop-deploy/vm/vagrant-puppet-vm/README.md @@ -0,0 +1,92 @@ + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +------------------------------------------------------------------------------------------------------------------------------------------------------ + +#BigTop VM provisioner + +## Overview + +The Vagrantfile definition creates a bigtop virtual hadoop cluster for you, by pulling from existing publishing bigtop repositories. +This cluster can be used: + +- to test bigtop smoke tests +- to test bigtop puppet recipes + +Eventually, we may also add ability to build AND provision bigtop in a vagrant recipe, which would essentially +give full validation of the BigTop stack. + +## USAGE + +1) Install [vagrant-hostmanager plugin](https://github.com/smdahlen/vagrant-hostmanager) to better manage `/etc/hosts` + +``` +vagrant plugin install vagrant-hostmanager +``` + +2) Install [vagrant-cachier plugin](https://github.com/fgrehm/vagrant-cachier) to cache packages at local + +``` +vagrant plugin install vagrant-cachier +``` + +3) To provision a 3 node Apache Hadoop cluster on top of vagrant boxes + +``` +vagrant up +``` + +4) You can specify number of nodes you'd like to provision by modifying `num_instances` in vagrantconfig.yaml + +``` +num_instances: 5 +``` + +5) Test on local built packages is available by: + +first, build up local yum repo + +``` +cd bigtop; ./gradlew tachyon-yum +``` + +and then enable local yum in vagrantconfig.yaml + +``` +enable_local_yum: true +``` + +finally, spin up the cluster +``` +vagrant up +``` + +##Configure Apache Hadoop ecosystem components +* Choose the ecosystem you want to be deployed by modifying components in vagrantconfig.yaml + +``` +components: "hadoop,hbase,yarn,..." +``` + +By default, Apache Hadoop, YARN, and Apache HBase will be installed. +See `bigtop-deploy/puppet/config/site.csv.example` for more details. + +##Example: + +5) Run hbase-test.sh to evaluate the deployment. + +``` +../utils/hbase-test.sh +``` http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-vm/Vagrantfile ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-vm/Vagrantfile b/bigtop-deploy/vm/vagrant-puppet-vm/Vagrantfile new file mode 100755 index 0000000..2f31055 --- /dev/null +++ b/bigtop-deploy/vm/vagrant-puppet-vm/Vagrantfile @@ -0,0 +1,123 @@ +# -*- mode: ruby -*- +# vi: set ft=ruby : + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +require "yaml" + +_config = YAML.load(File.open(File.join(File.dirname(__FILE__), "vagrantconfig.yaml"), File::RDONLY).read) +CONF = _config + +# Repository +# Example for testing a Release candidate. +repo = CONF['repo'] +# repo = "http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64"; + +# number of instances +num_instances = CONF['num_instances'] + +# hadoop ecosystem components +components = CONF['components'] + +# Whether to run smoke tests +run_smoke_tests = CONF['run_smoke_tests'] + +# master node hostname +bigtop_master = "bigtop1.vagrant" + +$script = <<SCRIPT +service iptables stop +chkconfig iptables off +cat /dev/null > /etc/hosts +echo "Bigtop yum repo = #{repo}" +# Prepare puppet configuration file +mkdir /vagrant/config +cat > /vagrant/config/site.csv << EOF +hadoop_head_node,#{bigtop_master} +hadoop_storage_dirs,/data/1,/data/2 +bigtop_yumrepo_uri,#{repo} +jdk_package_name,java-1.7.0-openjdk-devel.x86_64 +components,#{components} +EOF +SCRIPT + +VAGRANTFILE_API_VERSION = "2" +Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| + # manage /etc/hosts by hostmanager plugin(https://github.com/smdahlen/vagrant-hostmanager) + # use vagrant-cachier to cache packages at local(https://github.com/fgrehm/vagrant-cachier) + config.hostmanager.enabled = true + + # use vagrant-cachier to cache packages at local(https://github.com/fgrehm/vagrant-cachier) + if Vagrant.has_plugin?("vagrant-cachier") + config.cache.scope = :box + end + + # nodes definition + (1..num_instances).each do |i| + config.vm.define "bigtop#{i}" do |bigtop| + + bigtop.vm.box = CONF['box'] + bigtop_hostname="bigtop#{i}.vagrant" + bigtop_ip="10.10.10.1#{i}" + + bigtop.vm.provider :virtualbox do |vb| + vb.customize ["modifyvm", :id, "--memory", CONF['memory_size']] + vb.customize ['modifyvm', :id, '--cpus', CONF['number_cpus']] + end + + bigtop.vm.network :private_network, ip: bigtop_ip + bigtop.vm.hostname = bigtop_hostname + + # three levels up is the bigtop "home" directory. + # the current directory has puppet recipes which we need for provisioning. + bigtop.vm.synced_folder "../../../", "/bigtop-home" + + # We also add the bigtop-home output/ dir, so that locally built rpms will be available. + puts "Adding rpms from /output/ for local testing." + + # This is a update to allow dev packages + # Force success - not worried if this step fails, since we generally only use it for development. + if CONF['enable_local_yum'] then + bigtop.vm.provision "shell", inline: "sudo echo \"gpgcheck=0\" >> /etc/yum.conf" + bigtop.vm.provision "shell", inline: "sudo yum-config-manager --add-repo file:///bigtop-home/output ; true" + end + + # carry on w/ installation + bigtop.vm.provision "shell", path: "../utils/setup-env.sh" + bigtop.vm.provision "shell", inline: $script + bigtop.vm.provision :hostmanager + + # deploy Apache Hadoop and it's ecosystem + bigtop.vm.provision :puppet do |puppet| + puppet.module_path = "../../puppet/modules/" + puppet.manifests_path = "../../puppet/manifests/" + puppet.manifest_file = "site.pp" + puppet.options = "--debug --confdir=/vagrant" + end + + if run_smoke_tests then + if i==num_instances then + puts "creating provisioner directive for running tests" + bigtop.vm.provision "shell", path: "../utils/smoke-tests.sh" + else + puts "Not creating provisioner directive for tests yet... only on vm #{i} of #{num_instances}" + end + end + + end + end + +end http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet-vm/vagrantconfig.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-vm/vagrantconfig.yaml b/bigtop-deploy/vm/vagrant-puppet-vm/vagrantconfig.yaml new file mode 100644 index 0000000..7f79acc --- /dev/null +++ b/bigtop-deploy/vm/vagrant-puppet-vm/vagrantconfig.yaml @@ -0,0 +1,8 @@ +memory_size: 4096 +number_cpus: 1 +box: "puppetlabs/centos-6.5-64-nocm" +repo: "http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.8.0/label=centos6/6/artifact/output/"; +num_instances: 1 +components: "hadoop,yarn" +enable_local_yum: false +run_smoke_tests: false http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet/.gitignore ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet/.gitignore b/bigtop-deploy/vm/vagrant-puppet/.gitignore deleted file mode 100644 index 04204c7..0000000 --- a/bigtop-deploy/vm/vagrant-puppet/.gitignore +++ /dev/null @@ -1 +0,0 @@ -config http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet/README.md ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet/README.md b/bigtop-deploy/vm/vagrant-puppet/README.md deleted file mode 100644 index e1e3d48..0000000 --- a/bigtop-deploy/vm/vagrant-puppet/README.md +++ /dev/null @@ -1,92 +0,0 @@ - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - ------------------------------------------------------------------------------------------------------------------------------------------------------- - -#BigTop VM provisioner - -## Overview - -The Vagrantfile definition creates a bigtop virtual hadoop cluster for you, by pulling from existing publishing bigtop repositories. -This cluster can be used: - -- to test bigtop smoke tests -- to test bigtop puppet recipes - -Eventually, we may also add ability to build AND provision bigtop in a vagrant recipe, which would essentially -give full validation of the BigTop stack. - -## USAGE - -1) Install [vagrant-hostmanager plugin](https://github.com/smdahlen/vagrant-hostmanager) to better manage `/etc/hosts` - -``` -vagrant plugin install vagrant-hostmanager -``` - -2) Install [vagrant-cachier plugin](https://github.com/fgrehm/vagrant-cachier) to cache packages at local - -``` -vagrant plugin install vagrant-cachier -``` - -3) To provision a 3 node Apache Hadoop cluster on top of vagrant boxes - -``` -vagrant up -``` - -4) You can specify number of nodes you'd like to provision by modifying `num_instances` in vagrantconfig.yaml - -``` -num_instances: 5 -``` - -5) Test on local built packages is available by: - -first, build up local yum repo - -``` -cd bigtop; ./gradlew tachyon-yum -``` - -and then enable local yum in vagrantconfig.yaml - -``` -enable_local_yum: true -``` - -finally, spin up the cluster -``` -vagrant up -``` - -##Configure Apache Hadoop ecosystem components -* Choose the ecosystem you want to be deployed by modifying components in vagrantconfig.yaml - -``` -components: "hadoop,hbase,yarn,..." -``` - -By default, Apache Hadoop, YARN, and Apache HBase will be installed. -See `bigtop-deploy/puppet/config/site.csv.example` for more details. - -##Example: - -5) Run hbase-test.sh to evaluate the deployment. - -``` -../utils/hbase-test.sh -``` http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet/Vagrantfile ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet/Vagrantfile b/bigtop-deploy/vm/vagrant-puppet/Vagrantfile deleted file mode 100755 index 2f31055..0000000 --- a/bigtop-deploy/vm/vagrant-puppet/Vagrantfile +++ /dev/null @@ -1,123 +0,0 @@ -# -*- mode: ruby -*- -# vi: set ft=ruby : - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -require "yaml" - -_config = YAML.load(File.open(File.join(File.dirname(__FILE__), "vagrantconfig.yaml"), File::RDONLY).read) -CONF = _config - -# Repository -# Example for testing a Release candidate. -repo = CONF['repo'] -# repo = "http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64"; - -# number of instances -num_instances = CONF['num_instances'] - -# hadoop ecosystem components -components = CONF['components'] - -# Whether to run smoke tests -run_smoke_tests = CONF['run_smoke_tests'] - -# master node hostname -bigtop_master = "bigtop1.vagrant" - -$script = <<SCRIPT -service iptables stop -chkconfig iptables off -cat /dev/null > /etc/hosts -echo "Bigtop yum repo = #{repo}" -# Prepare puppet configuration file -mkdir /vagrant/config -cat > /vagrant/config/site.csv << EOF -hadoop_head_node,#{bigtop_master} -hadoop_storage_dirs,/data/1,/data/2 -bigtop_yumrepo_uri,#{repo} -jdk_package_name,java-1.7.0-openjdk-devel.x86_64 -components,#{components} -EOF -SCRIPT - -VAGRANTFILE_API_VERSION = "2" -Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| - # manage /etc/hosts by hostmanager plugin(https://github.com/smdahlen/vagrant-hostmanager) - # use vagrant-cachier to cache packages at local(https://github.com/fgrehm/vagrant-cachier) - config.hostmanager.enabled = true - - # use vagrant-cachier to cache packages at local(https://github.com/fgrehm/vagrant-cachier) - if Vagrant.has_plugin?("vagrant-cachier") - config.cache.scope = :box - end - - # nodes definition - (1..num_instances).each do |i| - config.vm.define "bigtop#{i}" do |bigtop| - - bigtop.vm.box = CONF['box'] - bigtop_hostname="bigtop#{i}.vagrant" - bigtop_ip="10.10.10.1#{i}" - - bigtop.vm.provider :virtualbox do |vb| - vb.customize ["modifyvm", :id, "--memory", CONF['memory_size']] - vb.customize ['modifyvm', :id, '--cpus', CONF['number_cpus']] - end - - bigtop.vm.network :private_network, ip: bigtop_ip - bigtop.vm.hostname = bigtop_hostname - - # three levels up is the bigtop "home" directory. - # the current directory has puppet recipes which we need for provisioning. - bigtop.vm.synced_folder "../../../", "/bigtop-home" - - # We also add the bigtop-home output/ dir, so that locally built rpms will be available. - puts "Adding rpms from /output/ for local testing." - - # This is a update to allow dev packages - # Force success - not worried if this step fails, since we generally only use it for development. - if CONF['enable_local_yum'] then - bigtop.vm.provision "shell", inline: "sudo echo \"gpgcheck=0\" >> /etc/yum.conf" - bigtop.vm.provision "shell", inline: "sudo yum-config-manager --add-repo file:///bigtop-home/output ; true" - end - - # carry on w/ installation - bigtop.vm.provision "shell", path: "../utils/setup-env.sh" - bigtop.vm.provision "shell", inline: $script - bigtop.vm.provision :hostmanager - - # deploy Apache Hadoop and it's ecosystem - bigtop.vm.provision :puppet do |puppet| - puppet.module_path = "../../puppet/modules/" - puppet.manifests_path = "../../puppet/manifests/" - puppet.manifest_file = "site.pp" - puppet.options = "--debug --confdir=/vagrant" - end - - if run_smoke_tests then - if i==num_instances then - puts "creating provisioner directive for running tests" - bigtop.vm.provision "shell", path: "../utils/smoke-tests.sh" - else - puts "Not creating provisioner directive for tests yet... only on vm #{i} of #{num_instances}" - end - end - - end - end - -end http://git-wip-us.apache.org/repos/asf/bigtop/blob/d8a50657/bigtop-deploy/vm/vagrant-puppet/vagrantconfig.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet/vagrantconfig.yaml b/bigtop-deploy/vm/vagrant-puppet/vagrantconfig.yaml deleted file mode 100644 index 7f79acc..0000000 --- a/bigtop-deploy/vm/vagrant-puppet/vagrantconfig.yaml +++ /dev/null @@ -1,8 +0,0 @@ -memory_size: 4096 -number_cpus: 1 -box: "puppetlabs/centos-6.5-64-nocm" -repo: "http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.8.0/label=centos6/6/artifact/output/"; -num_instances: 1 -components: "hadoop,yarn" -enable_local_yum: false -run_smoke_tests: false
