Repository: bigtop Updated Branches: refs/heads/master 7c716c644 -> 3298063c6
BIGTOP-1178. Clusterize the puppetized vagrant deployer. Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/3298063c Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/3298063c Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/3298063c Branch: refs/heads/master Commit: 3298063c68ffb89f829f23f8c88f3270110749ec Parents: 7c716c6 Author: Evans Ye <innovansye.gmail.com> Authored: Tue Apr 8 13:11:46 2014 -0700 Committer: Konstantin Boudnik <[email protected]> Committed: Tue Apr 8 13:11:46 2014 -0700 ---------------------------------------------------------------------- bigtop-deploy/vm/vagrant-puppet/README | 16 ---- bigtop-deploy/vm/vagrant-puppet/README.md | 52 ++++++++++ bigtop-deploy/vm/vagrant-puppet/Vagrantfile | 105 ++++++++++++++++++++- bigtop-deploy/vm/vagrant-puppet/hbase-test.sh | 17 +++- bigtop-deploy/vm/vagrant-puppet/provision.sh | 21 +++++ bigtop-deploy/vm/vagrant-puppet/startup.sh | 40 ++++++++ 6 files changed, 229 insertions(+), 22 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/bigtop/blob/3298063c/bigtop-deploy/vm/vagrant-puppet/README ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet/README b/bigtop-deploy/vm/vagrant-puppet/README deleted file mode 100644 index 819ce0b..0000000 --- a/bigtop-deploy/vm/vagrant-puppet/README +++ /dev/null @@ -1,16 +0,0 @@ -Set up a vagrant box with bigtop ecosystem deployed by puppet - -Usage: - -1) Choose the ecosystem you want to be deployed by modify the components in provision.sh: - - components,hadoop,hbase,... - - By default, hadoop and hbase will be installed. - See bigtop-deploy/puppet/config/site.csv.example for more details. - - -2) vagrant up - - -3) Run hbase-test.sh to evaluate the deployment. http://git-wip-us.apache.org/repos/asf/bigtop/blob/3298063c/bigtop-deploy/vm/vagrant-puppet/README.md ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet/README.md b/bigtop-deploy/vm/vagrant-puppet/README.md new file mode 100644 index 0000000..ce7fc8e --- /dev/null +++ b/bigtop-deploy/vm/vagrant-puppet/README.md @@ -0,0 +1,52 @@ + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +------------------------------------------------------------------------------------------------------------------------------------------------------ + +#Deploy Apache Hadoop VM(s) on top of vagrant boxes using puppet + +##Preparation +* You need to install [vagrant-hostmanager plugin](https://github.com/smdahlen/vagrant-hostmanager) to better manage `/etc/hosts` +<pre> +$ vagrant plugin install vagrant-hostmanager +</pre> + +##Usage + +* To provision a 3 node Apache Hadoop cluster on top of vagrant boxes +<pre> +$ ./startup.sh --cluster +</pre> + +* See options with -h specified +<pre> +$ ./startup.sh -h +usage: startup.sh [options] + -s, --standalone deploy a standalone hadoop vm + -c, --cluster deploy a 3 node hadoop cluster + -h, --help +</pre> + +* Run hbase-test.sh to evaluate the deployment. + +##Configure Apache Hadoop ecosystem components +* Choose the ecosystem you want to be deployed by modify components in provision.sh: + +<pre> + components,hadoop,hbase,... +</pre> + + By default, Apache Hadoop and Apache HBase will be installed. + See `bigtop-deploy/puppet/config/site.csv.example` for more details. http://git-wip-us.apache.org/repos/asf/bigtop/blob/3298063c/bigtop-deploy/vm/vagrant-puppet/Vagrantfile ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet/Vagrantfile b/bigtop-deploy/vm/vagrant-puppet/Vagrantfile index d852253..b19c2fa 100755 --- a/bigtop-deploy/vm/vagrant-puppet/Vagrantfile +++ b/bigtop-deploy/vm/vagrant-puppet/Vagrantfile @@ -1,6 +1,21 @@ # -*- mode: ruby -*- # vi: set ft=ruby : +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + VAGRANTFILE_API_VERSION = "2" Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| vmbox = "puppetlab-centos-64-nocm" @@ -8,11 +23,21 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| # Virtual Box implementation vmboxurl = "http://puppet-vagrant-boxes.puppetlabs.com/centos-64-x64-vbox4210-nocm.box"; - hostname = "vagrant.bigtop1" + # Network environment settings + bigtop1_hostname = "bigtop1.vagrant" + bigtop1_ip = "10.10.10.12" + bigtop2_hostname = "bigtop2.vagrant" + bigtop2_ip = "10.10.10.13" + bigtop3_hostname = "bigtop3.vagrant" + bigtop3_ip = "10.10.10.14" # The bigtop puppet recipes bigtop_puppet = "../../puppet" + # manage /etc/hosts by hostmanager plugin(https://github.com/smdahlen/vagrant-hostmanager) + config.hostmanager.enabled = true + + # First node in the cluster config.vm.define :bigtop1 do |bigtop1| @@ -23,18 +48,20 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| vb.customize ["modifyvm", :id, "--memory", "1024"] end - bigtop1.vm.network :private_network, ip: "10.10.10.12" - bigtop1.vm.hostname = hostname + bigtop1.vm.network :private_network, ip: bigtop1_ip + bigtop1.vm.hostname = bigtop1_hostname bigtop1.vm.synced_folder bigtop_puppet, "/bigtop-puppet" # Shell provisioner to setup basic vm environment bigtop1.vm.provision :shell do |shell| shell.path = "provision.sh" - shell.args = hostname + shell.args = bigtop1_hostname + " " + bigtop1_ip + " " + bigtop1_hostname + " " + bigtop1_ip end - # Deploy hadoop and it's ecosystem + bigtop1.vm.provision :hostmanager + + # Deploy Apache Hadoop and it's ecosystem bigtop1.vm.provision :puppet do |puppet| puppet.module_path = bigtop_puppet + "/modules" puppet.manifests_path = bigtop_puppet + "/manifests" @@ -44,4 +71,72 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| end + + # Second node in the cluster + config.vm.define :bigtop2 do |bigtop2| + + bigtop2.vm.box = vmbox + bigtop2.vm.box_url = vmboxurl + + bigtop2.vm.provider :virtualbox do |vb| + vb.customize ["modifyvm", :id, "--memory", "1024"] + end + + bigtop2.vm.network :private_network, ip: bigtop2_ip + bigtop2.vm.hostname = bigtop2_hostname + + bigtop2.vm.synced_folder bigtop_puppet, "/bigtop-puppet" + + # Shell provisioner to setup basic vm environment + bigtop2.vm.provision :shell do |shell| + shell.path = "provision.sh" + shell.args = bigtop1_hostname + " " + bigtop1_ip + " " + bigtop2_hostname + " " + bigtop2_ip + end + + bigtop2.vm.provision :hostmanager + + # Deploy Apache Hadoop and it's ecosystem + bigtop2.vm.provision :puppet do |puppet| + puppet.module_path = bigtop_puppet + "/modules" + puppet.manifests_path = bigtop_puppet + "/manifests" + puppet.manifest_file = "site.pp" + puppet.options = "--debug --confdir=/bigtop-puppet" + end + + end + + + # Third node in the cluster + config.vm.define :bigtop3 do |bigtop3| + + bigtop3.vm.box = vmbox + bigtop3.vm.box_url = vmboxurl + + bigtop3.vm.provider :virtualbox do |vb| + vb.customize ["modifyvm", :id, "--memory", "1024"] + end + + bigtop3.vm.network :private_network, ip: bigtop3_ip + bigtop3.vm.hostname = bigtop3_hostname + + bigtop3.vm.synced_folder bigtop_puppet, "/bigtop-puppet" + + # Shell provisioner to setup basic vm environment + bigtop3.vm.provision :shell do |shell| + shell.path = "provision.sh" + shell.args = bigtop1_hostname + " " + bigtop1_ip + " " + bigtop3_hostname + " " + bigtop3_ip + end + + bigtop3.vm.provision :hostmanager + + # Deploy Apache Hadoop and it's ecosystem + bigtop3.vm.provision :puppet do |puppet| + puppet.module_path = bigtop_puppet + "/modules" + puppet.manifests_path = bigtop_puppet + "/manifests" + puppet.manifest_file = "site.pp" + puppet.options = "--debug --confdir=/bigtop-puppet" + end + + end + end http://git-wip-us.apache.org/repos/asf/bigtop/blob/3298063c/bigtop-deploy/vm/vagrant-puppet/hbase-test.sh ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet/hbase-test.sh b/bigtop-deploy/vm/vagrant-puppet/hbase-test.sh index c4820b8..89b2b76 100755 --- a/bigtop-deploy/vm/vagrant-puppet/hbase-test.sh +++ b/bigtop-deploy/vm/vagrant-puppet/hbase-test.sh @@ -1,6 +1,21 @@ #!/bin/bash -vagrant ssh -c "hbase shell <<EOF +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +vagrant ssh bigtop1 -c "hbase shell <<EOF create 't1','cf1' put 't1', 'row1', 'cf1:q1', 'value1' scan 't1' http://git-wip-us.apache.org/repos/asf/bigtop/blob/3298063c/bigtop-deploy/vm/vagrant-puppet/provision.sh ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet/provision.sh b/bigtop-deploy/vm/vagrant-puppet/provision.sh index 6219470..92c58f3 100755 --- a/bigtop-deploy/vm/vagrant-puppet/provision.sh +++ b/bigtop-deploy/vm/vagrant-puppet/provision.sh @@ -1,7 +1,28 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + # Install puppet agent yum -y install http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm yum -y install puppet-2.7.23-1.el6.noarch +service iptables stop +chkconfig iptables off +cat /dev/null > /etc/hosts + # Prepare puppet configuration file cat > /bigtop-puppet/config/site.csv << EOF hadoop_head_node,$1 http://git-wip-us.apache.org/repos/asf/bigtop/blob/3298063c/bigtop-deploy/vm/vagrant-puppet/startup.sh ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet/startup.sh b/bigtop-deploy/vm/vagrant-puppet/startup.sh new file mode 100644 index 0000000..34b52ad --- /dev/null +++ b/bigtop-deploy/vm/vagrant-puppet/startup.sh @@ -0,0 +1,40 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +usage() { + echo "usage: `basename $0` [options]" + echo " -s, --standalone deploy a standalone hadoop VM" + echo " -c, --cluster deploy a 3 node hadoop cluster" + echo " -h, --help" + exit 1 +} + +case "$1" in +-s|--standalone) + vagrant up bigtop1 + shift;; +-c|--cluster) + vagrant up --provision-with shell && vagrant provision --provision-with hostmanager,puppet + shift;; +-h|--help) + usage + shift;; +"") + usage + shift + break;; +esac
