Repository: bigtop Updated Branches: refs/heads/master 8bea68ff0 -> 755896bbd
BIGTOP-1908. Move bigtop-deploy's dockerfiles into a centralized docker place Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/755896bb Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/755896bb Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/755896bb Branch: refs/heads/master Commit: 755896bbd9913608deb414564ecb961ddf2619d0 Parents: 8bea68f Author: Evans Ye <[email protected]> Authored: Tue Sep 22 18:12:59 2015 +0000 Committer: Evans Ye <[email protected]> Committed: Sat Oct 24 02:40:10 2015 +0800 ---------------------------------------------------------------------- .../vm/vagrant-puppet-docker/README.md | 39 +++++--------------- .../vm/vagrant-puppet-docker/Vagrantfile | 13 ------- .../vm/vagrant-puppet-docker/centos/Dockerfile | 33 ----------------- .../vm/vagrant-puppet-docker/debian/Dockerfile | 33 ----------------- .../vm/vagrant-puppet-docker/docker-hadoop.sh | 29 +++------------ docker/bigtop-deploy/centos-6/Dockerfile | 34 +++++++++++++++++ docker/bigtop-deploy/centos-6/build.sh | 18 +++++++++ docker/bigtop-deploy/debian-8/Dockerfile | 34 +++++++++++++++++ docker/bigtop-deploy/debian-8/build.sh | 18 +++++++++ 9 files changed, 119 insertions(+), 132 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/bigtop-deploy/vm/vagrant-puppet-docker/README.md ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/README.md b/bigtop-deploy/vm/vagrant-puppet-docker/README.md index 6d609b6..79e50f7 100644 --- a/bigtop-deploy/vm/vagrant-puppet-docker/README.md +++ b/bigtop-deploy/vm/vagrant-puppet-docker/README.md @@ -57,74 +57,55 @@ ssh into a docker container! ``` service docker restart -docker pull bigtop/seed:centos-6.4 +docker pull bigtop/deploy:centos-6 ``` - Now, you can start your cluster: -``` -cd bigtop/bigtop-deploy/vm/vagrant-puppet-docker -./docker-hadoop.sh --build-image --create 3 -``` In case of errors you can attempt running as root, or else, ping the mailing list. ## USAGE -1) Build up the base Docker image that supports Vagrant. - -``` -./docker-hadoop.sh --build-image -``` - -2) Create a Bigtop Hadoop cluster by given # of node. (will place a file called config.rb) +1) Create a Bigtop Hadoop cluster by given # of node. (will place a file called config.rb) ``` ./docker-hadoop.sh --create 3 ``` -3) Destroy the cluster. +2) Destroy the cluster. ``` ./docker-hadoop.sh --destroy ``` -4) Update your cluster after doing configuration changes. (re-run puppet apply) +3) Update your cluster after doing configuration changes. (re-run puppet apply) ``` ./docker-hadoop.sh --provision ``` -5) Run Bigtop smoke tests +4) Run Bigtop smoke tests ``` ./docker-hadoop.sh --smoke-tests ``` -6) Chain your operations with-in one command. +5) Chain your operations with-in one command. ``` -./docker-hadoop.sh --build-image --create 5 --smoke-tests --destroy +./docker-hadoop.sh --create 5 --smoke-tests --destroy ``` Commands will be executed by following order: ``` -build-image => create 5 node cluster => destroy the cluster -``` - -7) Run hbase-test.sh to evaluate the deployment. - -``` -../utils/hbase-test.sh +create 5 node cluster => run smoke tests => destroy the cluster ``` -8) See helper message: +6) See helper message: ``` ./docker-hadoop.sh -h usage: docker-hadoop.sh [options] - -b, --build-image Build base Docker image for Bigtop Hadoop - (must be exectued at least once before creating cluster) -c NUM_INSTANCES, --create=NUM_INSTANCES Create a docker based Bigtop Hadoop cluster -p, --provision Deploy configuration changes -s, --smoke-tests Run Bigtop smoke tests @@ -149,7 +130,7 @@ docker: ``` boot2docker: - memory_size: "4196" + memory_size: "4096" number_cpus: "2" ``` http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile b/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile index cc910b8..4927501 100755 --- a/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile +++ b/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile @@ -40,19 +40,6 @@ Vagrant.require_version ">= 1.6.0" VAGRANTFILE_API_VERSION = "2" Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| - # image definition - config.vm.define "image", autostart: false do |image| - image.vm.provider "docker" do |d| - d.build_dir = CONF["distro"] - d.build_args = ["-t",CONF["docker"]["image"]] - d.vagrant_machine = "boot2docker" - d.vagrant_vagrantfile = "./boot2docker/Vagrantfile" - d.has_ssh = false # We have decided to pipe commands to docker instead of ssh, due to ssh hang issue in vagrant. - end - image.ssh.username = "root" - image.ssh.port = 22 - end - # nodes definition (1..$num_instances).each do |i| config.vm.define "bigtop#{i}" do |bigtop| http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/bigtop-deploy/vm/vagrant-puppet-docker/centos/Dockerfile ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/centos/Dockerfile b/bigtop-deploy/vm/vagrant-puppet-docker/centos/Dockerfile deleted file mode 100644 index d5d1f09..0000000 --- a/bigtop-deploy/vm/vagrant-puppet-docker/centos/Dockerfile +++ /dev/null @@ -1,33 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -FROM bigtop/seed:centos-6.4 - -# enable ssh -RUN yum -y install openssh-server openssh-clients sudo -RUN sed -i.bak s/UsePAM\ yes/UsePAM\ no/ /etc/ssh/sshd_config -RUN ssh-keygen -q -N "" -t dsa -f /etc/ssh/ssh_host_dsa_key -RUN ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key - -# requiretty off -RUN sed -i.bak 's/requiretty/!requiretty/' /etc/sudoers - -# setup vagrant account -RUN mkdir /root/.ssh/ -RUN chmod 0750 /root/.ssh -RUN wget http://github.com/mitchellh/vagrant/raw/master/keys/vagrant.pub --no-check-certificate -O /root/.ssh/authorized_keys -RUN chmod 0640 /root/.ssh/authorized_keys - -CMD /usr/sbin/sshd -D http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/bigtop-deploy/vm/vagrant-puppet-docker/debian/Dockerfile ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/debian/Dockerfile b/bigtop-deploy/vm/vagrant-puppet-docker/debian/Dockerfile deleted file mode 100644 index 2e089c3..0000000 --- a/bigtop-deploy/vm/vagrant-puppet-docker/debian/Dockerfile +++ /dev/null @@ -1,33 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -FROM debian:jessie - -# enable ssh - -RUN apt-get update -RUN apt-get install -y openssh-server sudo wget - -# requiretty off -RUN sed -i.bak 's/requiretty/!requiretty/' /etc/sudoers - -# setup vagrant account -RUN mkdir -p /root/.ssh -RUN chmod 0750 /root/.ssh -RUN wget http://github.com/mitchellh/vagrant/raw/master/keys/vagrant.pub --no-check-certificate -O /root/.ssh/authorized_keys -RUN chmod 0640 /root/.ssh/authorized_keys - -RUN mkdir -p /var/run/sshd -CMD /usr/sbin/sshd -D http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh b/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh index 626a408..9e30521 100755 --- a/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh +++ b/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh @@ -1,4 +1,5 @@ #!/bin/bash + # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -18,8 +19,6 @@ usage() { echo "usage: $PROG [-C file ] args" echo " -C file Use alternate file for vagrantconfig.yaml" echo " commands:" - echo " -b, --build-image Build base Docker image for Bigtop Hadoop" - echo " (must be exectued at least once before creating cluster)" echo " -c NUM_INSTANCES, --create=NUM_INSTANCES Create a Docker based Bigtop Hadoop cluster" echo " -p, --provision Deploy configuration changes" echo " -s, --smoke-tests Run Bigtop smoke tests" @@ -28,17 +27,6 @@ usage() { exit 1 } -build-image() { - echo "\$vagrantyamlconf = \"$vagrantyamlconf\"" > config.rb - vagrant up image --provider docker - { - echo "echo -e '\nBUILD IMAGE SUCCESS.\n'" |vagrant ssh image - } || { - >&2 echo -e "\nBUILD IMAGE FAILED!\n" - exit 2 - } -} - create() { echo "\$num_instances = $1" > config.rb echo "\$vagrantyamlconf = \"$vagrantyamlconf\"" >> config.rb @@ -47,7 +35,7 @@ create() { echo "Docker container(s) startup failed!"; exit 1; fi - nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`) + nodes=(`vagrant status |grep bigtop |awk '{print $1}'`) hadoop_head_node=(`echo "hostname -f" |vagrant ssh ${nodes[0]} |tail -n 1`) repo=$(get-yaml-config repo) components="[`echo $(get-yaml-config components) | sed 's/ /, /g'`]" @@ -73,7 +61,7 @@ create() { } provision() { - nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`) + nodes=(`vagrant status |grep bigtop |awk '{print $1}'`) for node in ${nodes[*]}; do bigtop-puppet $node & done @@ -81,19 +69,15 @@ provision() { } smoke-tests() { - nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`) + nodes=(`vagrant status |grep bigtop |awk '{print $1}'`) smoke_test_components="`echo $(get-yaml-config smoke_test_components) | sed 's/ /,/g'`" echo "/bigtop-home/bigtop-deploy/vm/utils/smoke-tests.sh \"$smoke_test_components\"" |vagrant ssh ${nodes[0]} } destroy() { - nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`) + vagrant destroy -f rm -rvf ./hosts ./config.rb - for node in ${nodes[*]}; do - vagrant destroy -f $node - done - wait } bigtop-puppet() { @@ -120,9 +104,6 @@ fi vagrantyamlconf="vagrantconfig.yaml" while [ $# -gt 0 ]; do case "$1" in - -b|--build-image) - build-image - shift;; -c|--create) if [ $# -lt 2 ]; then echo "Create requires a number" 1>&2 http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/docker/bigtop-deploy/centos-6/Dockerfile ---------------------------------------------------------------------- diff --git a/docker/bigtop-deploy/centos-6/Dockerfile b/docker/bigtop-deploy/centos-6/Dockerfile new file mode 100644 index 0000000..823dfc3 --- /dev/null +++ b/docker/bigtop-deploy/centos-6/Dockerfile @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +FROM bigtop/puppet:centos-6 + +COPY bigtop_toolchain /etc/puppet/modules/bigtop_toolchain + +# enable ssh +RUN yum -y install openssh-server openssh-clients vim && \ + ssh-keygen -q -N "" -t dsa -f /etc/ssh/ssh_host_dsa_key && \ + ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key && \ + # requiretty off + sed -i.bak 's/requiretty/!requiretty/' /etc/sudoers && \ + # setup vagrant account + mkdir /root/.ssh && \ + chmod 0700 /root/.ssh && \ + wget http://github.com/mitchellh/vagrant/raw/master/keys/vagrant.pub --no-check-certificate -O /root/.ssh/authorized_keys && \ + chmod 0600 /root/.ssh/authorized_keys && \ + # Install required Puppet modules + puppet apply -e "include bigtop_toolchain::puppet-modules" + +CMD /usr/sbin/sshd -D http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/docker/bigtop-deploy/centos-6/build.sh ---------------------------------------------------------------------- diff --git a/docker/bigtop-deploy/centos-6/build.sh b/docker/bigtop-deploy/centos-6/build.sh new file mode 100755 index 0000000..c9a22c8 --- /dev/null +++ b/docker/bigtop-deploy/centos-6/build.sh @@ -0,0 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +cp -r ../../../bigtop_toolchain . +docker build -t bigtop/deploy:centos-6 . +rm -rf bigtop_toolchain http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/docker/bigtop-deploy/debian-8/Dockerfile ---------------------------------------------------------------------- diff --git a/docker/bigtop-deploy/debian-8/Dockerfile b/docker/bigtop-deploy/debian-8/Dockerfile new file mode 100644 index 0000000..bd0e8d3 --- /dev/null +++ b/docker/bigtop-deploy/debian-8/Dockerfile @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +FROM bigtop/puppet:debian-8 + +COPY bigtop_toolchain /etc/puppet/modules/bigtop_toolchain + +# enable ssh +RUN apt-get update && \ + apt-get install -y openssh-server vim && \ + mkdir -p /var/run/sshd && \ + # requiretty off + sed -i.bak 's/requiretty/!requiretty/' /etc/sudoers && \ + # setup vagrant account + mkdir -p /root/.ssh && \ + chmod 0700 /root/.ssh && \ + wget http://github.com/mitchellh/vagrant/raw/master/keys/vagrant.pub --no-check-certificate -O /root/.ssh/authorized_keys && \ + chmod 0600 /root/.ssh/authorized_keys && \ + # Install required Puppet modules + puppet apply -e "include bigtop_toolchain::puppet-modules" + +CMD /usr/sbin/sshd -D http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/docker/bigtop-deploy/debian-8/build.sh ---------------------------------------------------------------------- diff --git a/docker/bigtop-deploy/debian-8/build.sh b/docker/bigtop-deploy/debian-8/build.sh new file mode 100755 index 0000000..e16d3ba --- /dev/null +++ b/docker/bigtop-deploy/debian-8/build.sh @@ -0,0 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +cp -r ../../../bigtop_toolchain . +docker build -t bigtop/deploy:debian-8 . +rm -rf bigtop_toolchain
