BIGTOP-2772. Add --dryrun mode into helper script and document
Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/5874ba53 Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/5874ba53 Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/5874ba53 Branch: refs/heads/branch-1.2 Commit: 5874ba53dba27454cb100b5637ba426430ce6d1d Parents: ef20c6c Author: Evans Ye <[email protected]> Authored: Tue May 23 22:36:30 2017 +0800 Committer: Evans Ye <[email protected]> Committed: Sat Jul 1 12:52:54 2017 +0000 ---------------------------------------------------------------------- docker/sandbox/README.md | 42 +++++++++++--------- docker/sandbox/build.sh | 1 + .../sandbox/site.yaml.template.centos-6_hadoop | 2 +- .../sandbox/site.yaml.template.debian-8_hadoop | 2 +- 4 files changed, 26 insertions(+), 21 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/bigtop/blob/5874ba53/docker/sandbox/README.md ---------------------------------------------------------------------- diff --git a/docker/sandbox/README.md b/docker/sandbox/README.md index 711903c..d7a6105 100644 --- a/docker/sandbox/README.md +++ b/docker/sandbox/README.md @@ -19,52 +19,56 @@ A tool to build and run big data pseudo cluster using Docker. ## How to run -* Make sure you have Docker installed. We've tested this using [Docker for Mac](https://docs.docker.com/docker-for-mac/) +Make sure you have Docker installed. We've tested this using [Docker for Mac](https://docs.docker.com/docker-for-mac/) -* Currently supported OS list: +Currently supported OS list: * centos-6 * debian-8 - * ubuntu-14.04 + * ubuntu-16.04 -* Running Hadoop +### Run Hadoop HDFS ``` -docker run -ti --privileged -p 50070:50070 bigtop/sandbox:centos-6_hadoop +docker run -d -p 50070:50070 bigtop/sandbox:centos-6_hdfs ``` -* Running Spark (Standalone mode) +### Run Spark on YARN ``` -docker run -ti --privileged -p 8080:8080 bigtop/sandbox:debian-8_spark +docker run -d -p 50070:50070 -p 8088:8088 -p 8080:8080 bigtop/sandbox:debian-8_hdfs_yarn_spark ``` -* Running Hadoop + HBase +### Run Hadoop HDFS + HBase ``` -docker run -ti --privileged -p 50070:50070 -p 60010:60010 bigtop/sandbox:ubuntu-14.04_hbase +docker run -d -p 50070:50070 -p 60010:60010 bigtop/sandbox:ubuntu-16.04_hdfs_hbase ``` ## How to build -### Examples +### Build a Hadoop HDFS sandbox image -* Build sandbox image that has Hadoop provisioned +``` +./build.sh -a bigtop -o centos-6 -c hdfs +``` + +### Build a Hadoop HDFS, Hadoop YARN, and Spark on YARN sandbox image ``` -./build.sh -a bigtop -o debian-8 -c hadoop +./build.sh -a bigtop -o debian-8 -c "hdfs, yarn, spark" ``` -* Build sandbox image that has Hadoop and Spark provisioned +### Build a Hadoop HDFS and HBase sandbox image ``` -./build.sh -a bigtop -o debian-8 -c "hadoop, spark" +./build.sh -a bigtop -o ubuntu-16.04 -c "hdfs, hbase" ``` -* Build sandbox image that has Hadoop and HBase provisioned +### Use --dryrun to skip the build and get Dockerfile and configuration ``` -./build.sh -a bigtop -o debian-8 -c "hadoop, yarn, hbase" +./build.sh -a bigtop -o ubuntu-16.04 -c "hdfs, hbase" --dryrun ``` ### Change the repository of packages @@ -73,10 +77,10 @@ docker run -ti --privileged -p 50070:50070 -p 60010:60010 bigtop/sandbox:ubuntu- ``` export REPO=http://ci.bigtop.apache.org:8080/job/Bigtop-trunk-repos/BUILD_ENVIRONMENTS=centos-6%2Clabel=docker-slave-06//ws/output -./build.sh -a bigtop -o centos-6 -c "hadoop, spark, ignite" +./build.sh -a bigtop -o centos-6 -c "hdfs, yarn, spark, ignite" ``` -### Customize your Hadoop stack +### Customize your Big Data Stack * Edit *site.yaml.template.centos-6_hadoop* to create your own prefered stack @@ -89,7 +93,7 @@ vim site.yaml.template.centos-6_hadoop_ignite ``` ... -hadoop_cluster_node::cluster_components: [hadoop, yarn, ignite] +hadoop_cluster_node::cluster_components: [hdfs, yarn, ignite] ... ``` http://git-wip-us.apache.org/repos/asf/bigtop/blob/5874ba53/docker/sandbox/build.sh ---------------------------------------------------------------------- diff --git a/docker/sandbox/build.sh b/docker/sandbox/build.sh index 4dfe2d3..58a76e7 100755 --- a/docker/sandbox/build.sh +++ b/docker/sandbox/build.sh @@ -145,6 +145,7 @@ usage() { echo " -c, --components Specify components to build." echo " You need to specify a comma separated, quoted string." echo " For example: --components \"hadoop, yarn\"" + echo " -d, --dryrun Generate the Dockerfile and configuration and skip the build." echo " -f, --file Specify a written site.yaml config file." echo " -o, --operating-system Specify an OS from Bigtop supported OS list." echo " RPM base: ${RPMS[*]}" http://git-wip-us.apache.org/repos/asf/bigtop/blob/5874ba53/docker/sandbox/site.yaml.template.centos-6_hadoop ---------------------------------------------------------------------- diff --git a/docker/sandbox/site.yaml.template.centos-6_hadoop b/docker/sandbox/site.yaml.template.centos-6_hadoop index 4fd7497..51fd682 100644 --- a/docker/sandbox/site.yaml.template.centos-6_hadoop +++ b/docker/sandbox/site.yaml.template.centos-6_hadoop @@ -16,5 +16,5 @@ bigtop::hadoop_head_node: "head.node.fqdn" hadoop::hadoop_storage_dirs: [/data/1, /data/2] bigtop::bigtop_repo_uri: http://bigtop-repos.s3.amazonaws.com/releases/1.2.0/centos/6/x86_64 -hadoop_cluster_node::cluster_components: [hdfs, yarn, mapreduce] +hadoop_cluster_node::cluster_components: [hdfs, yarn, mapred-app] bigtop::jdk_package_name: java-1.8.0-openjdk-devel.x86_64 http://git-wip-us.apache.org/repos/asf/bigtop/blob/5874ba53/docker/sandbox/site.yaml.template.debian-8_hadoop ---------------------------------------------------------------------- diff --git a/docker/sandbox/site.yaml.template.debian-8_hadoop b/docker/sandbox/site.yaml.template.debian-8_hadoop index d87146b..21c5d9d 100644 --- a/docker/sandbox/site.yaml.template.debian-8_hadoop +++ b/docker/sandbox/site.yaml.template.debian-8_hadoop @@ -16,5 +16,5 @@ bigtop::hadoop_head_node: "head.node.fqdn" hadoop::hadoop_storage_dirs: [/data/1, /data/2] bigtop::bigtop_repo_uri: http://bigtop-repos.s3.amazonaws.com/releases/1.2.0/debian/8/x86_64 -hadoop_cluster_node::cluster_components: [hdfs, yarn, mapreduce] +hadoop_cluster_node::cluster_components: [hdfs, yarn, mapred-app] bigtop::jdk_package_name: openjdk-8-jdk
