Repository: bigtop Updated Branches: refs/heads/master d21a37a2c -> 999e73499
BIGTOP-2874: juju bundle refresh (august 2017) Closes #281 Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/999e7349 Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/999e7349 Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/999e7349 Branch: refs/heads/master Commit: 999e7349990a24551be4a2eaa415e80dc0e5e51e Parents: d21a37a Author: Kevin W Monroe <[email protected]> Authored: Fri Jul 7 18:23:48 2017 +0000 Committer: Kevin W Monroe <[email protected]> Committed: Tue Sep 5 11:29:55 2017 -0500 ---------------------------------------------------------------------- bigtop-deploy/juju/hadoop-hbase/README.md | 12 +- bigtop-deploy/juju/hadoop-hbase/bundle-dev.yaml | 139 ---------------- .../juju/hadoop-hbase/bundle-local.yaml | 11 +- bigtop-deploy/juju/hadoop-hbase/bundle.yaml | 23 +-- bigtop-deploy/juju/hadoop-kafka/README.md | 6 +- bigtop-deploy/juju/hadoop-kafka/bundle-dev.yaml | 160 ------------------- .../juju/hadoop-kafka/bundle-local.yaml | 11 +- bigtop-deploy/juju/hadoop-kafka/bundle.yaml | 23 +-- bigtop-deploy/juju/hadoop-processing/README.md | 6 +- .../juju/hadoop-processing/bundle-dev.yaml | 102 ------------ .../juju/hadoop-processing/bundle-local.yaml | 6 +- .../juju/hadoop-processing/bundle.yaml | 14 +- bigtop-deploy/juju/hadoop-spark/README.md | 12 +- bigtop-deploy/juju/hadoop-spark/bundle-dev.yaml | 120 -------------- .../juju/hadoop-spark/bundle-local.yaml | 2 +- bigtop-deploy/juju/hadoop-spark/bundle.yaml | 14 +- bigtop-deploy/juju/spark-processing/README.md | 4 +- .../juju/spark-processing/bundle-dev.yaml | 76 --------- .../juju/spark-processing/bundle-local.yaml | 6 +- bigtop-deploy/juju/spark-processing/bundle.yaml | 10 +- .../src/charm/kafka/layer-kafka/config.yaml | 8 + .../charm/spark/layer-spark/tests/10-test-ha.py | 4 +- 22 files changed, 106 insertions(+), 663 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-hbase/README.md ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-hbase/README.md b/bigtop-deploy/juju/hadoop-hbase/README.md index 2825327..57fe119 100644 --- a/bigtop-deploy/juju/hadoop-hbase/README.md +++ b/bigtop-deploy/juju/hadoop-hbase/README.md @@ -158,7 +158,7 @@ Show the list of Zookeeper nodes with the following: juju run --unit zookeeper/0 'echo "ls /" | /usr/lib/zookeeper/bin/zkCli.sh' -To access the HDFS web console, find the `PUBLIC-ADDRESS` of the namenode +To access the HDFS web console, find the `Public address` of the namenode application and expose it: juju status namenode @@ -169,7 +169,7 @@ The web interface will be available at the following URL: http://NAMENODE_PUBLIC_IP:50070 Similarly, to access the Resource Manager web consoles, find the -`PUBLIC-ADDRESS` of the resourcemanager application and expose it: +`Public address` of the resourcemanager application and expose it: juju status resourcemanager juju expose resourcemanager @@ -179,7 +179,7 @@ The YARN and Job History web interfaces will be available at the following URLs: http://RESOURCEMANAGER_PUBLIC_IP:8088 http://RESOURCEMANAGER_PUBLIC_IP:19888 -Finally, to access the HBase web console, find the `PUBLIC-ADDRESS` of any +Finally, to access the HBase web console, find the `Public address` of any hbase unit and expose the application: juju status hbase @@ -193,9 +193,9 @@ The web interface will be available at the following URL: # Monitoring This bundle includes Ganglia for system-level monitoring of the namenode, -resourcemanager, slave, hbase, and zookeeper units. Metrics are sent to a +resourcemanager, slave, and zookeeper units. Metrics are sent to a centralized ganglia unit for easy viewing in a browser. To view the ganglia web -interface, find the `PUBLIC-ADDRESS` of the Ganglia application and expose it: +interface, find the `Public address` of the Ganglia application and expose it: juju status ganglia juju expose ganglia @@ -208,7 +208,7 @@ The web interface will be available at: # Logging This bundle includes rsyslog to collect syslog data from the namenode, -resourcemanager, slave, hbase, and zookeeper units. These logs are sent to a +resourcemanager, slave, and zookeeper units. These logs are sent to a centralized rsyslog unit for easy syslog analysis. One method of viewing this log data is to simply cat syslog from the rsyslog unit: http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-hbase/bundle-dev.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-hbase/bundle-dev.yaml b/bigtop-deploy/juju/hadoop-hbase/bundle-dev.yaml deleted file mode 100644 index f7bcf56..0000000 --- a/bigtop-deploy/juju/hadoop-hbase/bundle-dev.yaml +++ /dev/null @@ -1,139 +0,0 @@ -services: - namenode: - charm: "cs:~bigdata-dev/xenial/hadoop-namenode" - constraints: "mem=7G root-disk=32G" - num_units: 1 - annotations: - gui-x: "500" - gui-y: "800" - to: - - "0" - resourcemanager: - charm: "cs:~bigdata-dev/xenial/hadoop-resourcemanager" - constraints: "mem=7G root-disk=32G" - num_units: 1 - annotations: - gui-x: "500" - gui-y: "0" - to: - - "0" - slave: - charm: "cs:~bigdata-dev/xenial/hadoop-slave" - constraints: "mem=7G root-disk=32G" - num_units: 3 - annotations: - gui-x: "0" - gui-y: "400" - to: - - "1" - - "2" - - "3" - plugin: - charm: "cs:~bigdata-dev/xenial/hadoop-plugin" - annotations: - gui-x: "1000" - gui-y: "400" - client: - charm: "cs:xenial/hadoop-client-8" - constraints: "mem=3G" - num_units: 1 - annotations: - gui-x: "1250" - gui-y: "400" - to: - - "4" - hbase: - charm: "cs:~bigdata-dev/xenial/hbase" - constraints: "mem=7G root-disk=32G" - num_units: 3 - annotations: - gui-x: "0" - gui-y: "0" - to: - - "1" - - "2" - - "3" - zookeeper: - charm: "cs:~bigdata-dev/xenial/zookeeper" - constraints: "mem=3G root-disk=32G" - num_units: 3 - annotations: - gui-x: "500" - gui-y: "400" - to: - - "5" - - "6" - - "7" - ganglia: - charm: "cs:xenial/ganglia-12" - num_units: 1 - annotations: - gui-x: "0" - gui-y: "800" - to: - - "4" - ganglia-node: - charm: "cs:xenial/ganglia-node-7" - annotations: - gui-x: "250" - gui-y: "400" - rsyslog: - charm: "cs:~bigdata-dev/xenial/rsyslog-7" - num_units: 1 - annotations: - gui-x: "1000" - gui-y: "800" - to: - - "4" - rsyslog-forwarder-ha: - charm: "cs:~bigdata-dev/xenial/rsyslog-forwarder-ha-7" - annotations: - gui-x: "750" - gui-y: "400" -series: xenial -relations: - - [resourcemanager, namenode] - - [namenode, slave] - - [resourcemanager, slave] - - [plugin, namenode] - - [plugin, resourcemanager] - - [client, plugin] - - [hbase, plugin] - - [hbase, zookeeper] - - ["ganglia-node:juju-info", "namenode:juju-info"] - - ["ganglia-node:juju-info", "resourcemanager:juju-info"] - - ["ganglia-node:juju-info", "slave:juju-info"] - - ["ganglia-node:juju-info", "hbase:juju-info"] - - ["ganglia-node:juju-info", "zookeeper:juju-info"] - - ["ganglia:node", "ganglia-node:node"] - - ["rsyslog-forwarder-ha:juju-info", "namenode:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "resourcemanager:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "slave:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "hbase:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "zookeeper:juju-info"] - - ["rsyslog:aggregator", "rsyslog-forwarder-ha:syslog"] -machines: - "0": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "1": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "2": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "3": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "4": - series: "xenial" - constraints: "mem=3G" - "5": - series: "xenial" - constraints: "mem=3G root-disk=32G" - "6": - series: "xenial" - constraints: "mem=3G root-disk=32G" - "7": - series: "xenial" - constraints: "mem=3G root-disk=32G" http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-hbase/bundle-local.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-hbase/bundle-local.yaml b/bigtop-deploy/juju/hadoop-hbase/bundle-local.yaml index 26e0f3e..17523e1 100644 --- a/bigtop-deploy/juju/hadoop-hbase/bundle-local.yaml +++ b/bigtop-deploy/juju/hadoop-hbase/bundle-local.yaml @@ -1,3 +1,9 @@ +series: xenial +description: > + This is a fourteen unit big data cluster that includes Hadoop 2.7 and + HBase 1.1 from Apache Bigtop. Use it when you need a distributed big data + store with MapReduce processing capabilities. It will run on 8 machines in + your cloud. services: namenode: charm: "/home/ubuntu/charms/xenial/hadoop-namenode" @@ -47,7 +53,7 @@ services: constraints: "mem=7G root-disk=32G" num_units: 3 annotations: - gui-x: "0" + gui-x: "1000" gui-y: "0" to: - "1" @@ -90,7 +96,6 @@ services: annotations: gui-x: "750" gui-y: "400" -series: xenial relations: - [resourcemanager, namenode] - [namenode, slave] @@ -103,13 +108,11 @@ relations: - ["ganglia-node:juju-info", "namenode:juju-info"] - ["ganglia-node:juju-info", "resourcemanager:juju-info"] - ["ganglia-node:juju-info", "slave:juju-info"] - - ["ganglia-node:juju-info", "hbase:juju-info"] - ["ganglia-node:juju-info", "zookeeper:juju-info"] - ["ganglia:node", "ganglia-node:node"] - ["rsyslog-forwarder-ha:juju-info", "namenode:juju-info"] - ["rsyslog-forwarder-ha:juju-info", "resourcemanager:juju-info"] - ["rsyslog-forwarder-ha:juju-info", "slave:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "hbase:juju-info"] - ["rsyslog-forwarder-ha:juju-info", "zookeeper:juju-info"] - ["rsyslog:aggregator", "rsyslog-forwarder-ha:syslog"] machines: http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-hbase/bundle.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-hbase/bundle.yaml b/bigtop-deploy/juju/hadoop-hbase/bundle.yaml index b51712a..9f00e95 100644 --- a/bigtop-deploy/juju/hadoop-hbase/bundle.yaml +++ b/bigtop-deploy/juju/hadoop-hbase/bundle.yaml @@ -1,6 +1,12 @@ +series: xenial +description: > + This is a fourteen unit big data cluster that includes Hadoop 2.7 and + HBase 1.1 from Apache Bigtop. Use it when you need a distributed big data + store with MapReduce processing capabilities. It will run on 8 machines in + your cloud. services: namenode: - charm: "cs:xenial/hadoop-namenode-24" + charm: "cs:xenial/hadoop-namenode-31" constraints: "mem=7G root-disk=32G" num_units: 1 annotations: @@ -9,7 +15,7 @@ services: to: - "0" resourcemanager: - charm: "cs:xenial/hadoop-resourcemanager-26" + charm: "cs:xenial/hadoop-resourcemanager-33" constraints: "mem=7G root-disk=32G" num_units: 1 annotations: @@ -18,7 +24,7 @@ services: to: - "0" slave: - charm: "cs:xenial/hadoop-slave-25" + charm: "cs:xenial/hadoop-slave-32" constraints: "mem=7G root-disk=32G" num_units: 3 annotations: @@ -29,7 +35,7 @@ services: - "2" - "3" plugin: - charm: "cs:xenial/hadoop-plugin-24" + charm: "cs:xenial/hadoop-plugin-31" annotations: gui-x: "1000" gui-y: "400" @@ -43,18 +49,18 @@ services: to: - "4" hbase: - charm: "cs:xenial/hbase-25" + charm: "cs:xenial/hbase-32" constraints: "mem=7G root-disk=32G" num_units: 3 annotations: - gui-x: "0" + gui-x: "1000" gui-y: "0" to: - "1" - "2" - "3" zookeeper: - charm: "cs:xenial/zookeeper-30" + charm: "cs:xenial/zookeeper-37" constraints: "mem=3G root-disk=32G" num_units: 3 annotations: @@ -90,7 +96,6 @@ services: annotations: gui-x: "750" gui-y: "400" -series: xenial relations: - [resourcemanager, namenode] - [namenode, slave] @@ -103,13 +108,11 @@ relations: - ["ganglia-node:juju-info", "namenode:juju-info"] - ["ganglia-node:juju-info", "resourcemanager:juju-info"] - ["ganglia-node:juju-info", "slave:juju-info"] - - ["ganglia-node:juju-info", "hbase:juju-info"] - ["ganglia-node:juju-info", "zookeeper:juju-info"] - ["ganglia:node", "ganglia-node:node"] - ["rsyslog-forwarder-ha:juju-info", "namenode:juju-info"] - ["rsyslog-forwarder-ha:juju-info", "resourcemanager:juju-info"] - ["rsyslog-forwarder-ha:juju-info", "slave:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "hbase:juju-info"] - ["rsyslog-forwarder-ha:juju-info", "zookeeper:juju-info"] - ["rsyslog:aggregator", "rsyslog-forwarder-ha:syslog"] machines: http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-kafka/README.md ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-kafka/README.md b/bigtop-deploy/juju/hadoop-kafka/README.md index 4caf774..e87f1b7 100644 --- a/bigtop-deploy/juju/hadoop-kafka/README.md +++ b/bigtop-deploy/juju/hadoop-kafka/README.md @@ -180,7 +180,7 @@ Show the list of Zookeeper nodes with the following: juju run --unit zookeeper/0 'echo "ls /" | /usr/lib/zookeeper/bin/zkCli.sh' -To access the HDFS web console, find the `PUBLIC-ADDRESS` of the namenode +To access the HDFS web console, find the `Public address` of the namenode application and expose it: juju status namenode @@ -191,7 +191,7 @@ The web interface will be available at the following URL: http://NAMENODE_PUBLIC_IP:50070 Similarly, to access the Resource Manager web consoles, find the -`PUBLIC-ADDRESS` of the resourcemanager application and expose it: +`Public address` of the resourcemanager application and expose it: juju status resourcemanager juju expose resourcemanager @@ -207,7 +207,7 @@ The YARN and Job History web interfaces will be available at the following URLs: This bundle includes Ganglia for system-level monitoring of the namenode, resourcemanager, slave, kafka, and zookeeper units. Metrics are sent to a centralized ganglia unit for easy viewing in a browser. To view the ganglia web -interface, find the `PUBLIC-ADDRESS` of the Ganglia application and expose it: +interface, find the `Public address` of the Ganglia application and expose it: juju status ganglia juju expose ganglia http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-kafka/bundle-dev.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-kafka/bundle-dev.yaml b/bigtop-deploy/juju/hadoop-kafka/bundle-dev.yaml deleted file mode 100644 index f715f78..0000000 --- a/bigtop-deploy/juju/hadoop-kafka/bundle-dev.yaml +++ /dev/null @@ -1,160 +0,0 @@ -services: - namenode: - charm: "cs:~bigdata-dev/xenial/hadoop-namenode" - constraints: "mem=7G root-disk=32G" - num_units: 1 - annotations: - gui-x: "500" - gui-y: "800" - to: - - "0" - resourcemanager: - charm: "cs:~bigdata-dev/xenial/hadoop-resourcemanager" - constraints: "mem=7G root-disk=32G" - num_units: 1 - annotations: - gui-x: "500" - gui-y: "0" - to: - - "0" - slave: - charm: "cs:~bigdata-dev/xenial/hadoop-slave" - constraints: "mem=7G root-disk=32G" - num_units: 3 - annotations: - gui-x: "0" - gui-y: "400" - to: - - "1" - - "2" - - "3" - plugin: - charm: "cs:~bigdata-dev/xenial/hadoop-plugin" - annotations: - gui-x: "1000" - gui-y: "400" - client: - charm: "cs:xenial/hadoop-client-8" - constraints: "mem=3G" - num_units: 1 - annotations: - gui-x: "1250" - gui-y: "400" - to: - - "4" -# TODO: Charm bigtop flume - flume-hdfs: - charm: "cs:~bigdata-dev/xenial/apache-flume-hdfs-37" - num_units: 1 - annotations: - gui-x: "1500" - gui-y: "400" - to: - - "4" - zookeeper: - charm: "cs:~bigdata-dev/xenial/zookeeper" - constraints: "mem=3G root-disk=32G" - num_units: 3 - annotations: - gui-x: "500" - gui-y: "400" - to: - - "5" - - "6" - - "7" - kafka: - charm: "cs:~bigdata-dev/xenial/kafka" - constraints: "mem=3G" - num_units: 1 - annotations: - gui-x: "1250" - gui-y: "800" - to: - - "8" -# NOTE: flume-kafka cannot be colocated with flume-hdfs as they both use /etc/flume/conf - flume-kafka: - charm: "cs:~bigdata-dev/xenial/apache-flume-kafka-11" - num_units: 1 - annotations: - gui-x: "1500" - gui-y: "800" - to: - - "8" - ganglia: - charm: "cs:xenial/ganglia-12" - num_units: 1 - annotations: - gui-x: "0" - gui-y: "800" - to: - - "4" - ganglia-node: - charm: "cs:xenial/ganglia-node-7" - annotations: - gui-x: "250" - gui-y: "400" - rsyslog: - charm: "cs:~bigdata-dev/xenial/rsyslog-7" - num_units: 1 - annotations: - gui-x: "1000" - gui-y: "800" - to: - - "4" - rsyslog-forwarder-ha: - charm: "cs:~bigdata-dev/xenial/rsyslog-forwarder-ha-7" - annotations: - gui-x: "750" - gui-y: "400" -series: xenial -relations: - - [resourcemanager, namenode] - - [namenode, slave] - - [resourcemanager, slave] - - [plugin, namenode] - - [plugin, resourcemanager] - - [client, plugin] - - [flume-hdfs, plugin] - - [flume-kafka, flume-hdfs] - - [flume-kafka, kafka] - - [kafka, zookeeper] - - ["ganglia-node:juju-info", "namenode:juju-info"] - - ["ganglia-node:juju-info", "resourcemanager:juju-info"] - - ["ganglia-node:juju-info", "slave:juju-info"] - - ["ganglia-node:juju-info", "kafka:juju-info"] - - ["ganglia-node:juju-info", "zookeeper:juju-info"] - - ["ganglia:node", "ganglia-node:node"] - - ["rsyslog-forwarder-ha:juju-info", "namenode:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "resourcemanager:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "slave:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "kafka:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "zookeeper:juju-info"] - - ["rsyslog:aggregator", "rsyslog-forwarder-ha:syslog"] -machines: - "0": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "1": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "2": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "3": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "4": - series: "xenial" - constraints: "mem=3G" - "5": - series: "xenial" - constraints: "mem=3G root-disk=32G" - "6": - series: "xenial" - constraints: "mem=3G root-disk=32G" - "7": - series: "xenial" - constraints: "mem=3G root-disk=32G" - "8": - series: "xenial" - constraints: "mem=3G" http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-kafka/bundle-local.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-kafka/bundle-local.yaml b/bigtop-deploy/juju/hadoop-kafka/bundle-local.yaml index 82bfb16..e1cdc31 100644 --- a/bigtop-deploy/juju/hadoop-kafka/bundle-local.yaml +++ b/bigtop-deploy/juju/hadoop-kafka/bundle-local.yaml @@ -1,3 +1,8 @@ +series: xenial +description: > + This is a fourteen unit big data cluster that includes Hadoop 2.7 and + Kafka 0.10 from Apache Bigtop. Use it to store streams of data in HDFS + for future analysis with MapReduce. It will run on 9 machines in your cloud. services: namenode: charm: "/home/ubuntu/charms/xenial/hadoop-namenode" @@ -44,7 +49,7 @@ services: - "4" # TODO: Charm bigtop flume flume-hdfs: - charm: "cs:~bigdata-dev/xenial/apache-flume-hdfs-37" + charm: "cs:xenial/apache-flume-hdfs-7" num_units: 1 annotations: gui-x: "1500" @@ -71,9 +76,10 @@ services: gui-y: "800" to: - "8" +# TODO: Charm bigtop flume # NOTE: flume-kafka cannot be colocated with flume-hdfs as they both use /etc/flume/conf flume-kafka: - charm: "cs:~bigdata-dev/xenial/apache-flume-kafka-11" + charm: "cs:xenial/apache-flume-kafka-1" num_units: 1 annotations: gui-x: "1500" @@ -106,7 +112,6 @@ services: annotations: gui-x: "750" gui-y: "400" -series: xenial relations: - [resourcemanager, namenode] - [namenode, slave] http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-kafka/bundle.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-kafka/bundle.yaml b/bigtop-deploy/juju/hadoop-kafka/bundle.yaml index 832d50d..8d62f65 100644 --- a/bigtop-deploy/juju/hadoop-kafka/bundle.yaml +++ b/bigtop-deploy/juju/hadoop-kafka/bundle.yaml @@ -1,6 +1,11 @@ +series: xenial +description: > + This is a fourteen unit big data cluster that includes Hadoop 2.7 and + Kafka 0.10 from Apache Bigtop. Use it to store streams of data in HDFS + for future analysis with MapReduce. It will run on 9 machines in your cloud. services: namenode: - charm: "cs:xenial/hadoop-namenode-24" + charm: "cs:xenial/hadoop-namenode-31" constraints: "mem=7G root-disk=32G" num_units: 1 annotations: @@ -9,7 +14,7 @@ services: to: - "0" resourcemanager: - charm: "cs:xenial/hadoop-resourcemanager-26" + charm: "cs:xenial/hadoop-resourcemanager-33" constraints: "mem=7G root-disk=32G" num_units: 1 annotations: @@ -18,7 +23,7 @@ services: to: - "0" slave: - charm: "cs:xenial/hadoop-slave-25" + charm: "cs:xenial/hadoop-slave-32" constraints: "mem=7G root-disk=32G" num_units: 3 annotations: @@ -29,7 +34,7 @@ services: - "2" - "3" plugin: - charm: "cs:xenial/hadoop-plugin-24" + charm: "cs:xenial/hadoop-plugin-31" annotations: gui-x: "1000" gui-y: "400" @@ -44,7 +49,7 @@ services: - "4" # TODO: Charm bigtop flume flume-hdfs: - charm: "cs:~bigdata-dev/xenial/apache-flume-hdfs-37" + charm: "cs:xenial/apache-flume-hdfs-7" num_units: 1 annotations: gui-x: "1500" @@ -52,7 +57,7 @@ services: to: - "4" zookeeper: - charm: "cs:xenial/zookeeper-30" + charm: "cs:xenial/zookeeper-37" constraints: "mem=3G root-disk=32G" num_units: 3 annotations: @@ -63,7 +68,7 @@ services: - "6" - "7" kafka: - charm: "cs:xenial/kafka-27" + charm: "cs:xenial/kafka-35" constraints: "mem=3G" num_units: 1 annotations: @@ -71,9 +76,10 @@ services: gui-y: "800" to: - "8" +# TODO: Charm bigtop flume # NOTE: flume-kafka cannot be colocated with flume-hdfs as they both use /etc/flume/conf flume-kafka: - charm: "cs:~bigdata-dev/xenial/apache-flume-kafka-11" + charm: "cs:xenial/apache-flume-kafka-1" num_units: 1 annotations: gui-x: "1500" @@ -106,7 +112,6 @@ services: annotations: gui-x: "750" gui-y: "400" -series: xenial relations: - [resourcemanager, namenode] - [namenode, slave] http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-processing/README.md ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-processing/README.md b/bigtop-deploy/juju/hadoop-processing/README.md index 9ad44b6..936e07b 100644 --- a/bigtop-deploy/juju/hadoop-processing/README.md +++ b/bigtop-deploy/juju/hadoop-processing/README.md @@ -137,7 +137,7 @@ of YARN NodeManager units with the following: juju run --application namenode "su hdfs -c 'hdfs dfsadmin -report'" juju run --application resourcemanager "su yarn -c 'yarn node -list'" -To access the HDFS web console, find the `PUBLIC-ADDRESS` of the namenode +To access the HDFS web console, find the `Public address` of the namenode application and expose it: juju status namenode @@ -148,7 +148,7 @@ The web interface will be available at the following URL: http://NAMENODE_PUBLIC_IP:50070 Similarly, to access the Resource Manager web consoles, find the -`PUBLIC-ADDRESS` of the resourcemanager application and expose it: +`Public address` of the resourcemanager application and expose it: juju status resourcemanager juju expose resourcemanager @@ -164,7 +164,7 @@ The YARN and Job History web interfaces will be available at the following URLs: This bundle includes Ganglia for system-level monitoring of the namenode, resourcemanager, slave, and client units. Metrics are sent to a centralized ganglia unit for easy viewing in a browser. To view the ganglia web interface, -find the `PUBLIC-ADDRESS` of the Ganglia application and expose it: +find the `Public address` of the Ganglia application and expose it: juju status ganglia juju expose ganglia http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-processing/bundle-dev.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-processing/bundle-dev.yaml b/bigtop-deploy/juju/hadoop-processing/bundle-dev.yaml deleted file mode 100644 index 42b5f38..0000000 --- a/bigtop-deploy/juju/hadoop-processing/bundle-dev.yaml +++ /dev/null @@ -1,102 +0,0 @@ -services: - namenode: - charm: "cs:~bigdata-dev/xenial/hadoop-namenode" - constraints: "mem=7G root-disk=32G" - num_units: 1 - annotations: - gui-x: "500" - gui-y: "800" - to: - - "0" - resourcemanager: - charm: "cs:~bigdata-dev/xenial/hadoop-resourcemanager" - constraints: "mem=7G root-disk=32G" - num_units: 1 - annotations: - gui-x: "500" - gui-y: "0" - to: - - "0" - slave: - charm: "cs:~bigdata-dev/xenial/hadoop-slave" - constraints: "mem=7G root-disk=32G" - num_units: 3 - annotations: - gui-x: "0" - gui-y: "400" - to: - - "1" - - "2" - - "3" - plugin: - charm: "cs:~bigdata-dev/xenial/hadoop-plugin" - annotations: - gui-x: "1000" - gui-y: "400" - client: - charm: "cs:xenial/hadoop-client-8" - constraints: "mem=3G" - num_units: 1 - annotations: - gui-x: "1250" - gui-y: "400" - to: - - "4" - ganglia: - charm: "cs:xenial/ganglia-12" - num_units: 1 - annotations: - gui-x: "0" - gui-y: "800" - to: - - "4" - ganglia-node: - charm: "cs:xenial/ganglia-node-7" - annotations: - gui-x: "250" - gui-y: "400" - rsyslog: - charm: "cs:~bigdata-dev/xenial/rsyslog-7" - num_units: 1 - annotations: - gui-x: "1000" - gui-y: "800" - to: - - "4" - rsyslog-forwarder-ha: - charm: "cs:~bigdata-dev/xenial/rsyslog-forwarder-ha-7" - annotations: - gui-x: "750" - gui-y: "400" -series: xenial -relations: - - [resourcemanager, namenode] - - [namenode, slave] - - [resourcemanager, slave] - - [plugin, namenode] - - [plugin, resourcemanager] - - [client, plugin] - - ["ganglia-node:juju-info", "namenode:juju-info"] - - ["ganglia-node:juju-info", "resourcemanager:juju-info"] - - ["ganglia-node:juju-info", "slave:juju-info"] - - ["ganglia:node", "ganglia-node:node"] - - ["rsyslog-forwarder-ha:juju-info", "namenode:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "resourcemanager:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "slave:juju-info"] - - ["rsyslog:aggregator", "rsyslog-forwarder-ha:syslog"] -machines: - "0": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "1": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "2": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "3": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "4": - series: "xenial" - constraints: "mem=3G" http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-processing/bundle-local.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-processing/bundle-local.yaml b/bigtop-deploy/juju/hadoop-processing/bundle-local.yaml index 736f73b..439a92a 100644 --- a/bigtop-deploy/juju/hadoop-processing/bundle-local.yaml +++ b/bigtop-deploy/juju/hadoop-processing/bundle-local.yaml @@ -1,3 +1,8 @@ +series: xenial +description: > + This is an eight unit big data cluster that includes Hadoop 2.7 from Apache + Bigtop. Use it to store data in HDFS and run MapReduce analysis jobs. It + will run on 5 machines in your cloud. services: namenode: charm: "/home/ubuntu/charms/xenial/hadoop-namenode" @@ -68,7 +73,6 @@ services: annotations: gui-x: "750" gui-y: "400" -series: xenial relations: - [resourcemanager, namenode] - [namenode, slave] http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-processing/bundle.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-processing/bundle.yaml b/bigtop-deploy/juju/hadoop-processing/bundle.yaml index fbd973d..e86fad0 100644 --- a/bigtop-deploy/juju/hadoop-processing/bundle.yaml +++ b/bigtop-deploy/juju/hadoop-processing/bundle.yaml @@ -1,6 +1,11 @@ +series: xenial +description: > + This is an eight unit big data cluster that includes Hadoop 2.7 from Apache + Bigtop. Use it to store data in HDFS and run MapReduce analysis jobs. It + will run on 5 machines in your cloud. services: namenode: - charm: "cs:xenial/hadoop-namenode-24" + charm: "cs:xenial/hadoop-namenode-31" constraints: "mem=7G root-disk=32G" num_units: 1 annotations: @@ -9,7 +14,7 @@ services: to: - "0" resourcemanager: - charm: "cs:xenial/hadoop-resourcemanager-26" + charm: "cs:xenial/hadoop-resourcemanager-33" constraints: "mem=7G root-disk=32G" num_units: 1 annotations: @@ -18,7 +23,7 @@ services: to: - "0" slave: - charm: "cs:xenial/hadoop-slave-25" + charm: "cs:xenial/hadoop-slave-32" constraints: "mem=7G root-disk=32G" num_units: 3 annotations: @@ -29,7 +34,7 @@ services: - "2" - "3" plugin: - charm: "cs:xenial/hadoop-plugin-24" + charm: "cs:xenial/hadoop-plugin-31" annotations: gui-x: "1000" gui-y: "400" @@ -68,7 +73,6 @@ services: annotations: gui-x: "750" gui-y: "400" -series: xenial relations: - [resourcemanager, namenode] - [namenode, slave] http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-spark/README.md ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-spark/README.md b/bigtop-deploy/juju/hadoop-spark/README.md index edd0463..7849c09 100644 --- a/bigtop-deploy/juju/hadoop-spark/README.md +++ b/bigtop-deploy/juju/hadoop-spark/README.md @@ -146,7 +146,7 @@ of YARN NodeManager units with the following: juju run --application namenode "su hdfs -c 'hdfs dfsadmin -report'" juju run --application resourcemanager "su yarn -c 'yarn node -list'" -To access the HDFS web console, find the `PUBLIC-ADDRESS` of the namenode +To access the HDFS web console, find the `Public address` of the namenode application and expose it: juju status namenode @@ -157,7 +157,7 @@ The web interface will be available at the following URL: http://NAMENODE_PUBLIC_IP:50070 Similarly, to access the Resource Manager web consoles, find the -`PUBLIC-ADDRESS` of the resourcemanager application and expose it: +`Public address` of the resourcemanager application and expose it: juju status resourcemanager juju expose resourcemanager @@ -167,7 +167,7 @@ The YARN and Job History web interfaces will be available at the following URLs: http://RESOURCEMANAGER_PUBLIC_IP:8088 http://RESOURCEMANAGER_PUBLIC_IP:19888 -Finally, to access the Spark web console, find the `PUBLIC-ADDRESS` of the +Finally, to access the Spark web console, find the `Public address` of the spark application and expose it: juju status spark @@ -181,9 +181,9 @@ The web interface will be available at the following URL: # Monitoring This bundle includes Ganglia for system-level monitoring of the namenode, -resourcemanager, slave, and spark units. Metrics are sent to a -centralized ganglia unit for easy viewing in a browser. To view the ganglia web -interface, find the `PUBLIC-ADDRESS` of the Ganglia application and expose it: +resourcemanager, and slave units. Metrics are sent to a centralized ganglia +unit for easy viewing in a browser. To view the ganglia web interface, find +the `Public address` of the Ganglia application and expose it: juju status ganglia juju expose ganglia http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-spark/bundle-dev.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-spark/bundle-dev.yaml b/bigtop-deploy/juju/hadoop-spark/bundle-dev.yaml deleted file mode 100644 index 99678f1..0000000 --- a/bigtop-deploy/juju/hadoop-spark/bundle-dev.yaml +++ /dev/null @@ -1,120 +0,0 @@ -series: xenial -description: > - This is a five unit big data cluster that includes Hadoop 2.7.3 and Spark 2.1 - from Apache Bigtop. Use it to analyse batch data with MapReduce or streaming - data with Spark. It will run on 5 machines in your cloud. -services: - namenode: - charm: "cs:~bigdata-dev/xenial/hadoop-namenode" - constraints: "mem=7G root-disk=32G" - num_units: 1 - annotations: - gui-x: "500" - gui-y: "800" - to: - - "0" - resourcemanager: - charm: "cs:~bigdata-dev/xenial/hadoop-resourcemanager" - constraints: "mem=7G root-disk=32G" - num_units: 1 - annotations: - gui-x: "500" - gui-y: "0" - to: - - "0" - slave: - charm: "cs:~bigdata-dev/xenial/hadoop-slave" - constraints: "mem=7G root-disk=32G" - num_units: 3 - annotations: - gui-x: "0" - gui-y: "400" - to: - - "1" - - "2" - - "3" - plugin: - charm: "cs:~bigdata-dev/xenial/hadoop-plugin" - annotations: - gui-x: "1000" - gui-y: "400" - client: - charm: "cs:xenial/hadoop-client-8" - constraints: "mem=7G root-disk=32G" - num_units: 1 - annotations: - gui-x: "1250" - gui-y: "400" - to: - - "4" - spark: - charm: "cs:~bigdata-dev/xenial/spark" - constraints: "mem=7G root-disk=32G" - num_units: 1 - options: - spark_execution_mode: "yarn-client" - annotations: - gui-x: "1000" - gui-y: "0" - to: - - "4" - ganglia: - charm: "cs:xenial/ganglia-12" - num_units: 1 - annotations: - gui-x: "0" - gui-y: "800" - to: - - "4" - ganglia-node: - charm: "cs:xenial/ganglia-node-7" - annotations: - gui-x: "250" - gui-y: "400" - rsyslog: - charm: "cs:~bigdata-dev/xenial/rsyslog-7" - num_units: 1 - annotations: - gui-x: "1000" - gui-y: "800" - to: - - "4" - rsyslog-forwarder-ha: - charm: "cs:~bigdata-dev/xenial/rsyslog-forwarder-ha-7" - annotations: - gui-x: "750" - gui-y: "400" -relations: - - [resourcemanager, namenode] - - [namenode, slave] - - [resourcemanager, slave] - - [plugin, namenode] - - [plugin, resourcemanager] - - [spark, plugin] - - [client, plugin] - - ["ganglia-node:juju-info", "namenode:juju-info"] - - ["ganglia-node:juju-info", "resourcemanager:juju-info"] - - ["ganglia-node:juju-info", "slave:juju-info"] - - ["ganglia-node:juju-info", "spark:juju-info"] - - ["ganglia:node", "ganglia-node:node"] - - ["rsyslog-forwarder-ha:juju-info", "namenode:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "resourcemanager:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "slave:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "spark:juju-info"] - - ["rsyslog:aggregator", "rsyslog-forwarder-ha:syslog"] -machines: - "0": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "1": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "2": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "3": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "4": - series: "xenial" - constraints: "mem=7G root-disk=32G" http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-spark/bundle-local.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-spark/bundle-local.yaml b/bigtop-deploy/juju/hadoop-spark/bundle-local.yaml index 297779a..d777b37 100644 --- a/bigtop-deploy/juju/hadoop-spark/bundle-local.yaml +++ b/bigtop-deploy/juju/hadoop-spark/bundle-local.yaml @@ -1,6 +1,6 @@ series: xenial description: > - This is a five unit big data cluster that includes Hadoop 2.7.3 and Spark 2.1 + This is a nine unit big data cluster that includes Hadoop 2.7 and Spark 2.1 from Apache Bigtop. Use it to analyse batch data with MapReduce or streaming data with Spark. It will run on 5 machines in your cloud. services: http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/hadoop-spark/bundle.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/hadoop-spark/bundle.yaml b/bigtop-deploy/juju/hadoop-spark/bundle.yaml index d26b6ae..05786a0 100644 --- a/bigtop-deploy/juju/hadoop-spark/bundle.yaml +++ b/bigtop-deploy/juju/hadoop-spark/bundle.yaml @@ -1,11 +1,11 @@ series: xenial description: > - This is a five unit big data cluster that includes Hadoop 2.7.3 and Spark 2.1 + This is a nine unit big data cluster that includes Hadoop 2.7 and Spark 2.1 from Apache Bigtop. Use it to analyse batch data with MapReduce or streaming data with Spark. It will run on 5 machines in your cloud. services: namenode: - charm: "cs:xenial/hadoop-namenode-24" + charm: "cs:xenial/hadoop-namenode-31" constraints: "mem=7G root-disk=32G" num_units: 1 annotations: @@ -14,7 +14,7 @@ services: to: - "0" resourcemanager: - charm: "cs:xenial/hadoop-resourcemanager-26" + charm: "cs:xenial/hadoop-resourcemanager-33" constraints: "mem=7G root-disk=32G" num_units: 1 annotations: @@ -23,7 +23,7 @@ services: to: - "0" slave: - charm: "cs:xenial/hadoop-slave-25" + charm: "cs:xenial/hadoop-slave-32" constraints: "mem=7G root-disk=32G" num_units: 3 annotations: @@ -34,7 +34,7 @@ services: - "2" - "3" plugin: - charm: "cs:xenial/hadoop-plugin-24" + charm: "cs:xenial/hadoop-plugin-31" annotations: gui-x: "1000" gui-y: "400" @@ -48,7 +48,7 @@ services: to: - "4" spark: - charm: "cs:xenial/spark-49" + charm: "cs:xenial/spark-56" constraints: "mem=7G root-disk=32G" num_units: 1 options: @@ -95,12 +95,10 @@ relations: - ["ganglia-node:juju-info", "namenode:juju-info"] - ["ganglia-node:juju-info", "resourcemanager:juju-info"] - ["ganglia-node:juju-info", "slave:juju-info"] - - ["ganglia-node:juju-info", "spark:juju-info"] - ["ganglia:node", "ganglia-node:node"] - ["rsyslog-forwarder-ha:juju-info", "namenode:juju-info"] - ["rsyslog-forwarder-ha:juju-info", "resourcemanager:juju-info"] - ["rsyslog-forwarder-ha:juju-info", "slave:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "spark:juju-info"] - ["rsyslog:aggregator", "rsyslog-forwarder-ha:syslog"] machines: "0": http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/spark-processing/README.md ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/spark-processing/README.md b/bigtop-deploy/juju/spark-processing/README.md index a499a38..b785f69 100644 --- a/bigtop-deploy/juju/spark-processing/README.md +++ b/bigtop-deploy/juju/spark-processing/README.md @@ -122,7 +122,7 @@ From the command line, show the list of Zookeeper nodes with the following: juju run --unit zookeeper/0 'echo "ls /" | /usr/lib/zookeeper/bin/zkCli.sh' -To access the Spark web console, find the `PUBLIC-ADDRESS` of the spark +To access the Spark web console, find the `Public address` of the spark application and expose it: juju status spark @@ -138,7 +138,7 @@ The web interface will be available at the following URL: This bundle includes Ganglia for system-level monitoring of the spark and zookeeper units. Metrics are sent to a centralized ganglia unit for easy viewing in a browser. To view the ganglia web interface, find the -`PUBLIC-ADDRESS` of the Ganglia application and expose it: +`Public address` of the Ganglia application and expose it: juju status ganglia juju expose ganglia http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/spark-processing/bundle-dev.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/spark-processing/bundle-dev.yaml b/bigtop-deploy/juju/spark-processing/bundle-dev.yaml deleted file mode 100644 index 4ace0da..0000000 --- a/bigtop-deploy/juju/spark-processing/bundle-dev.yaml +++ /dev/null @@ -1,76 +0,0 @@ -services: - spark: - charm: "cs:~bigdata-dev/xenial/spark" - constraints: "mem=7G root-disk=32G" - num_units: 2 - annotations: - gui-x: "500" - gui-y: "0" - to: - - "0" - - "1" - zookeeper: - charm: "cs:~bigdata-dev/xenial/zookeeper" - constraints: "mem=3G root-disk=32G" - num_units: 3 - annotations: - gui-x: "500" - gui-y: "400" - to: - - "2" - - "3" - - "4" - ganglia: - charm: "cs:xenial/ganglia-12" - num_units: 1 - annotations: - gui-x: "0" - gui-y: "800" - to: - - "5" - ganglia-node: - charm: "cs:xenial/ganglia-node-7" - annotations: - gui-x: "250" - gui-y: "400" - rsyslog: - charm: "cs:~bigdata-dev/xenial/rsyslog-7" - num_units: 1 - annotations: - gui-x: "1000" - gui-y: "800" - to: - - "5" - rsyslog-forwarder-ha: - charm: "cs:~bigdata-dev/xenial/rsyslog-forwarder-ha-7" - annotations: - gui-x: "750" - gui-y: "400" -series: xenial -relations: - - [spark, zookeeper] - - ["ganglia-node:juju-info", "spark:juju-info"] - - ["ganglia-node:juju-info", "zookeeper:juju-info"] - - ["ganglia:node", "ganglia-node:node"] - - ["rsyslog-forwarder-ha:juju-info", "spark:juju-info"] - - ["rsyslog-forwarder-ha:juju-info", "zookeeper:juju-info"] - - ["rsyslog:aggregator", "rsyslog-forwarder-ha:syslog"] -machines: - "0": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "1": - series: "xenial" - constraints: "mem=7G root-disk=32G" - "2": - series: "xenial" - constraints: "mem=3G root-disk=32G" - "3": - series: "xenial" - constraints: "mem=3G root-disk=32G" - "4": - series: "xenial" - constraints: "mem=3G root-disk=32G" - "5": - series: "xenial" - constraints: "mem=3G" http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/spark-processing/bundle-local.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/spark-processing/bundle-local.yaml b/bigtop-deploy/juju/spark-processing/bundle-local.yaml index 95a28f0..68d5beb 100644 --- a/bigtop-deploy/juju/spark-processing/bundle-local.yaml +++ b/bigtop-deploy/juju/spark-processing/bundle-local.yaml @@ -1,3 +1,8 @@ +series: xenial +description: > + This is a seven unit big data cluster that includes Spark 2.1 from Apache + Bigtop. Use it to analyse streaming data or execute Spark jobs interactively + from the spark-shell. It will run on 6 machines in your cloud. services: spark: charm: "/home/ubuntu/charms/xenial/spark" @@ -46,7 +51,6 @@ services: annotations: gui-x: "750" gui-y: "400" -series: xenial relations: - [spark, zookeeper] - ["ganglia-node:juju-info", "spark:juju-info"] http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-deploy/juju/spark-processing/bundle.yaml ---------------------------------------------------------------------- diff --git a/bigtop-deploy/juju/spark-processing/bundle.yaml b/bigtop-deploy/juju/spark-processing/bundle.yaml index c2ba92a..5e7dd04 100644 --- a/bigtop-deploy/juju/spark-processing/bundle.yaml +++ b/bigtop-deploy/juju/spark-processing/bundle.yaml @@ -1,6 +1,11 @@ +series: xenial +description: > + This is a seven unit big data cluster that includes Spark 2.1 from Apache + Bigtop. Use it to analyse streaming data or execute Spark jobs interactively + from the spark-shell. It will run on 6 machines in your cloud. services: spark: - charm: "cs:xenial/spark-49" + charm: "cs:xenial/spark-56" constraints: "mem=7G root-disk=32G" num_units: 2 options: @@ -13,7 +18,7 @@ services: - "0" - "1" zookeeper: - charm: "cs:xenial/zookeeper-30" + charm: "cs:xenial/zookeeper-37" constraints: "mem=3G root-disk=32G" num_units: 3 annotations: @@ -49,7 +54,6 @@ services: annotations: gui-x: "750" gui-y: "400" -series: xenial relations: - [spark, zookeeper] - ["ganglia-node:juju-info", "spark:juju-info"] http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-packages/src/charm/kafka/layer-kafka/config.yaml ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/config.yaml b/bigtop-packages/src/charm/kafka/layer-kafka/config.yaml index c8e723a..337180a 100644 --- a/bigtop-packages/src/charm/kafka/layer-kafka/config.yaml +++ b/bigtop-packages/src/charm/kafka/layer-kafka/config.yaml @@ -1,4 +1,12 @@ options: + bigtop_version: + type: string + default: '1.2.1' + description: | + Apache Bigtop release version. The default, '1.2.1' will use the + current GA release for all hiera data, puppet recipes, + and installable packages. Set this to 'master' to use the latest + upstream bits. network_interface: default: "" type: string http://git-wip-us.apache.org/repos/asf/bigtop/blob/999e7349/bigtop-packages/src/charm/spark/layer-spark/tests/10-test-ha.py ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/spark/layer-spark/tests/10-test-ha.py b/bigtop-packages/src/charm/spark/layer-spark/tests/10-test-ha.py index 8ab0aec..59a8ede 100755 --- a/bigtop-packages/src/charm/spark/layer-spark/tests/10-test-ha.py +++ b/bigtop-packages/src/charm/spark/layer-spark/tests/10-test-ha.py @@ -68,10 +68,12 @@ class TestDeployment(unittest.TestCase): print("Removing master: {} ".format(master)) self.d.remove_unit(master) time.sleep(120) - self.d.sentry.wait_for_messages({"spark-test-ha": ["ready (standalone - HA)", "ready (standalone - HA)"]}, timeout=900) + print("Waiting for remaining units to agree on master.") + time.sleep(120) + masters_count = 0 for unit in self.d.sentry['spark-test-ha']: ip = unit.info['public-address']
