Repository: bigtop Updated Branches: refs/heads/master 2f8311b18 -> b5b366fe1
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hbase/layer-hbase/README.md ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/hbase/layer-hbase/README.md b/bigtop-packages/src/charm/hbase/layer-hbase/README.md index d5b714a..a9be1ac 100644 --- a/bigtop-packages/src/charm/hbase/layer-hbase/README.md +++ b/bigtop-packages/src/charm/hbase/layer-hbase/README.md @@ -21,58 +21,31 @@ store. Use HBase when you need random, realtime read/write access to your Big Data. This project's goal is the hosting of very large tables -- billions of rows X -millions of columns -- atop clusters of commodity hardware. - -HBase is an open-source, distributed, versioned, column-oriented store modeled -after Google's Bigtable: A Distributed Storage System for Structured Data by -Chang et al. Just as Bigtable leverages the distributed data storage provided -by the Google File System, HBase provides Bigtable-like capabilities on top of -Hadoop and HDFS. - -HBase provides: - -- Linear and modular scalability. -- Strictly consistent reads and writes. -- Automatic and configurable sharding of tables -- Automatic failover support between RegionServers. -- Convenient base classes for backing Hadoop MapReduce jobs with HBase tables. -- Easy to use Java API for client access. -- Block cache and Bloom Filters for real-time queries. -- Query predicate push down via server side Filters -- Thrift gateway and a REST-ful Web service that supports XML, Protobuf, - and binary data encoding options -- Extensible jruby-based (JIRB) shell -- Support for exporting metrics via the Hadoop metrics subsystem to files - or Ganglia; or via JMX. - -See [the homepage](http://hbase.apache.org) for more information. - -This charm deploys the hbase master and regionserver components of the -[Apache Bigtop][] platform. +millions of columns -- atop clusters of commodity hardware. Learn more at +[hbase.apache.org][]. +This charm deploys version 1.1.9 of the HBase master and regionserver +components from [Apache Bigtop][]. + +[hbase.apache.org]: http://hbase.apache.org/ [Apache Bigtop]: http://bigtop.apache.org/ # Deploying -A working Juju installation is assumed to be present. If Juju is not yet set -up, please follow the [getting-started][] instructions prior to deploying this -charm. +This charm requires Juju 2.0 or greater. If Juju is not yet set up, please +follow the [getting-started][] instructions prior to deploying this charm. An HBase deployment consists of HBase masters and HBase RegionServers. In a distributed HBase environment, one master and one regionserver are -deployed on each unit. HBase makes sure that only one master is active and -the rest are in standby mode in case the active one fails. +deployed on each unit. HBase ensures that only one master is active with +the rest in standby mode in case the active master fails. Because HBase requires HDFS, this charm is recommended to be deployed as part of the `hadoop-hbase` bundle: juju deploy hadoop-hbase -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart -hadoop-processing`. - This will deploy an Apache Bigtop Hadoop cluster with 3 HBase units. More information about this deployment can be found in the [bundle readme](https://jujucharms.com/hadoop-hbase/). @@ -86,7 +59,6 @@ in this environment, configure a Juju model with appropriate proxy and/or mirror options. See [Configuring Models][] for more information. [getting-started]: https://jujucharms.com/docs/stable/getting-started -[juju-quickstart]: https://launchpad.net/juju-quickstart [Configuring Models]: https://jujucharms.com/docs/stable/models-config @@ -113,25 +85,16 @@ application is functioning as expected. Run the action as follows: juju run-action hbase/0 smoke-test -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action do hbase/0 smoke-test`. - Watch the progress of the smoke test actions with: watch -n 2 juju show-action-status -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action status`. - Eventually, the action should settle to `status: completed`. If it reports `status: failed`, the application is not working as expected. Get more information about a specific smoke test with: juju show-action-output <action-id> -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action fetch <action-id>`. - ## HBase web UI HBase provides a web console that can be used to verify information about the cluster. To access it, find the `PUBLIC-ADDRESS` of any hbase unit and @@ -149,9 +112,6 @@ The web interface will be available at the following URL: Once the deployment has been verified, there are a number of actions available in this charm. -> **Note**: Actions described below assume Juju 2.0 or greater. If using an -earlier version of Juju, the action syntax is: -`juju action do hbase/0 <action_name> <action_args>; juju action fetch <id>`. Run a performance test: @@ -189,6 +149,18 @@ following events will cause a restart: - Upgrading the hbase charm. +# Issues + +Apache Bigtop tracks issues using JIRA (Apache account required). File an +issue for this charm at: + +https://issues.apache.org/jira/secure/CreateIssue!default.jspa + +Ensure `Bigtop` is selected as the project. Typically, charm issues are filed +in the `deployment` component with the latest stable release selected as the +affected version. Any uncertain fields may be left blank. + + # Contact Information - <[email protected]> @@ -196,10 +168,10 @@ following events will cause a restart: # Resources -- [Apache HBase home page](https://hbase.apache.org/) -- [Apache HBase issue tracker](https://issues.apache.org/jira/browse/HBASE) +- [Apache HBase home page](http://hbase.apache.org/) - [Apache Bigtop home page](http://bigtop.apache.org/) +- [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html) - [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html) -- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop) +- [Juju Big Data](https://jujucharms.com/big-data) +- [Juju Bigtop charms](https://jujucharms.com/q/bigtop) - [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju) -- [Juju community](https://jujucharms.com/community) http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hbase/layer-hbase/actions/perf-test ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/hbase/layer-hbase/actions/perf-test b/bigtop-packages/src/charm/hbase/layer-hbase/actions/perf-test index 2209d37..215e0dc 100755 --- a/bigtop-packages/src/charm/hbase/layer-hbase/actions/perf-test +++ b/bigtop-packages/src/charm/hbase/layer-hbase/actions/perf-test @@ -42,4 +42,5 @@ benchmark-finish DURATION=`expr $STOP - $START` benchmark-composite "${DURATION}" 'secs' 'asc' -benchmark-raw ${RESULT_LOG} +action-set meta.raw=${RESULT_LOG} +action-set outcome="success" http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hbase/layer-hbase/actions/smoke-test ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/hbase/layer-hbase/actions/smoke-test b/bigtop-packages/src/charm/hbase/layer-hbase/actions/smoke-test index bc6f45b..652b706 100755 --- a/bigtop-packages/src/charm/hbase/layer-hbase/actions/smoke-test +++ b/bigtop-packages/src/charm/hbase/layer-hbase/actions/smoke-test @@ -44,6 +44,6 @@ echo "HBase shell return code was: " $status if [ $status -eq 0 ]; then action-set outcome="success" else - action-set outcome="failed with return code ${status}" - action-fail "smoke-test failed" + action-set outcome="failure" + action-fail "smoke-test failed with return code ${status}" fi http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hbase/layer-hbase/metadata.yaml ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/hbase/layer-hbase/metadata.yaml b/bigtop-packages/src/charm/hbase/layer-hbase/metadata.yaml index 821f3fa..49fef47 100644 --- a/bigtop-packages/src/charm/hbase/layer-hbase/metadata.yaml +++ b/bigtop-packages/src/charm/hbase/layer-hbase/metadata.yaml @@ -1,9 +1,9 @@ name: hbase -summary: Apache Bitop HBase +summary: HBase from Apache Bigtop maintainer: Juju Big Data <[email protected]> description: > - HBase is the Hadoop database. This charm provides HBase from the - Apache Bigtop project. + HBase is the Hadoop database. This charm provides version 1.1.9 of the + HBase application from Apache Bigtop. tags: [] requires: zookeeper: http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/kafka/layer-kafka/README.md ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/README.md b/bigtop-packages/src/charm/kafka/layer-kafka/README.md index 93fca06..9d7f605 100644 --- a/bigtop-packages/src/charm/kafka/layer-kafka/README.md +++ b/bigtop-packages/src/charm/kafka/layer-kafka/README.md @@ -21,7 +21,8 @@ Software Foundation written in Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Learn more at [kafka.apache.org][]. -This charm deploys the Kafka component of the [Apache Bigtop][] platform. +This charm deploys version 0.10.1 of the Kafka component from +[Apache Bigtop][]. [kafka.apache.org]: http://kafka.apache.org/ [Apache Bigtop]: http://bigtop.apache.org/ @@ -29,9 +30,8 @@ This charm deploys the Kafka component of the [Apache Bigtop][] platform. # Deploying -A working Juju installation is assumed to be present. If Juju is not yet set -up, please follow the [getting-started][] instructions prior to deploying this -charm. +This charm requires Juju 2.0 or greater. If Juju is not yet set up, please +follow the [getting-started][] instructions prior to deploying this charm. Kafka requires the Zookeeper distributed coordination service. Deploy and relate them as follows: @@ -52,9 +52,6 @@ mirror options. See [Configuring Models][] for more information. # Using Once deployed, there are a number of actions available in this charm. -> **Note**: Actions described below assume Juju 2.0 or greater. If using an -earlier version of Juju, the action syntax is: -`juju action do kafka/0 <action_name> <action_args>; juju action fetch <id>`. List the zookeeper servers that our kafka brokers are connected to. The following will list `<ip>:<port>` information for each @@ -110,25 +107,16 @@ topics. Run the action as follows: juju run-action slave/0 smoke-test -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action do kafka/0 smoke-test`. - Watch the progress of the smoke test actions with: watch -n 2 juju show-action-status -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action status`. - Eventually, the action should settle to `status: completed`. If it reports `status: failed`, the application is not working as expected. Get more information about a specific smoke test with: juju show-action-output <action-id> -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action fetch <action-id>`. - # Scaling @@ -143,9 +131,6 @@ ready units, create a replicated topic as follows: juju run-action kafka/0 create-topic topic=my-replicated-topic \ partitions=1 replication=2 -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action do kafka/0 create-topic <args>`. - Query the description of the recently created topic: juju run --unit kafka/0 'kafka-topics.sh --describe \ @@ -191,9 +176,6 @@ network interface name or a CIDR range specifying a subnet. For example: juju config kafka network_interface=eth0 juju config kafka network_interface=10.0.2.0/24 -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju set-config kafka network_interface=eth0`. - Each kafka machine in the cluster will lookup the IP address of that network interface, or find the first network interface with an IP address in the specified subnet, and bind kafka to that address. @@ -205,16 +187,24 @@ run "juju resolved" on each unit: juju config kafka network_interface=eth0 juju resolved kafka/0 -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju set-config kafka network_interface=eth0; -juju resolved -r kafka/0`. - To go back to listening on any network interface on the machine, simply pass ``0.0.0.0`` to ``network_interface``. juju config kafka network_interface=0.0.0.0 +# Issues + +Apache Bigtop tracks issues using JIRA (Apache account required). File an +issue for this charm at: + +https://issues.apache.org/jira/secure/CreateIssue!default.jspa + +Ensure `Bigtop` is selected as the project. Typically, charm issues are filed +in the `deployment` component with the latest stable release selected as the +affected version. Any uncertain fields may be left blank. + + # Contact Information - <[email protected]> @@ -222,10 +212,10 @@ machine, simply pass ``0.0.0.0`` to ``network_interface``. # Resources +- [Apache Kafka home page](http://kafka.apache.org/) - [Apache Bigtop home page](http://bigtop.apache.org/) +- [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html) - [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html) -- [Apache Kafka home page](http://kafka.apache.org/) -- [Apache Kafka issue tracker](https://issues.apache.org/jira/browse/KAFKA) -- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop) +- [Juju Big Data](https://jujucharms.com/big-data) +- [Juju Bigtop charms](https://jujucharms.com/q/bigtop) - [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju) -- [Juju community](https://jujucharms.com/community) http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/kafka/layer-kafka/actions/create-topic ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/actions/create-topic b/bigtop-packages/src/charm/kafka/layer-kafka/actions/create-topic index b402fe8..159eac7 100755 --- a/bigtop-packages/src/charm/kafka/layer-kafka/actions/create-topic +++ b/bigtop-packages/src/charm/kafka/layer-kafka/actions/create-topic @@ -43,8 +43,9 @@ if host.service_available('kafka-server') and host.service_running('kafka-server '--replication-factor', topic_replication, capture_output=True) except subprocess.CalledProcessError as e: - kafkautils.fail('Kafka command failed', e.output) + kafkautils.fail('Kafka command failed: {}'.format(e.output)) else: - hookenv.action_set({'output': output}) + hookenv.action_set({'raw': output}) + hookenv.action_set({'outcome': 'success'}) else: - kafkautils.fail('Kafka service is not running', 'Please start kafka-server') + kafkautils.fail('kafka-server service is not running') http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/kafka/layer-kafka/actions/kafkautils.py ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/actions/kafkautils.py b/bigtop-packages/src/charm/kafka/layer-kafka/actions/kafkautils.py index 52e5299..8925027 100644 --- a/bigtop-packages/src/charm/kafka/layer-kafka/actions/kafkautils.py +++ b/bigtop-packages/src/charm/kafka/layer-kafka/actions/kafkautils.py @@ -19,8 +19,8 @@ import sys from charmhelpers.core import hookenv -def fail(msg, output='<No output>'): - hookenv.action_set({'output': output}) +def fail(msg): + hookenv.action_set({'outcome': 'failure'}) hookenv.action_fail(msg) sys.exit() http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-topics ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-topics b/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-topics index 0ef3a3f..0241586 100755 --- a/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-topics +++ b/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-topics @@ -35,8 +35,9 @@ if host.service_available('kafka-server') and host.service_running('kafka-server '--zookeeper', zookeepers, '--list', capture_output=True) except subprocess.CalledProcessError as e: - kafkautils.fail('Kafka command failed', e.output) + kafkautils.fail('Kafka command failed: {}'.format(e.output)) else: - hookenv.action_set({'output': output}) + hookenv.action_set({'raw': output}) + hookenv.action_set({'outcome': 'success'}) else: - kafkautils.fail('Kafka service is not running') + kafkautils.fail('kafka-server service is not running') http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-zks ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-zks b/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-zks index 517d60c..be9541a 100755 --- a/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-zks +++ b/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-zks @@ -22,15 +22,16 @@ from charms.reactive import is_state if not is_state('kafka.started'): - kafkautils.fail('Kafka service not yet ready', 'Please deploy kafka and required relations') + kafkautils.fail('Kafka service not yet ready') # List zookeepers if kafka is running if host.service_available('kafka-server') and host.service_running('kafka-server'): zookeepers = kafkautils.get_zookeepers() if zookeepers: - hookenv.action_set({'output': zookeepers}) + hookenv.action_set({'raw': zookeepers}) + hookenv.action_set({'outcome': 'success'}) else: - kafkautils.fail('No zookeeper.connect string found', 'Please relate kafka to zookeeper') + kafkautils.fail('No zookeeper.connect string found') else: - kafkautils.fail('Kafka service is not running', 'Please start kafka-server') + kafkautils.fail('kafka-server service is not running') http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/kafka/layer-kafka/actions/read-topic ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/actions/read-topic b/bigtop-packages/src/charm/kafka/layer-kafka/actions/read-topic index 8b7a27d..a1053e7 100755 --- a/bigtop-packages/src/charm/kafka/layer-kafka/actions/read-topic +++ b/bigtop-packages/src/charm/kafka/layer-kafka/actions/read-topic @@ -47,8 +47,9 @@ if host.service_available('kafka-server') and host.service_running('kafka-server '--no-wait-at-logend', capture_output=True) except subprocess.CalledProcessError as e: - kafkautils.fail('Kafka command failed', e.output) + kafkautils.fail('Kafka command failed: {}'.format(e.output)) else: - hookenv.action_set({'output': output}) + hookenv.action_set({'raw': output}) + hookenv.action_set({'outcome': 'success'}) else: - kafkautils.fail('Kafka service is not running') + kafkautils.fail('kafka-server service is not running') http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/kafka/layer-kafka/actions/smoke-test ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/actions/smoke-test b/bigtop-packages/src/charm/kafka/layer-kafka/actions/smoke-test index 5c1435a..a6cd3b6 100755 --- a/bigtop-packages/src/charm/kafka/layer-kafka/actions/smoke-test +++ b/bigtop-packages/src/charm/kafka/layer-kafka/actions/smoke-test @@ -21,6 +21,7 @@ import subprocess from charmhelpers.core import hookenv, host from charms.reactive import is_state from jujubigdata.utils import run_as +from time import time if not is_state('kafka.started'): @@ -28,7 +29,7 @@ if not is_state('kafka.started'): # Define smoke test params -topic_name = "smoketest" +topic_name = "smoketest_{}".format(int(time())) topic_partitions = 1 topic_replication = 1 @@ -48,7 +49,7 @@ if host.service_available('kafka-server') and host.service_running('kafka-server '--replication-factor', topic_replication, capture_output=True) except subprocess.CalledProcessError as e: - kafkautils.fail('Kafka command failed', e.output) + kafkautils.fail('Kafka command failed: {}'.format(e.output)) # List topics try: @@ -56,9 +57,9 @@ if host.service_available('kafka-server') and host.service_running('kafka-server '--zookeeper', zookeepers, '--list', capture_output=True) except subprocess.CalledProcessError as e: - kafkautils.fail('Kafka command failed', e.output) + kafkautils.fail('Kafka command failed: {}'.format(e.output)) # If we haven't failed yet, we passed hookenv.action_set({'outcome': 'success'}) else: - kafkautils.fail('Kafka service is not running') + kafkautils.fail('kafka-server service is not running') http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/kafka/layer-kafka/actions/write-topic ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/actions/write-topic b/bigtop-packages/src/charm/kafka/layer-kafka/actions/write-topic index 6a56265..813c541 100755 --- a/bigtop-packages/src/charm/kafka/layer-kafka/actions/write-topic +++ b/bigtop-packages/src/charm/kafka/layer-kafka/actions/write-topic @@ -46,8 +46,9 @@ if host.service_available('kafka-server') and host.service_running('kafka-server capture_output=True, input=bytes(data, 'UTF-8')) except subprocess.CalledProcessError as e: - kafkautils.fail('Kafka command failed', e.output) + kafkautils.fail('Kafka command failed: {}'.format(e.output)) else: - hookenv.action_set({'output': output}) + hookenv.action_set({'raw': output}) + hookenv.action_set({'outcome': 'success'}) else: - kafkautils.fail('Kafka service is not running') + kafkautils.fail('kafka-server service is not running') http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/kafka/layer-kafka/metadata.yaml ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/metadata.yaml b/bigtop-packages/src/charm/kafka/layer-kafka/metadata.yaml index 5a11394..ffd64f4 100644 --- a/bigtop-packages/src/charm/kafka/layer-kafka/metadata.yaml +++ b/bigtop-packages/src/charm/kafka/layer-kafka/metadata.yaml @@ -1,26 +1,11 @@ name: kafka -summary: High-performance distributed messaging system from Apache Bigtop +summary: Kafka from Apache Bigtop maintainer: Juju Big Data <[email protected]> -description: | - Fast - A single Kafka broker can handle hundreds of megabytes of reads and writes per - second from thousands of clients. +description: > + Kafka is a high-performance, scalable, distributed messaging system. - Scalable - Kafka is designed to allow a single cluster to serve as the central data - backbone for a large organization. It can be elastically and transparently - expanded without downtime. Data streams are partitioned and spread over a - cluster of machines to allow data streams larger than the capability of any - single machine and to allow clusters of co-ordinated consumers. - - Durable - Messages are persisted on disk and replicated within the cluster to prevent - data loss. Each broker can handle terabytes of messages without performance - impact. - - Distributed by Design - Kafka has a modern cluster-centric design that offers strong durability and - fault-tolerance guarantees. + This charm provides version 0.10.1 of the Kafka application from Apache + Bigtop. tags: [] provides: client: http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/mahout/layer-mahout/README.md ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/mahout/layer-mahout/README.md b/bigtop-packages/src/charm/mahout/layer-mahout/README.md index 2ecbf5a..2b64b20 100644 --- a/bigtop-packages/src/charm/mahout/layer-mahout/README.md +++ b/bigtop-packages/src/charm/mahout/layer-mahout/README.md @@ -16,33 +16,27 @@ --> # Overview -The Apache Mahout project's goal is to build an environment for quickly creating -scalable performant machine learning applications. +The Apache Mahout project's goal is to build an environment for quickly +creating scalable, performant machine learning applications. Learn more at +[mahout.apache.org][]. -Apache Mahout software provides three major features: - * A simple and extensible programming environment and framework for building - scalable algorithms - * A wide variety of premade algorithms for Scala + Apache Spark, H2O, Apache - Flink - * Samsara, a vector math experimentation environment with R-like syntax which - works at scale +This charm deploys version 0.12.2 of the Mahout component from +[Apache Bigtop][]. + +[mahout.apache.org]: http://mahout.apache.org/ +[Apache Bigtop]: http://bigtop.apache.org/ # Deploying -A working Juju installation is assumed to be present. If Juju is not yet set -up, please follow the [getting-started][] instructions prior to deploying this -charm. +This charm requires Juju 2.0 or greater. If Juju is not yet set up, please +follow the [getting-started][] instructions prior to deploying this charm. This charm is intended to be used with one of the [apache bigtop bundles][]. For example: juju deploy hadoop-processing -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart -hadoop-processing`. - This will deploy an Apache Bigtop Hadoop cluster. More information about this deployment can be found in the [bundle readme](https://jujucharms.com/hadoop-processing/). @@ -58,7 +52,6 @@ mirror options. See [Configuring Models][] for more information. [getting-started]: https://jujucharms.com/docs/stable/getting-started [apache bigtop bundles]: https://jujucharms.com/u/bigdata-charmers/#bundles -[juju-quickstart]: https://launchpad.net/juju-quickstart [Configuring Models]: https://jujucharms.com/docs/stable/models-config @@ -85,24 +78,27 @@ application is functioning as expected. Run the action as follows: juju run-action mahout/0 smoke-test -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action do mahout/0 smoke-test`. - Watch the progress of the smoke test actions with: watch -n 2 juju show-action-status -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action status`. - Eventually, the action should settle to `status: completed`. If it reports `status: failed`, the application is not working as expected. Get more information about a specific smoke test with: juju show-action-output <action-id> -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action fetch <action-id>`. + +# Issues + +Apache Bigtop tracks issues using JIRA (Apache account required). File an +issue for this charm at: + +https://issues.apache.org/jira/secure/CreateIssue!default.jspa + +Ensure `Bigtop` is selected as the project. Typically, charm issues are filed +in the `deployment` component with the latest stable release selected as the +affected version. Any uncertain fields may be left blank. # Contact Information @@ -112,10 +108,10 @@ of Juju, the syntax is `juju action fetch <action-id>`. # Resources +- [Apache Mahout home page](http://mahout.apache.org/) - [Apache Bigtop home page](http://bigtop.apache.org/) +- [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html) - [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html) -- [Apache Mahout home page](https://mahout.apache.org/) -- [Apache Mahout issue tracker](https://issues.apache.org/jira/browse/MAHOUT) -- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop) +- [Juju Big Data](https://jujucharms.com/big-data) +- [Juju Bigtop charms](https://jujucharms.com/q/bigtop) - [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju) -- [Juju community](https://jujucharms.com/community) http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/mahout/layer-mahout/actions/smoke-test ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/mahout/layer-mahout/actions/smoke-test b/bigtop-packages/src/charm/mahout/layer-mahout/actions/smoke-test index cc98b46..f903da4 100755 --- a/bigtop-packages/src/charm/mahout/layer-mahout/actions/smoke-test +++ b/bigtop-packages/src/charm/mahout/layer-mahout/actions/smoke-test @@ -62,3 +62,4 @@ hadoop jar /usr/lib/mahout/mahout-mr-*-job.jar \ -s SIMILARITY_LOGLIKELIHOOD 2>&1 | tee -a ${RESULT_LOG} EOF echo 'mahout smoke-test complete' +action-set outcome="success" http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/mahout/layer-mahout/metadata.yaml ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/mahout/layer-mahout/metadata.yaml b/bigtop-packages/src/charm/mahout/layer-mahout/metadata.yaml index 7a71d70..bf8ee0b 100644 --- a/bigtop-packages/src/charm/mahout/layer-mahout/metadata.yaml +++ b/bigtop-packages/src/charm/mahout/layer-mahout/metadata.yaml @@ -1,9 +1,12 @@ name: mahout -summary: Create scalable and performant machine learning applications +summary: Mahout from Apache Bigtop maintainer: Juju Big Data <[email protected]> description: > The Apache Mahout project's goal is to build an environment for quickly - creating scalable and performant machine learning applications. + creating scalable, performant machine learning applications. + + This charm provides version 0.12.2 of the Mahout application from Apache + Bigtop. tags: [] subordinate: true requires: http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/pig/layer-pig/README.md ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/pig/layer-pig/README.md b/bigtop-packages/src/charm/pig/layer-pig/README.md index 2334876..f38763d 100644 --- a/bigtop-packages/src/charm/pig/layer-pig/README.md +++ b/bigtop-packages/src/charm/pig/layer-pig/README.md @@ -19,35 +19,24 @@ Apache Pig is a platform for creating MapReduce programs used with Hadoop. It consists of a high-level language (Pig Latin) for expressing data analysis programs, coupled with infrastructure for evaluating these programs. Learn more -at [pig.apache.org](http://pig.apache.org). +at [pig.apache.org][]. -This charm deploys the Pig component of the Apache Bigtop platform and -supports running Pig in two execution modes: +This charm deploys version 0.15.0 of the Pig component from [Apache Bigtop][]. - * Local Mode: Pig runs using your local host and file system. Specify local - mode using the -x flag: `pig -x local` - * Mapreduce Mode: Pig runs using a Hadoop cluster and HDFS. This is the default - mode; you can, optionally, specify it using the -x flag: - `pig` or `pig -x mapreduce` +[pig.apache.org]: http://pig.apache.org/ +[Apache Bigtop]: http://bigtop.apache.org/ -# Deploying / Using +# Deploying -A working Juju installation is assumed to be present. If Juju is not yet set -up, please follow the -[getting-started](https://jujucharms.com/docs/2.0/getting-started) -instructions prior to deploying this charm. +This charm requires Juju 2.0 or greater. If Juju is not yet set up, please +follow the [getting-started][] instructions prior to deploying this charm. -This charm is intended to be used with one of the -[apache bigtop bundles](https://jujucharms.com/u/bigdata-charmers/#bundles). +This charm is intended to be deployed via one of the [apache bigtop bundles][]. For example: juju deploy hadoop-processing -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, use [juju-quickstart](https://launchpad.net/juju-quickstart) with the -following syntax: `juju quickstart hadoop-processing`. - This will deploy an Apache Bigtop Hadoop cluster. More information about this deployment can be found in the [bundle readme](https://jujucharms.com/hadoop-processing/). @@ -56,21 +45,14 @@ Now add Pig and relate it to the cluster via the hadoop-plugin: juju deploy pig juju add-relation pig plugin -Once deployment is complete, Apache Pig will be available to execute Pig Latin -jobs on your data. You can run Pig in a variety of modes: - -## Local Mode -Run Pig in local mode on the Pig unit with the following: - - juju ssh pig/0 - pig -x local - -## MapReduce Mode -MapReduce mode is the default for Pig. To run in this mode, ssh to the Pig unit -and run pig as follows: +## Network-Restricted Environments +Charms can be deployed in environments with limited network access. To deploy +in this environment, configure a Juju model with appropriate proxy and/or +mirror options. See [Configuring Models][] for more information. - juju ssh pig/0 - pig +[getting-started]: https://jujucharms.com/docs/stable/getting-started +[apache bigtop bundles]: https://jujucharms.com/u/bigdata-charmers/#bundles +[Configuring Models]: https://jujucharms.com/docs/stable/models-config # Verifying @@ -84,7 +66,7 @@ are ready: This is particularly useful when combined with `watch` to track the on-going progress of the deployment: - watch -n 0.5 juju status + watch -n 2 juju status The message column will provide information about a given unit's state. This charm is ready for use once the status message indicates that it is @@ -96,15 +78,9 @@ application is functioning as expected. Run the action as follows: juju run-action pig/0 smoke-test -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action do pig/0 smoke-test`. - Watch the progress of the smoke test actions with: - watch -n 0.5 juju show-action-status - -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action status`. + watch -n 2 juju show-action-status Eventually, the action should settle to `status: completed`. If it reports `status: failed`, the application is not working as expected. Get @@ -112,17 +88,36 @@ more information about a specific smoke test with: juju show-action-output <action-id> -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action fetch <action-id>`. +# Using -# Network-Restricted Environments +Once the deployment has been verified, Apache Pig will be available to execute +Pig Latin jobs on your data. You can run Pig in a variety of modes: -Charms can be deployed in environments with limited network access. To deploy -in this environment, configure a Juju model with appropriate -proxy and/or mirror options. See -[Configuring Models](https://jujucharms.com/docs/2.0/models-config) for more -information. +## Local Mode +Run Pig in local mode on the Pig unit with the following: + + juju ssh pig/0 + pig -x local + +## MapReduce Mode +MapReduce mode is the default for Pig. To run in this mode, ssh to the Pig unit +and run pig as follows: + + juju ssh pig/0 + pig + + +# Issues + +Apache Bigtop tracks issues using JIRA (Apache account required). File an +issue for this charm at: + +https://issues.apache.org/jira/secure/CreateIssue!default.jspa + +Ensure `Bigtop` is selected as the project. Typically, charm issues are filed +in the `deployment` component with the latest stable release selected as the +affected version. Any uncertain fields may be left blank. # Contact Information @@ -132,9 +127,10 @@ information. # Resources +- [Apache Pig home page](http://pig.apache.org/) - [Apache Bigtop home page](http://bigtop.apache.org/) - [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html) - [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html) -- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop) +- [Juju Big Data](https://jujucharms.com/big-data) +- [Juju Bigtop charms](https://jujucharms.com/q/bigtop) - [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju) -- [Juju community](https://jujucharms.com/community) http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/pig/layer-pig/actions/smoke-test ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/pig/layer-pig/actions/smoke-test b/bigtop-packages/src/charm/pig/layer-pig/actions/smoke-test index 378d3da..87f128d 100755 --- a/bigtop-packages/src/charm/pig/layer-pig/actions/smoke-test +++ b/bigtop-packages/src/charm/pig/layer-pig/actions/smoke-test @@ -18,16 +18,15 @@ import sys sys.path.append('lib') -from charmhelpers.core import hookenv -from charms.layer.apache_bigtop_base import Bigtop -from charms.reactive import is_state - -from jujubigdata.utils import run_as - -from path import Path +from charmhelpers.core import hookenv # noqa: E402 +from charms.layer.apache_bigtop_base import Bigtop # noqa: E402 +from charms.reactive import is_state # noqa: E402 +from jujubigdata.utils import run_as # noqa: E402 +from path import Path # noqa: E402 def fail(msg): + hookenv.action_set({'outcome': 'failure'}) hookenv.action_fail(msg) sys.exit() @@ -63,8 +62,8 @@ else: # The smoke tests analyze /etc/passwd. Successful runs will have an 'ubuntu' # passwd entry in the output. +hookenv.action_set({'raw': output}) if 'ubuntu' not in output: - hookenv.action_set({'output': output}) fail('Unable to analyze passwd file with Pig') else: hookenv.action_set({'outcome': 'success'}) http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/pig/layer-pig/metadata.yaml ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/pig/layer-pig/metadata.yaml b/bigtop-packages/src/charm/pig/layer-pig/metadata.yaml index 2898c6a..4aa1f66 100644 --- a/bigtop-packages/src/charm/pig/layer-pig/metadata.yaml +++ b/bigtop-packages/src/charm/pig/layer-pig/metadata.yaml @@ -1,9 +1,10 @@ name: pig -summary: Create Pig Latin programs for data warehouse analysis +summary: Pig from Apache Bigtop maintainer: Juju Big Data <[email protected]> description: | Apache Pig is a platform for creating MapReduce programs used with Hadoop. It consists of a high-level language (Pig Latin) for expressing data analysis programs, coupled with infrastructure for evaluating these programs. - Learn more at http://pig.apache.org. + + This charm provides version 0.15.0 of the Pig application from Apache Bigtop. tags: [] http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/spark/layer-spark/README.md ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/spark/layer-spark/README.md b/bigtop-packages/src/charm/spark/layer-spark/README.md index 6a55c82..c482ea9 100644 --- a/bigtop-packages/src/charm/spark/layer-spark/README.md +++ b/bigtop-packages/src/charm/spark/layer-spark/README.md @@ -17,36 +17,18 @@ # Overview Apache Spark is a fast and general purpose engine for large-scale data -processing. This charm deploys the Spark component of the [Apache Bigtop][] -platform. Key features: +processing. Learn more at [spark.apache.org][]. - * **Speed** - - Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster - on disk. Spark has an advanced DAG execution engine that supports cyclic data - flow and in-memory computing. - - * **Ease of Use** - - Write applications quickly in Java, Scala or Python. Spark offers over 80 - high-level operators that make it easy to build parallel apps for use - interactively from the Scala and Python shells. - - * **General Purpose Engine** - - Combine SQL, streaming, and complex analytics. Spark powers a stack of - high-level tools including Shark for SQL, MLlib for machine learning, GraphX, - and Spark Streaming. Combine these frameworks seamlessly in the same - application. +This charm deploys version 2.1.0 of the Spark component from [Apache Bigtop][]. +[spark.apache.org]: http://spark.apache.org/ [Apache Bigtop]: http://bigtop.apache.org/ # Deploying -A working Juju installation is assumed to be present. If Juju is not yet set -up, please follow the [getting-started][] instructions prior to deploying this -charm. +This charm requires Juju 2.0 or greater. If Juju is not yet set up, please +follow the [getting-started][] instructions prior to deploying this charm. This charm supports running Spark in a variety of modes: @@ -82,26 +64,17 @@ the unit acting as master, query Zookeeper as follows: ### YARN-client and YARN-cluster This charm leverages our pluggable Hadoop model with the `hadoop-plugin` -interface. This means that this charm can be related to a base Apache Hadoop +interface. This means that this charm can be related to an Apache Hadoop cluster to run Spark jobs there. The suggested deployment method is to use the -[hadoop-processing][] bundle and add a relation between spark and the plugin. +[hadoop-spark][] bundle: - - juju deploy hadoop-processing - juju add-relation plugin spark - -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart -hadoop-processing`. + juju deploy hadoop-spark To switch among the above execution modes, set the `spark_execution_mode` config variable: juju config spark spark_execution_mode=<new_mode> -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju set spark spark_execution_mode=<new_mode>`. - See the **Configuring** section below for supported mode options. ## Network-Restricted Environments @@ -110,8 +83,7 @@ in this environment, configure a Juju model with appropriate proxy and/or mirror options. See [Configuring Models][] for more information. [getting-started]: https://jujucharms.com/docs/stable/getting-started -[hadoop-processing]: https://jujucharms.com/hadoop-processing/ -[juju-quickstart]: https://launchpad.net/juju-quickstart +[hadoop-spark]: https://jujucharms.com/hadoop-spark/ [Configuring Models]: https://jujucharms.com/docs/stable/models-config @@ -138,25 +110,16 @@ application is functioning as expected. Run the action as follows: juju run-action spark/0 smoke-test -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action do spark/0 smoke-test`. - Watch the progress of the smoke test actions with: watch -n 2 juju show-action-status -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action status`. - Eventually, the action should settle to `status: completed`. If it reports `status: failed`, the application is not working as expected. Get more information about a specific smoke test with: juju show-action-output <action-id> -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action fetch <action-id>`. - ## Spark Master web UI Spark provides a web console that can be used to verify information about the cluster. To access it, find the `PUBLIC-ADDRESS` of the spark application @@ -270,27 +233,16 @@ the `spark_execution_mode` config variable. # Benchmarking -This charm provides several benchmarks, including the [Spark Bench][] -benchmarking suite (if enabled), to gauge the performance of the environment. +This charm provides benchmarks to gauge the performance of the Spark cluster. Each benchmark is an action that can be run with `juju run-action`: - $ juju actions spark | grep Bench - connectedcomponent Run the Spark Bench ConnectedComponent benchmark. - decisiontree Run the Spark Bench DecisionTree benchmark. - kmeans Run the Spark Bench KMeans benchmark. - linearregression Run the Spark Bench LinearRegression benchmark. - logisticregression Run the Spark Bench LogisticRegression benchmark. - matrixfactorization Run the Spark Bench MatrixFactorization benchmark. - pagerank Run the Spark Bench PageRank benchmark. - pca Run the Spark Bench PCA benchmark. - pregeloperation Run the Spark Bench PregelOperation benchmark. - shortestpaths Run the Spark Bench ShortestPaths benchmark. - sql Run the Spark Bench SQL benchmark. - stronglyconnectedcomponent Run the Spark Bench StronglyConnectedComponent benchmark. - svdplusplus Run the Spark Bench SVDPlusPlus benchmark. - svm Run the Spark Bench SVM benchmark. - - $ juju run-action spark/0 svdplusplus + $ juju actions spark + ... + pagerank Calculate PageRank for a sample data set + sparkpi Calculate Pi + ... + + $ juju run-action spark/0 pagerank Action queued with id: 339cec1f-e903-4ee7-85ca-876fb0c3d28e $ juju show-action-output 339cec1f-e903-4ee7-85ca-876fb0c3d28e @@ -299,27 +251,27 @@ Each benchmark is an action that can be run with `juju run-action`: composite: direction: asc units: secs - value: "200.754000" - raw: | - SVDPlusPlus,2016-11-02-03:08:26,200.754000,85.974071,.428255,0,SVDPlusPlus-MLlibConfig,,,,,10,,,50000,4.0,1.3, - start: 2016-11-02T03:08:26Z - stop: 2016-11-02T03:11:47Z - results: - duration: - direction: asc - units: secs - value: "200.754000" - throughput: - direction: desc - units: x/sec - value: ".428255" + value: "83" + start: 2017-04-12T23:22:38Z + stop: 2017-04-12T23:24:01Z + output: '{''status'': ''completed''}' status: completed timing: - completed: 2016-11-02 03:11:48 +0000 UTC - enqueued: 2016-11-02 03:08:21 +0000 UTC - started: 2016-11-02 03:08:26 +0000 UTC + completed: 2017-04-12 23:24:02 +0000 UTC + enqueued: 2017-04-12 23:22:36 +0000 UTC + started: 2017-04-12 23:22:37 +0000 UTC + + +# Issues + +Apache Bigtop tracks issues using JIRA (Apache account required). File an +issue for this charm at: + +https://issues.apache.org/jira/secure/CreateIssue!default.jspa -[Spark Bench]: https://github.com/SparkTC/spark-bench +Ensure `Bigtop` is selected as the project. Typically, charm issues are filed +in the `deployment` component with the latest stable release selected as the +affected version. Any uncertain fields may be left blank. # Contact Information @@ -329,9 +281,10 @@ Each benchmark is an action that can be run with `juju run-action`: # Resources +- [Apache Spark home page](http://spark.apache.org/) - [Apache Bigtop home page](http://bigtop.apache.org/) - [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html) - [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html) -- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop) +- [Juju Big Data](https://jujucharms.com/big-data) +- [Juju Bigtop charms](https://jujucharms.com/q/bigtop) - [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju) -- [Juju community](https://jujucharms.com/community) http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/spark/layer-spark/actions/pagerank ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/spark/layer-spark/actions/pagerank b/bigtop-packages/src/charm/spark/layer-spark/actions/pagerank index b2d85e6..2650e74 100755 --- a/bigtop-packages/src/charm/spark/layer-spark/actions/pagerank +++ b/bigtop-packages/src/charm/spark/layer-spark/actions/pagerank @@ -21,17 +21,16 @@ import sys from path import Path from time import time -from charmhelpers.contrib.benchmark import Benchmark from charmhelpers.core import hookenv +from charms.benchmark import Benchmark from charms.reactive import is_state from jujubigdata import utils -def fail(msg, output): - print(msg) - hookenv.action_set({'output': output}) +def fail(msg): + hookenv.action_set({'outcome': 'failure'}) hookenv.action_fail(msg) - sys.exit(1) + sys.exit() def main(): @@ -39,7 +38,7 @@ def main(): if not is_state('spark.started'): msg = 'Spark is not started yet' - fail(msg, 'error') + fail(msg) # gather params and create dir to store results num_iter = hookenv.action_get('iterations') @@ -54,7 +53,7 @@ def main(): sample = "/home/ubuntu/SparkBench/PageRank/web-Google.txt" if not os.path.isfile(sample): msg = 'Could not find pagerank sample data' - fail('{}: {}'.format(msg, sample), 'error') + fail('{}: {}'.format(msg, sample)) # Benchmark input data is packed into our sparkbench.tgz, which makes # it available on all spark units. In yarn mode, however, the nodemanagers @@ -69,12 +68,12 @@ def main(): capture_output=True) except subprocess.CalledProcessError as e: msg = 'Unable to copy pagerank sample data to hdfs' - fail('{}: {}'.format(msg, e), 'error') + fail('{}: {}'.format(msg, e)) else: sample = "/user/ubuntu/web-Google.txt" else: msg = 'Spark is configured for yarn mode, but HDFS is not ready yet' - fail(msg, 'error') + fail(msg) # find jar location spark_home = "/usr/lib/spark" @@ -86,7 +85,7 @@ def main(): if not example_jar_path: msg = 'Could not find {}'.format(example_jar_name) - fail(msg, 'error') + fail(msg) print('Calculating PageRank') bench.start() @@ -107,14 +106,17 @@ def main(): stderr=subprocess.STDOUT) except subprocess.CalledProcessError as e: msg = 'SparkPageRank command failed: {}'.format(' '.join(arg_list)) - fail('{}: {}'.format(msg, e), 'error') + fail('{}: {}'.format(msg, e)) stop = int(time()) bench.finish() duration = stop - start - bench.set_composite_score(duration, 'secs') - subprocess.check_call(['benchmark-raw', result_log]) + bench.set_composite_score(duration, units='secs', direction='asc') + + # Tell the user how they can get the full result log + scp_log_msg = "juju scp {}:{} .".format(hookenv.local_unit(), result_log) + hookenv.action_set({'meta.fetchcmd': scp_log_msg}) with open(result_log) as log: success = False @@ -125,9 +127,9 @@ def main(): if not success: msg = 'Spark-submit failed to calculate pagerank' - fail(msg, 'error') + fail(msg) - hookenv.action_set({'output': {'status': 'completed'}}) + hookenv.action_set({'outcome': 'success'}) if __name__ == '__main__': http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/spark/layer-spark/actions/restart-spark-job-history-server ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/spark/layer-spark/actions/restart-spark-job-history-server b/bigtop-packages/src/charm/spark/layer-spark/actions/restart-spark-job-history-server index 411c335..331239a 100755 --- a/bigtop-packages/src/charm/spark/layer-spark/actions/restart-spark-job-history-server +++ b/bigtop-packages/src/charm/spark/layer-spark/actions/restart-spark-job-history-server @@ -15,22 +15,22 @@ # limitations under the License. import sys +from charmhelpers.core import hookenv, host +from charms.reactive import is_state -try: - from charmhelpers.core import host, hookenv, unitdata - from jujubigdata import utils - charm_ready = True -except ImportError: - charm_ready = False -if not charm_ready: - from subprocess import call - call(['action-fail', 'Spark service not yet ready']) - sys.exit(1) +def fail(msg): + hookenv.action_set({'outcome': 'failure'}) + hookenv.action_fail(msg) + sys.exit() + + +if not is_state('spark.started'): + msg = 'Spark is not started yet' + fail(msg) if not host.service_available('spark-history'): - from subprocess import call - call(['action-fail', 'Spark history service not available']) - sys.exit(1) + msg = 'Spark history service not available' + fail(msg) host.service_restart('spark-history-server') http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/spark/layer-spark/actions/sparkbench ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/spark/layer-spark/actions/sparkbench b/bigtop-packages/src/charm/spark/layer-spark/actions/sparkbench index 906a30e..3f626e3 100755 --- a/bigtop-packages/src/charm/spark/layer-spark/actions/sparkbench +++ b/bigtop-packages/src/charm/spark/layer-spark/actions/sparkbench @@ -16,8 +16,9 @@ set -ex if ! charms.reactive is_state 'spark.started'; then + action-set outcome="failure" action-fail 'Spark not yet ready' - exit + exit 1 fi # Do not call this script directly. Call it via one of the symlinks. The @@ -138,11 +139,14 @@ if [ -f "${SB_APPS}" ]; then # send raw data (benchmark-raw takes a file) echo ${DATA} > ${RESULT_LOG} benchmark-raw ${RESULT_LOG} + action-set outcome="success" else - echo "ERROR: Invalid benchmark (${BENCHMARK})" + action-set outcome="failure" + action-fail "ERROR: Invalid benchmark (${BENCHMARK})" exit 1 fi else - echo "ERROR: Could not find SparkBench application list" + action-set outcome="failure" + action-fail "ERROR: Could not find SparkBench application list" exit 1 fi http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/spark/layer-spark/actions/sparkpi ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/spark/layer-spark/actions/sparkpi b/bigtop-packages/src/charm/spark/layer-spark/actions/sparkpi index 99fded6..93a4f10 100755 --- a/bigtop-packages/src/charm/spark/layer-spark/actions/sparkpi +++ b/bigtop-packages/src/charm/spark/layer-spark/actions/sparkpi @@ -21,22 +21,23 @@ from path import Path from time import time import subprocess -from charmhelpers.contrib.benchmark import Benchmark from charmhelpers.core import hookenv +from charms.benchmark import Benchmark from charms.reactive import is_state -def fail(msg, output): - hookenv.action_set({'output': output}) +def fail(msg): + hookenv.action_set({'outcome': 'failure'}) hookenv.action_fail(msg) - sys.exit(1) + sys.exit() def main(): bench = Benchmark() if not is_state('spark.started'): - fail('Spark not yet ready', 'error') + msg = 'Spark is not started yet' + fail(msg) num_partitions = hookenv.action_get('partitions') or '' @@ -64,7 +65,8 @@ def main(): example_jar_path = os.path.join(root, example_jar_name) if not example_jar_path: - fail('could not find {}'.format(example_jar_name), 'error') + msg = 'Could not find {}'.format(example_jar_name) + fail(msg) with open(result_log, 'w') as log_file: arg_list = [ @@ -84,26 +86,32 @@ def main(): except subprocess.CalledProcessError as e: print('smoke test command failed: ') print('{}'.format(' '.join(arg_list))) - fail('spark-submit failed: {}'.format(e), 'error') + fail('spark-submit failed: {}'.format(e)) stop = int(time()) bench.finish() duration = stop - start - bench.set_composite_score(duration, 'secs') - subprocess.check_call(['benchmark-raw', result_log]) + bench.set_composite_score(duration, units='secs', direction='asc') + + # Tell the user how they can get the full result log + scp_log_msg = "juju scp {}:{} .".format(hookenv.local_unit(), result_log) + hookenv.action_set({'meta.fetchcmd': scp_log_msg}) with open(result_log) as log: success = False for line in log.readlines(): if 'Pi is roughly 3.1' in line: success = True + pi_calc = line break if not success: - fail('spark-submit did not calculate pi', 'error') + msg = 'Spark-submit failed to calculate pi' + fail(msg) - hookenv.action_set({'output': {'status': 'completed'}}) + hookenv.action_set({'meta.raw': pi_calc}) + hookenv.action_set({'outcome': 'success'}) if __name__ == '__main__': http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/spark/layer-spark/actions/start-spark-job-history-server ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/spark/layer-spark/actions/start-spark-job-history-server b/bigtop-packages/src/charm/spark/layer-spark/actions/start-spark-job-history-server index 9677a38..cc218db 100755 --- a/bigtop-packages/src/charm/spark/layer-spark/actions/start-spark-job-history-server +++ b/bigtop-packages/src/charm/spark/layer-spark/actions/start-spark-job-history-server @@ -15,22 +15,22 @@ # limitations under the License. import sys +from charmhelpers.core import hookenv, host +from charms.reactive import is_state -try: - from charmhelpers.core import host, hookenv, unitdata - from jujubigdata import utils - charm_ready = True -except ImportError: - charm_ready = False -if not charm_ready: - from subprocess import call - call(['action-fail', 'Spark service not yet ready']) - sys.exit(1) +def fail(msg): + hookenv.action_set({'outcome': 'failure'}) + hookenv.action_fail(msg) + sys.exit() + + +if not is_state('spark.started'): + msg = 'Spark is not started yet' + fail(msg) if not host.service_available('spark-history-server'): - from subprocess import call - call(['action-fail', 'Spark history service not available']) - sys.exit(1) + msg = 'Spark history service not available' + fail(msg) host.service_start('spark-history-server') http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/spark/layer-spark/actions/stop-spark-job-history-server ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/spark/layer-spark/actions/stop-spark-job-history-server b/bigtop-packages/src/charm/spark/layer-spark/actions/stop-spark-job-history-server index fbe41cf..368b1e9 100755 --- a/bigtop-packages/src/charm/spark/layer-spark/actions/stop-spark-job-history-server +++ b/bigtop-packages/src/charm/spark/layer-spark/actions/stop-spark-job-history-server @@ -15,22 +15,22 @@ # limitations under the License. import sys +from charmhelpers.core import hookenv, host +from charms.reactive import is_state -try: - from charmhelpers.core import host, hookenv, unitdata - from jujubigdata import utils - charm_ready = True -except ImportError: - charm_ready = False -if not charm_ready: - from subprocess import call - call(['action-fail', 'Spark service not yet ready']) - sys.exit(1) +def fail(msg): + hookenv.action_set({'outcome': 'failure'}) + hookenv.action_fail(msg) + sys.exit() + + +if not is_state('spark.started'): + msg = 'Spark is not started yet' + fail(msg) if not host.service_available('spark-history-server'): - from subprocess import call - call(['action-fail', 'Spark history service not available']) - sys.exit(1) + msg = 'Spark history service not available' + fail(msg) host.service_stop('spark-history-server') http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/spark/layer-spark/actions/submit ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/spark/layer-spark/actions/submit b/bigtop-packages/src/charm/spark/layer-spark/actions/submit index a25a7af..ece0b20 100755 --- a/bigtop-packages/src/charm/spark/layer-spark/actions/submit +++ b/bigtop-packages/src/charm/spark/layer-spark/actions/submit @@ -16,8 +16,9 @@ set -e if ! charms.reactive is_state 'spark.started'; then + action-set outcome="failure" action-fail 'Spark not yet ready' - exit + exit 1 fi py_files="$(action-get py-files)" http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/spark/layer-spark/metadata.yaml ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/spark/layer-spark/metadata.yaml b/bigtop-packages/src/charm/spark/layer-spark/metadata.yaml index efec6f0..0666429 100644 --- a/bigtop-packages/src/charm/spark/layer-spark/metadata.yaml +++ b/bigtop-packages/src/charm/spark/layer-spark/metadata.yaml @@ -1,11 +1,11 @@ name: spark -summary: Apache Spark from Apache Bigtop platform +summary: Spark from Apache Bigtop maintainer: Juju Big Data <[email protected]> description: > Apache Spark is a fast and general engine for large-scale data processing. - Learn more at http://spark.apache.org. -tags: ["applications"] + This charm provides version 2.1.0 of the Spark application from Apache Bigtop. +tags: ["analytics"] provides: benchmark: interface: benchmark http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/zeppelin/layer-zeppelin/README.md ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/zeppelin/layer-zeppelin/README.md b/bigtop-packages/src/charm/zeppelin/layer-zeppelin/README.md index c76ad1e..a303bc2 100644 --- a/bigtop-packages/src/charm/zeppelin/layer-zeppelin/README.md +++ b/bigtop-packages/src/charm/zeppelin/layer-zeppelin/README.md @@ -18,35 +18,25 @@ Apache Zeppelin is a web-based notebook that enables interactive data analytics. It allows for beautiful data-driven, interactive, and collaborative documents -with SQL, Scala and more. +with SQL, Scala and more. Learn more at [zeppelin.apache.org][]. -As a Multi-purpose Notebook, Apache Zeppelin is the place for interactive: - - * Data Ingestion - * Data Discovery - * Data Analytics - * Data Visualization & Collaboration - -This charm deploys the Zeppelin component of the [Apache Bigtop][] platform. +This charm deploys version 0.7.0 of the Zeppelin component from +[Apache Bigtop][]. +[zeppelin.apache.org]: http://zeppelin.apache.org/ [Apache Bigtop]: http://bigtop.apache.org/ # Deploying -A working Juju installation is assumed to be present. If Juju is not yet set -up, please follow the [getting-started][] instructions prior to deploying this -charm. +This charm requires Juju 2.0 or greater. If Juju is not yet set up, please +follow the [getting-started][] instructions prior to deploying this charm. This charm is intended to be deployed via one of the [apache bigtop bundles][]. For example: juju deploy hadoop-processing -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart -hadoop-processing`. - This will deploy an Apache Bigtop Hadoop cluster. More information about this deployment can be found in the [bundle readme](https://jujucharms.com/hadoop-processing/). @@ -72,7 +62,6 @@ mirror options. See [Configuring Models][] for more information. [getting-started]: https://jujucharms.com/docs/stable/getting-started [apache bigtop bundles]: https://jujucharms.com/u/bigdata-charmers/#bundles -[juju-quickstart]: https://launchpad.net/juju-quickstart [Configuring Models]: https://jujucharms.com/docs/stable/models-config @@ -99,24 +88,27 @@ application is functioning as expected. Run the action as follows: juju run-action zeppelin/0 smoke-test -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action do zeppelin/0 smoke-test`. - Watch the progress of the smoke test actions with: watch -n 2 juju show-action-status -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action status`. - Eventually, the action should settle to `status: completed`. If it reports `status: failed`, the application is not working as expected. Get more information about a specific smoke test with: juju show-action-output <action-id> -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action fetch <action-id>`. + +# Issues + +Apache Bigtop tracks issues using JIRA (Apache account required). File an +issue for this charm at: + +https://issues.apache.org/jira/secure/CreateIssue!default.jspa + +Ensure `Bigtop` is selected as the project. Typically, charm issues are filed +in the `deployment` component with the latest stable release selected as the +affected version. Any uncertain fields may be left blank. # Contact Information @@ -126,9 +118,10 @@ of Juju, the syntax is `juju action fetch <action-id>`. # Resources +- [Apache Zeppelin home page](http://zeppelin.apache.org/) - [Apache Bigtop home page](http://bigtop.apache.org/) - [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html) - [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html) -- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop) +- [Juju Big Data](https://jujucharms.com/big-data) +- [Juju Bigtop charms](https://jujucharms.com/q/bigtop) - [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju) -- [Juju community](https://jujucharms.com/community) http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/zeppelin/layer-zeppelin/actions/restart ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/zeppelin/layer-zeppelin/actions/restart b/bigtop-packages/src/charm/zeppelin/layer-zeppelin/actions/restart index bb37376..6e822b4 100755 --- a/bigtop-packages/src/charm/zeppelin/layer-zeppelin/actions/restart +++ b/bigtop-packages/src/charm/zeppelin/layer-zeppelin/actions/restart @@ -18,12 +18,13 @@ import sys sys.path.append('lib') -from charmhelpers.core import hookenv -from charms.reactive import is_state -from charms.layer.bigtop_zeppelin import Zeppelin +from charmhelpers.core import hookenv # noqa: E402 +from charms.reactive import is_state # noqa: E402 +from charms.layer.bigtop_zeppelin import Zeppelin # noqa: E402 def fail(msg): + hookenv.action_set({'outcome': 'failure'}) hookenv.action_fail(msg) sys.exit() http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/zeppelin/layer-zeppelin/actions/smoke-test ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/zeppelin/layer-zeppelin/actions/smoke-test b/bigtop-packages/src/charm/zeppelin/layer-zeppelin/actions/smoke-test index 0339d46..7305822 100755 --- a/bigtop-packages/src/charm/zeppelin/layer-zeppelin/actions/smoke-test +++ b/bigtop-packages/src/charm/zeppelin/layer-zeppelin/actions/smoke-test @@ -16,7 +16,6 @@ # limitations under the License. import sys -import json from time import sleep from urllib.parse import urljoin from operator import itemgetter @@ -28,6 +27,7 @@ from charms.reactive import is_state def fail(msg): + hookenv.action_set({'outcome': 'failure'}) hookenv.action_fail(msg) sys.exit() @@ -35,23 +35,13 @@ def fail(msg): if not is_state('zeppelin.installed'): fail('Zeppelin not yet ready') - notebook_id = '2A94M5J1Z' zep_addr = 'localhost' base_url = 'http://{}:9080/api/notebook/'.format(zep_addr) -interp_url = urljoin(base_url, 'interpreter/bind/%s' % notebook_id) job_url = urljoin(base_url, 'job/%s' % notebook_id) para_url = urljoin(base_url, '%s/paragraph/' % notebook_id) try: - # bind interpreters - resp = requests.get(interp_url, timeout=60) - resp.raise_for_status() - interpreters = resp.json() - interp_ids = list(map(itemgetter('id'), interpreters['body'])) - resp = requests.put(interp_url, data=json.dumps(interp_ids), timeout=60) - resp.raise_for_status() - # run notebook resp = requests.post(job_url, timeout=60) resp.raise_for_status() @@ -91,3 +81,5 @@ try: fail('Notebook failed: {}'.format(errmsg)) except requests.exceptions.RequestException as e: fail('Request failed: {}: {}'.format(e.request.url, e)) + +hookenv.action_set({'outcome': 'success'}) http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/zeppelin/layer-zeppelin/lib/charms/layer/bigtop_zeppelin.py ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/zeppelin/layer-zeppelin/lib/charms/layer/bigtop_zeppelin.py b/bigtop-packages/src/charm/zeppelin/layer-zeppelin/lib/charms/layer/bigtop_zeppelin.py index b600f4b..bb7cf08 100644 --- a/bigtop-packages/src/charm/zeppelin/layer-zeppelin/lib/charms/layer/bigtop_zeppelin.py +++ b/bigtop-packages/src/charm/zeppelin/layer-zeppelin/lib/charms/layer/bigtop_zeppelin.py @@ -16,16 +16,17 @@ import os import json import time +import requests import socket -from urllib.parse import urljoin -import requests from path import Path +from urllib.parse import urljoin -from jujubigdata import utils from charmhelpers.core import hookenv, host, unitdata from charms import layer from charms.layer.apache_bigtop_base import Bigtop +from charms.reactive import is_state +from jujubigdata import utils class Zeppelin(object): @@ -43,19 +44,67 @@ class Zeppelin(object): def install(self): ''' - Trigger the Bigtop puppet recipe that handles the Zepplin service. + Perform initial one-time setup, workaround upstream bugs, and + trigger puppet. ''' # Dirs are handled by the bigtop deb, so no need to call out to # dist_config to do that work. However, we want to adjust the # groups for the `ubuntu` user for better interaction with Juju. self.dist_config.add_users() + + # Set ports based on layer.yaml options self._add_override('zeppelin::server::server_port', self.dist_config.port('zeppelin')) self._add_override('zeppelin::server::web_socket_port', self.dist_config.port('zeppelin_web')) + + # Default spark to local mode on initial install. This will be + # reconfigured if/when hadoop or spark relations are made. + self._add_override('zeppelin::server::spark_master_url', 'local[*]') + + ########## + # BUG: BIGTOP-2742 + # Default zeppelin init script looks for the literal '$(hostname)' + # string. Symlink it so it exists before the apt install from puppet + # tries to start the service. + import subprocess + host = subprocess.check_output(['hostname']).decode('utf8').strip() + zepp_pid = '/var/run/zeppelin/zeppelin-zeppelin-{}.pid'.format(host) + utils.run_as('root', 'mkdir', '-p', '/var/run/zeppelin') + utils.run_as('root', 'ln', '-sf', + zepp_pid, + '/var/run/zeppelin/zeppelin-zeppelin-$(hostname).pid') + ########## + self.trigger_bigtop() + ########## + # BUG: BIGTOP-2742 + # Puppet apply will call systemctl daemon-reload, which removes the + # symlink we just created. Now that the bits are on disk, update the + # init script $(hostname) that caused this mess to begin with. + zepp_init_script = '/etc/init.d/zeppelin' + utils.re_edit_in_place(zepp_init_script, { + r'^# pidfile.*': '# pidfile: {}'.format(zepp_pid), + }) + utils.run_as('root', 'systemctl', 'daemon-reload') + self.restart() + self.wait_for_api(30) + ########## + + ########## + # BUG: BIGTOP-2154 + # The zep deb depends on spark-core and spark-python. However, because + # of the unholy requirement to have hive tightly coupled to spark, + # we need to ensure spark-datanucleus is installed. Do this after the + # initial install so the bigtop repo is available to us. + utils.run_as('root', 'apt-get', 'install', '-qy', 'spark-datanucleus') + ########## + def trigger_bigtop(self): + ''' + Trigger the Bigtop puppet recipe that handles the Zeppelin service. + ''' bigtop = Bigtop() overrides = unitdata.kv().getrange('zeppelin.bigtop.overrides.', strip=True) @@ -65,6 +114,7 @@ class Zeppelin(object): ], overrides=overrides, ) + bigtop.trigger_puppet() self.wait_for_api(30) @@ -109,6 +159,12 @@ class Zeppelin(object): utils.run_as('hdfs', 'hdfs', 'dfs', '-mkdir', '-p', '/user/zeppelin') utils.run_as('hdfs', 'hdfs', 'dfs', '-chown', 'zeppelin', '/user/zeppelin') + # If spark is ready, let it handle the spark_master_url. Otherwise, + # zepp is in local mode; set it to yarn-client since hadoop is here. + if not is_state('spark.ready'): + self._add_override('zeppelin::server::spark_master_url', 'yarn-client') + self.trigger_bigtop() + def configure_spark(self, master_url): ''' Configure the zeppelin spark interpreter http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/zeppelin/layer-zeppelin/metadata.yaml ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/zeppelin/layer-zeppelin/metadata.yaml b/bigtop-packages/src/charm/zeppelin/layer-zeppelin/metadata.yaml index f6f4be6..f07614d 100644 --- a/bigtop-packages/src/charm/zeppelin/layer-zeppelin/metadata.yaml +++ b/bigtop-packages/src/charm/zeppelin/layer-zeppelin/metadata.yaml @@ -1,10 +1,13 @@ name: zeppelin -summary: A web-based notebook that enables interactive data analytics. +summary: Zeppelin from Apache Bigtop maintainer: Juju Big Data <[email protected]> -description: | +description: > Apache Zeppelin is a web-based notebook that enables interactive data analytics. You can make beautiful data-driven, interactive, and collaborative documents with SQL, Scala and more. + + This charm provides version 0.7.0 of the Zeppelin application from Apache + Bigtop. tags: ["analytics"] provides: client: http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/zeppelin/layer-zeppelin/reactive/zeppelin.py ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/zeppelin/layer-zeppelin/reactive/zeppelin.py b/bigtop-packages/src/charm/zeppelin/layer-zeppelin/reactive/zeppelin.py index 6d37fdc..a04a59b 100644 --- a/bigtop-packages/src/charm/zeppelin/layer-zeppelin/reactive/zeppelin.py +++ b/bigtop-packages/src/charm/zeppelin/layer-zeppelin/reactive/zeppelin.py @@ -88,9 +88,8 @@ def configure_hadoop(hadoop): set_state('zeppelin.hadoop.configured') -@when('zeppelin.installed') +@when('zeppelin.installed', 'zeppelin.hadoop.configured') @when_not('hadoop.ready') -@when('zeppelin.hadoop.configured') def unconfigure_hadoop(): zeppelin = Zeppelin() zeppelin.remove_hadoop_notebooks() http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/zookeeper/layer-zookeeper/README.md ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/zookeeper/layer-zookeeper/README.md b/bigtop-packages/src/charm/zookeeper/layer-zookeeper/README.md index 9c92d73..a3d12f1 100644 --- a/bigtop-packages/src/charm/zookeeper/layer-zookeeper/README.md +++ b/bigtop-packages/src/charm/zookeeper/layer-zookeeper/README.md @@ -20,18 +20,19 @@ Apache ZooKeeper is a high-performance coordination service for distributed applications. It exposes common services such as naming, configuration management, synchronization, and group services in a simple interface. Use it off-the-shelf to implement consensus, group management, leader election, and -presence protocols. +presence protocols. Learn more at [zookeeper.apache.org][]. -This charm provides the Zookeeper component of the [Apache Bigtop][] platform. +This charm provides version 3.4.6 of the Zookeeper component from +[Apache Bigtop][]. +[zookeeper.apache.org]: http://zookeeper.apache.org/ [Apache Bigtop]: http://bigtop.apache.org/ # Deploying -A working Juju installation is assumed to be present. If Juju is not yet set -up, please follow the [getting-started][] instructions prior to deploying this -charm. +This charm requires Juju 2.0 or greater. If Juju is not yet set up, please +follow the [getting-started][] instructions prior to deploying this charm. Deploy a Zookeeper unit. With only one unit, the application will be running in `standalone` mode: @@ -55,9 +56,6 @@ network interface name or a CIDR range specifying a subnet. For example: juju config zookeeper network_interface=eth0 juju config zookeeper network_interface=10.0.2.0/24 -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju set-config zookeeper network_interface=eth0`. - Each zookeeper unit in the cluster will lookup the IP address of that network interface, or find the first network interface with an IP address in the specified subnet, and bind Zookeeper to that address. @@ -69,18 +67,11 @@ run "juju resolved" on any failed units: juju config zookeeper network_interface=eth0 juju resolved zookeeper/0 -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju set-config zookeeper network_interface=eth0; -juju resolved -r zookeeper/0`. - To go back to listening on all interfaces, configure zookeeper with `network_interface=0.0.0.0`: juju config zookeeper network_interface=0.0.0.0 -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju set-config zookeeper network_interface=0.0.0.0`. - # Verifying @@ -105,25 +96,16 @@ application is functioning as expected. Run the action as follows: juju run-action zookeeper/0 smoke-test -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action do zookeeper/0 smoke-test`. - Watch the progress of the smoke test actions with: watch -n 2 juju show-action-status -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action status`. - Eventually, the action should settle to `status: completed`. If it reports `status: failed`, the application is not working as expected. Get more information about a specific smoke test with: juju show-action-output <action-id> -> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action fetch <action-id>`. - ## Utilities This charm includes Zookeeper command line utilities that can also be used to verify that the application is running as expected. Check the status of the @@ -176,6 +158,18 @@ that require Zookeeper as follows: ZK_port = relation_get('port') +# Issues + +Apache Bigtop tracks issues using JIRA (Apache account required). File an +issue for this charm at: + +https://issues.apache.org/jira/secure/CreateIssue!default.jspa + +Ensure `Bigtop` is selected as the project. Typically, charm issues are filed +in the `deployment` component with the latest stable release selected as the +affected version. Any uncertain fields may be left blank. + + # Contact Information - <[email protected]> @@ -183,10 +177,10 @@ that require Zookeeper as follows: # Resources +- [Apache Zookeeper home page](http://zookeeper.apache.org/) - [Apache Bigtop home page](http://bigtop.apache.org/) +- [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html) - [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html) -- [Apache Zookeeper home page](https://zookeeper.apache.org/) -- [Apache Zookeeper issue tracker](https://issues.apache.org/jira/browse/ZOOKEEPER) -- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop) +- [Juju Big Data](https://jujucharms.com/big-data) +- [Juju Bigtop charms](https://jujucharms.com/q/bigtop) - [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju) -- [Juju community](https://jujucharms.com/community) http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/zookeeper/layer-zookeeper/actions/restart ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/zookeeper/layer-zookeeper/actions/restart b/bigtop-packages/src/charm/zookeeper/layer-zookeeper/actions/restart index 7bb5136..acbd448 100755 --- a/bigtop-packages/src/charm/zookeeper/layer-zookeeper/actions/restart +++ b/bigtop-packages/src/charm/zookeeper/layer-zookeeper/actions/restart @@ -22,18 +22,15 @@ from charmhelpers.core import hookenv # noqa: E402 from charms.layer.bigtop_zookeeper import Zookeeper # noqa: E402 from charms.reactive import is_state # noqa: E402 -LOG = hookenv.log - -LOG("starting restart handler.") - def main(): if not is_state('zookeeper.started'): + hookenv.action_set({'outcome': 'failure'}) hookenv.action_fail('Cannot restart: Zookeeper has not yet started!') - return + sys.exit() - LOG("starting restart handler main.") - hookenv.status_set('maintenance', 'restarting ...') + hookenv.log("starting restart handler main.") + hookenv.status_set('maintenance', 'restarting zookeeper') zookeeper = Zookeeper() zookeeper.install() hookenv.status_set('active', 'ready {}'.format(zookeeper.quorum_check())) http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/zookeeper/layer-zookeeper/actions/smoke-test ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/zookeeper/layer-zookeeper/actions/smoke-test b/bigtop-packages/src/charm/zookeeper/layer-zookeeper/actions/smoke-test index 6481462..0f72ca0 100755 --- a/bigtop-packages/src/charm/zookeeper/layer-zookeeper/actions/smoke-test +++ b/bigtop-packages/src/charm/zookeeper/layer-zookeeper/actions/smoke-test @@ -23,12 +23,12 @@ from charms.layer.apache_bigtop_base import Bigtop # noqa: E402 from charms.reactive import is_state # noqa: E402 -def fail(msg, output=None): - if output: - hookenv.action_set({'output': output}) +def fail(msg): + hookenv.action_set({'outcome': 'failure'}) hookenv.action_fail(msg) sys.exit() + if not is_state('zookeeper.started'): fail('Charm is not yet ready to run the Bigtop smoke test(s)') @@ -38,6 +38,6 @@ smoke_components = ['zookeeper'] bigtop = Bigtop() result = bigtop.run_smoke_tests(smoke_components) if result == 'success': - hookenv.action_set({'outcome': result}) + hookenv.action_set({'outcome': 'success'}) else: - fail('{} smoke tests failed'.format(smoke_components), result) + fail('{} smoke tests failed with: {}'.format(smoke_components, result)) http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/zookeeper/layer-zookeeper/metadata.yaml ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/zookeeper/layer-zookeeper/metadata.yaml b/bigtop-packages/src/charm/zookeeper/layer-zookeeper/metadata.yaml index d9818d7..a563775 100644 --- a/bigtop-packages/src/charm/zookeeper/layer-zookeeper/metadata.yaml +++ b/bigtop-packages/src/charm/zookeeper/layer-zookeeper/metadata.yaml @@ -1,12 +1,15 @@ name: zookeeper +summary: Zookeeper from Apache Bigtop maintainer: Juju Big Data <[email protected]> -summary: High-performance coordination service for distributed applications -description: | +description: > Apache ZooKeeper is a centralized, reliable application for maintaining configuration information, naming, synchronization, and group services. All of these kinds of services are used in some form or another by distributed applications. In order to install and configure Apache HBase and other Hadoop ecosystem components, you need ZooKeeper. + + This charm provides version 3.4.6 of the Zookeeper application from Apache + Bigtop. tags: [] provides: zookeeper:
