BIGTOP-2482: Adding Pig charm (closes #121) Signed-off-by: Kevin W Monroe <[email protected]>
Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/41eecb9a Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/41eecb9a Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/41eecb9a Branch: refs/heads/master Commit: 41eecb9a1350206cbacbc6a7c60ce64c200d9f8e Parents: f7d471b Author: Kevin W Monroe <[email protected]> Authored: Tue Jun 7 21:59:24 2016 -0500 Committer: Kevin W Monroe <[email protected]> Committed: Sun Oct 9 15:13:47 2016 -0500 ---------------------------------------------------------------------- .../src/charm/pig/layer-pig/README.md | 140 ++ .../src/charm/pig/layer-pig/actions.yaml | 6 + .../src/charm/pig/layer-pig/actions/smoke-test | 70 + .../src/charm/pig/layer-pig/copyright | 16 + .../src/charm/pig/layer-pig/icon.svg | 1431 ++++++++++++++++++ .../src/charm/pig/layer-pig/layer.yaml | 13 + .../layer-pig/lib/charms/layer/bigtop_pig.py | 80 + .../src/charm/pig/layer-pig/metadata.yaml | 9 + .../src/charm/pig/layer-pig/reactive/pig.py | 37 + .../src/charm/pig/layer-pig/tests/01-deploy.py | 49 + .../src/charm/pig/layer-pig/tests/tests.yaml | 3 + 11 files changed, 1854 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/bigtop/blob/41eecb9a/bigtop-packages/src/charm/pig/layer-pig/README.md ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/pig/layer-pig/README.md b/bigtop-packages/src/charm/pig/layer-pig/README.md new file mode 100644 index 0000000..e174fd5 --- /dev/null +++ b/bigtop-packages/src/charm/pig/layer-pig/README.md @@ -0,0 +1,140 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +# Overview + +Apache Pig is a platform for creating MapReduce programs used with Hadoop. +It consists of a high-level language (Pig Latin) for expressing data analysis +programs, coupled with infrastructure for evaluating these programs. Learn more +at [pig.apache.org](http://pig.apache.org). + +This charm deploys the Pig component of the Apache Bigtop platform and +supports running Pig in two execution modes: + + * Local Mode: Pig runs using your local host and file system. Specify local + mode using the -x flag: `pig -x local` + * Mapreduce Mode: Pig runs using a Hadoop cluster and HDFS. This is the default + mode; you can, optionally, specify it using the -x flag: + `pig` or `pig -x mapreduce` + + +# Deploying / Using + +A working Juju installation is assumed to be present. If Juju is not yet set +up, please follow the +[getting-started](https://jujucharms.com/docs/2.0/getting-started) +instructions prior to deploying this charm. + +This charm is intended to be used with one of the +[apache bigtop bundles](https://jujucharms.com/u/bigdata-charmers/#bundles). +For example: + + juju deploy hadoop-processing + +> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, use [juju-quickstart](https://launchpad.net/juju-quickstart) with the +following syntax: `juju quickstart hadoop-processing`. + +This will deploy an Apache Bigtop Hadoop cluster. More information about this +deployment can be found in the [bundle readme](https://jujucharms.com/hadoop-processing/). + +Now add Pig and relate it to the cluster via the hadoop-plugin: + + juju deploy pig + juju add-relation pig plugin + +Once deployment is complete, Apache Pig will be available to execute Pig Latin +jobs on your data. You can run Pig in a variety of modes: + +## Local Mode +Run Pig in local mode on the Pig unit with the following: + + juju ssh pig/0 + pig -x local + +## MapReduce Mode +MapReduce mode is the default for Pig. To run in this mode, ssh to the Pig unit +and run pig as follows: + + juju ssh pig/0 + pig + + +# Verifying + +## Status +Apache Bigtop charms provide extended status reporting to indicate when they +are ready: + + juju status + +This is particularly useful when combined with `watch` to track the on-going +progress of the deployment: + + watch -n 0.5 juju status + +The message column will provide information about a given unit's state. +This charm is ready for use once the status message indicates that it is +ready. + +## Smoke Test +This charm provides a `smoke-test` action that can be used to verify the +application is functioning as expected. Run the action as follows: + + juju run-action pig/0 smoke-test + +> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action do pig/0 smoke-test`. + +Watch the progress of the smoke test actions with: + + watch -n 0.5 juju show-action-status + +> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action status`. + +Eventually, the action should settle to `status: completed`. If it +reports `status: failed`, the application is not working as expected. Get +more information about a specific smoke test with: + + juju show-action-output <action-id> + +> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action fetch <action-id>`. + + +# Network-Restricted Environments + +Charms can be deployed in environments with limited network access. To deploy +in this environment, configure a Juju model with appropriate +proxy and/or mirror options. See +[Configuring Models](https://jujucharms.com/docs/2.0/models-config) for more +information. + + +# Contact Information + +- <[email protected]> + + +# Resources + +- [Apache Bigtop](http://bigtop.apache.org/) home page +- [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html) +- [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html) +- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop) +- [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju) +- [Juju community](https://jujucharms.com/community) http://git-wip-us.apache.org/repos/asf/bigtop/blob/41eecb9a/bigtop-packages/src/charm/pig/layer-pig/actions.yaml ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/pig/layer-pig/actions.yaml b/bigtop-packages/src/charm/pig/layer-pig/actions.yaml new file mode 100644 index 0000000..f6e503e --- /dev/null +++ b/bigtop-packages/src/charm/pig/layer-pig/actions.yaml @@ -0,0 +1,6 @@ +smoke-test: + description: | + Verify the Bigtop Pig service is working by executing a smoke test. If + pig is in 'local' mode, the smoke test will execute a local /etc/passwd + analysis job. If pig is in 'mapreduce' mode, the Bigtop smoke test will + be executed, which exercises Pig with HDFS and MapReduce. http://git-wip-us.apache.org/repos/asf/bigtop/blob/41eecb9a/bigtop-packages/src/charm/pig/layer-pig/actions/smoke-test ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/pig/layer-pig/actions/smoke-test b/bigtop-packages/src/charm/pig/layer-pig/actions/smoke-test new file mode 100755 index 0000000..378d3da --- /dev/null +++ b/bigtop-packages/src/charm/pig/layer-pig/actions/smoke-test @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys +sys.path.append('lib') + +from charmhelpers.core import hookenv +from charms.layer.apache_bigtop_base import Bigtop +from charms.reactive import is_state + +from jujubigdata.utils import run_as + +from path import Path + + +def fail(msg): + hookenv.action_fail(msg) + sys.exit() + + +if not is_state('pig.installed'): + fail('Pig is not yet ready') + + +if is_state('hadoop.ready'): + hookenv.log('Testing Pig in mapreduce mode using the Bigtop smoke test') + # Remove any output from previous runs + run_as('ubuntu', + 'hdfs', 'dfs', '-rm', '-f', '-R', '/user/ubuntu/pig-output-wordcount') + + bigtop = Bigtop() + bigtop.run_smoke_tests(['pig']) + output = run_as('ubuntu', + 'hdfs', 'dfs', '-cat', 'pig-output-wordcount/part-r-00000', + capture_output=True) +else: + hookenv.log('Testing Pig in local mode (ignoring bigtop component smoke test)') + # Remove any output from previous runs + run_as('ubuntu', 'rm', '-R', '-f', '/tmp/test.pig.out') + + script = ("A = load '/etc/passwd' using PigStorage(':'); " + "B = foreach A generate $0 as id; store B into '/tmp/test.pig.out';") + Path('/tmp/test.pig').write_text(script) + run_as('ubuntu', 'pig', '-x', 'local', '-l', '/tmp/test.pig.log', '/tmp/test.pig') + + output = run_as('ubuntu', + 'cat', '/tmp/test.pig.out/part-m-00000', + capture_output=True) + +# The smoke tests analyze /etc/passwd. Successful runs will have an 'ubuntu' +# passwd entry in the output. +if 'ubuntu' not in output: + hookenv.action_set({'output': output}) + fail('Unable to analyze passwd file with Pig') +else: + hookenv.action_set({'outcome': 'success'}) http://git-wip-us.apache.org/repos/asf/bigtop/blob/41eecb9a/bigtop-packages/src/charm/pig/layer-pig/copyright ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/charm/pig/layer-pig/copyright b/bigtop-packages/src/charm/pig/layer-pig/copyright new file mode 100644 index 0000000..e900b97 --- /dev/null +++ b/bigtop-packages/src/charm/pig/layer-pig/copyright @@ -0,0 +1,16 @@ +Format: http://dep.debian.net/deps/dep5/ + +Files: * +Copyright: Copyright 2015, Canonical Ltd., All Rights Reserved. +License: Apache License 2.0 + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License.
