Kevin W Monroe has proposed merging lp:~bigdata-dev/charms/trusty/apache-hadoop-plugin/trunk into lp:charms/trusty/apache-hadoop-plugin.
Requested reviews: Kevin W Monroe (kwmonroe) For more details, see: https://code.launchpad.net/~bigdata-dev/charms/trusty/apache-hadoop-plugin/trunk/+merge/268670 -- Your team Juju Big Data Development is subscribed to branch lp:~bigdata-dev/charms/trusty/apache-hadoop-plugin/trunk.
=== modified file 'DEV-README.md' --- DEV-README.md 2015-06-29 14:23:33 +0000 +++ DEV-README.md 2015-08-20 23:12:25 +0000 @@ -27,11 +27,19 @@ Additionally, the `JAVA_HOME`, `HADOOP_HOME`, `HADOOP_CONF_DIR`, and other environment variables will be set via `/etc/environment`. This includes putting +<<<<<<< TREE the Hadoop bin and sbin directories on the `PATH`. There are [helpers](http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/files/head:/common/noarch/) in `charmhelpers.contrib.bigdata.utils` to assist with using the environment file. For example, to run the `hdfs` command to create a directory as the `ubuntu` user: +======= +the Hadoop bin and sbin directories on the `PATH`. There are +[helpers](https://git.launchpad.net/bigdata-data/tree/common/noarch) +in `charmhelpers.contrib.bigdata.utils` to assist with using the environment +file. For example, to run the `hdfs` command to create a directory as the +`ubuntu` user: +>>>>>>> MERGE-SOURCE from charmhelpers.contrib.bigdata.utils import run_as run_as('ubuntu', 'hdfs', 'dfs', '-mkdir', '-p', '/home/ubuntu/foo') @@ -102,10 +110,17 @@ ## Manual Deployment +<<<<<<< TREE The easiest way to deploy the core Apache Hadoop platform is to use one of the [apache bundles](https://jujucharms.com/u/bigdata-charmers/#bundles). However, to manually deploy the base Apache Hadoop platform without using one of the bundles, you can use the following: +======= +The easiest way to deploy an Apache Hadoop platform is to use one of +the [apache bundles](https://jujucharms.com/u/bigdata-charmers/#bundles). +However, to manually deploy the base Apache Hadoop platform without using one +of the bundles, you can use the following: +>>>>>>> MERGE-SOURCE juju deploy apache-hadoop-hdfs-master hdfs-master juju deploy apache-hadoop-hdfs-secondary secondary-namenode === modified file 'README.md' --- README.md 2015-06-18 17:12:11 +0000 +++ README.md 2015-08-20 23:12:25 +0000 @@ -22,7 +22,11 @@ If you wanted to also wanted to be able to analyze your data using Apache Pig, you could deploy it and attach it to the same plugin: +<<<<<<< TREE juju deploy cs:~bigdata-charmers/trusty/apache-pig pig +======= + juju deploy apache-pig pig +>>>>>>> MERGE-SOURCE juju add-relation plugin pig ## Benchmarking @@ -92,17 +96,19 @@ of these resources: sudo pip install jujuresources - juju resources fetch --all apache-hadoop-plugin/resources.yaml -d /tmp/resources - juju resources serve -d /tmp/resources + juju-resources fetch --all /path/to/resources.yaml -d /tmp/resources + juju-resources serve -d /tmp/resources This will fetch all of the resources needed by this charm and serve them via a -simple HTTP server. You can then set the `resources_mirror` config option to -have the charm use this server for retrieving resources. +simple HTTP server. The output from `juju-resources serve` will give you a +URL that you can set as the `resources_mirror` config option for this charm. +Setting this option will cause all resources required by this charm to be +downloaded from the configured URL. You can fetch the resources for all of the Apache Hadoop charms (`apache-hadoop-hdfs-master`, `apache-hadoop-yarn-master`, `apache-hadoop-compute-slave`, `apache-hadoop-plugin`, etc) into a single -directory and serve them all with a single `juju resources serve` instance. +directory and serve them all with a single `juju-resources serve` instance. ## Contact Information === modified file 'actions/parseTerasort.py' --- actions/parseTerasort.py 2015-06-08 20:05:37 +0000 +++ actions/parseTerasort.py 2015-08-20 23:12:25 +0000 @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!.venv/bin/python """ Simple script to parse cassandra-stress' transaction results and reformat them as JSON for sending back to juju === modified file 'actions/teragen' --- actions/teragen 2015-06-08 21:32:27 +0000 +++ actions/teragen 2015-08-20 23:12:25 +0000 @@ -3,6 +3,10 @@ SIZE=`action-get size` IN_DIR=`action-get indir` +# The plugin uses a venv to avoid conflicting with system bits; ensure our +# venv is activated. +. ${CHARM_DIR}/.venv/bin/activate + benchmark-start # NB: Escaped vars in the block below (e.g., \${HADOOP_HOME}) come from === modified file 'actions/terasort' --- actions/terasort 2015-06-08 21:32:27 +0000 +++ actions/terasort 2015-08-20 23:12:25 +0000 @@ -25,6 +25,9 @@ # from this outer scope su ubuntu << EOF . /etc/environment +# The plugin uses a venv to avoid conflicting with system bits; ensure our +# venv is activated. +. ${CHARM_DIR}/.venv/bin/activate mkdir -p /opt/terasort/results/$run @@ -45,4 +48,4 @@ EOF PATH=$OLDPATH -`cat /opt/terasort/results/$run/terasort.log | python $CHARM_DIR/actions/parseTerasort.py` +`cat /opt/terasort/results/$run/terasort.log | $CHARM_DIR/.venv/bin/python $CHARM_DIR/actions/parseTerasort.py` === modified file 'dist.yaml' --- dist.yaml 2015-04-16 15:45:57 +0000 +++ dist.yaml 2015-08-20 23:12:25 +0000 @@ -68,49 +68,3 @@ owner: 'hdfs' group: 'hadoop' perms: 0775 -ports: - # Ports that need to be exposed, overridden, or manually specified. - # Only expose ports serving a UI or external API (i.e., namenode and - # resourcemanager). Communication among units within the cluster does - # not need ports to be explicitly opened. - # If adding a port here, you will need to update - # charmhelpers.contrib.bigdata.handlers.apache or hooks/callbacks.py - # to ensure that it is supported. - namenode: - port: 8020 - exposed_on: 'hdfs-master' - nn_webapp_http: - port: 50070 - exposed_on: 'hdfs-master' - dn_webapp_http: - port: 50075 - exposed_on: 'compute-slave' - resourcemanager: - port: 8032 - exposed_on: 'yarn-master' - rm_webapp_http: - port: 8088 - exposed_on: 'yarn-master' - rm_log: - port: 19888 - nm_webapp_http: - port: 8042 - exposed_on: 'compute-slave' - jobhistory: - port: 10020 - jh_webapp_http: - port: 19888 - exposed_on: 'yarn-master' - # TODO: support SSL - #nn_webapp_https: - # port: 50470 - # exposed_on: 'hdfs-master' - #dn_webapp_https: - # port: 50475 - # exposed_on: 'compute-slave' - #rm_webapp_https: - # port: 8090 - # exposed_on: 'yarn-master' - #nm_webapp_https: - # port: 8044 - # exposed_on: 'compute-slave' === modified file 'hooks/callbacks.py' --- hooks/callbacks.py 2015-06-25 15:41:48 +0000 +++ hooks/callbacks.py 2015-08-20 23:12:25 +0000 @@ -24,28 +24,28 @@ def update_blocked_status(): if unitdata.kv().get('charm.active', False): return - rels = ( - ('Yarn', 'ResourceManager', ResourceManager()), + rels = [ ('HDFS', 'NameNode', NameNode()), - ) + ] missing_rel = [rel for rel, res, impl in rels if not impl.connected_units()] - missing_hosts = [rel for rel, res, impl in rels if not impl.am_i_registered()] - not_ready = [(rel, res) for rel, res, impl in rels if not impl.is_ready()] + rels.append(('Yarn', 'ResourceManager', ResourceManager())) + not_ready = [(rel, res) for rel, res, impl in rels if impl.connected_units() and not impl.is_ready()] + missing_hosts = [rel for rel, res, impl in rels if impl.connected_units() and not impl.am_i_registered()] if missing_rel: hookenv.status_set('blocked', 'Waiting for relation to %s master%s' % ( ' and '.join(missing_rel), 's' if len(missing_rel) > 1 else '', )), - elif missing_hosts: - hookenv.status_set('waiting', 'Waiting for /etc/hosts registration on %s' % ( - ' and '.join(missing_hosts), - )) elif not_ready: unready_rels, unready_ress = zip(*not_ready) hookenv.status_set('waiting', 'Waiting for %s to provide %s' % ( ' and '.join(unready_rels), ' and '.join(unready_ress), )) + elif missing_hosts: + hookenv.status_set('waiting', 'Waiting for /etc/hosts registration on %s' % ( + ' and '.join(missing_hosts), + )) def update_working_status(): @@ -56,5 +56,18 @@ def update_active_status(): - unitdata.kv().set('charm.active', True) - hookenv.status_set('active', 'Ready') + hdfs_ready = NameNode().is_ready() + yarn_connected = ResourceManager().connected_units() + yarn_ready = ResourceManager().is_ready() + if hdfs_ready and (not yarn_connected or yarn_ready): + unitdata.kv().set('charm.active', True) + hookenv.status_set('active', 'Ready%s' % ( + '' if yarn_ready else ' (HDFS only)' + )) + else: + clear_active_flag() + update_blocked_status() + + +def clear_active_flag(): + unitdata.kv().set('charm.active', False) === modified file 'hooks/common.py' --- hooks/common.py 2015-06-25 15:41:48 +0000 +++ hooks/common.py 2015-08-20 23:12:25 +0000 @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!.venv/bin/python # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at @@ -14,12 +14,21 @@ Common implementation for all hooks. """ +import os import jujuresources from charmhelpers.core import hookenv from charmhelpers.core import unitdata from charmhelpers.core import charmframework +# ensure that the venv is used for installing resources +# (venv is used to ensure library versions needed by plugin +# don't conflict with the charm the plugin is supporting) +os.environ['PATH'] = os.pathsep.join([ + os.path.join(hookenv.charm_dir(), '.venv/bin'), + os.environ['PATH']]) + + def bootstrap_resources(): """ Install required resources defined in resources.yaml @@ -51,7 +60,7 @@ # list of keys required to be in the dist.yaml client_reqs = ['vendor', 'hadoop_version', 'packages', 'groups', 'users', - 'dirs', 'ports'] + 'dirs'] dist_config = jujubigdata.utils.DistConfig(filename='dist.yaml', required_keys=client_reqs) hadoop = jujubigdata.handlers.HadoopBase(dist_config) @@ -71,26 +80,44 @@ ], }, { - 'name': 'plugin', + 'name': 'hdfs', 'provides': [ jujubigdata.relations.HadoopPlugin(), ], 'requires': [ hadoop.is_installed, + hdfs_relation, + ], + 'callbacks': [ + callbacks.update_working_status, + hdfs_relation.register_provided_hosts, + jujubigdata.utils.manage_etc_hosts, + hdfs.configure_client, + callbacks.update_active_status, + ], + 'cleanup': [ + callbacks.clear_active_flag, + callbacks.update_blocked_status, + ], + }, + { + 'name': 'yarn', + 'provides': [], + 'requires': [ + hadoop.is_installed, yarn_relation, - hdfs_relation, ], 'callbacks': [ callbacks.update_working_status, yarn_relation.register_provided_hosts, - hdfs_relation.register_provided_hosts, jujubigdata.utils.manage_etc_hosts, yarn.install_demo, yarn.configure_client, - hdfs.configure_client, callbacks.update_active_status, ], - 'cleanup': [], + 'cleanup': [ + callbacks.update_blocked_status, + ], }, ]) manager.manage() === modified file 'hooks/config-changed' --- hooks/config-changed 2015-02-09 18:13:28 +0000 +++ hooks/config-changed 2015-08-20 23:12:25 +0000 @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!.venv/bin/python # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at === modified file 'hooks/hadoop-plugin-relation-changed' --- hooks/hadoop-plugin-relation-changed 2015-04-28 13:38:46 +0000 +++ hooks/hadoop-plugin-relation-changed 2015-08-20 23:12:25 +0000 @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!.venv/bin/python # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at === added file 'hooks/hadoop-plugin-relation-departed' --- hooks/hadoop-plugin-relation-departed 1970-01-01 00:00:00 +0000 +++ hooks/hadoop-plugin-relation-departed 2015-08-20 23:12:25 +0000 @@ -0,0 +1,15 @@ +#!.venv/bin/python +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import common +common.manage() === modified file 'hooks/namenode-relation-changed' --- hooks/namenode-relation-changed 2015-05-07 15:27:21 +0000 +++ hooks/namenode-relation-changed 2015-08-20 23:12:25 +0000 @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!.venv/bin/python # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at === added file 'hooks/namenode-relation-departed' --- hooks/namenode-relation-departed 1970-01-01 00:00:00 +0000 +++ hooks/namenode-relation-departed 2015-08-20 23:12:25 +0000 @@ -0,0 +1,16 @@ +#!.venv/bin/python +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import common + +common.manage() === modified file 'hooks/resourcemanager-relation-changed' --- hooks/resourcemanager-relation-changed 2015-05-07 15:11:29 +0000 +++ hooks/resourcemanager-relation-changed 2015-08-20 23:12:25 +0000 @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!.venv/bin/python # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at === added file 'hooks/resourcemanager-relation-departed' --- hooks/resourcemanager-relation-departed 1970-01-01 00:00:00 +0000 +++ hooks/resourcemanager-relation-departed 2015-08-20 23:12:25 +0000 @@ -0,0 +1,16 @@ +#!.venv/bin/python +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import common + +common.manage() === modified file 'hooks/setup.py' --- hooks/setup.py 2015-06-25 15:41:48 +0000 +++ hooks/setup.py 2015-08-20 23:12:25 +0000 @@ -9,6 +9,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +import os import subprocess from glob import glob @@ -17,12 +18,15 @@ """ Do any setup required before the install hook. """ - install_pip() + setup_venv() install_bundled_resources() -def install_pip(): - subprocess.check_call(['apt-get', 'install', '-yq', 'python-pip', 'bzr']) +def setup_venv(): + if not os.path.exists('.venv/bin/python'): + subprocess.check_call(['apt-get', 'install', '-yq', 'python-virtualenv']) + subprocess.check_call(['virtualenv', '.venv']) + execfile('.venv/bin/activate_this.py', {'__file__': '.venv/bin/activate_this.py'}) def install_bundled_resources(): @@ -30,4 +34,4 @@ Install the bundled resources libraries. """ archives = glob('resources/python/*') - subprocess.check_call(['pip', 'install'] + archives) + subprocess.check_call(['.venv/bin/pip', 'install'] + archives) === modified file 'hooks/start' --- hooks/start 2015-02-09 18:13:28 +0000 +++ hooks/start 2015-08-20 23:12:25 +0000 @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!.venv/bin/python # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at === modified file 'hooks/stop' --- hooks/stop 2015-02-09 18:13:28 +0000 +++ hooks/stop 2015-08-20 23:12:25 +0000 @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!.venv/bin/python # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at === modified file 'resources.yaml' --- resources.yaml 2015-06-25 22:43:49 +0000 +++ resources.yaml 2015-08-20 23:12:25 +0000 @@ -4,7 +4,7 @@ pathlib: pypi: path.py>=7.0 jujubigdata: - pypi: jujubigdata>=2.0.2,<3.0.0 + pypi: jujubigdata>=4.0.0,<5.0.0 charm-benchmark: pypi: charm-benchmark>=1.0.1,<2.0.0 java-installer: @@ -12,19 +12,19 @@ # If replaced with an alternate implementation, it must output *only* two # lines containing the JAVA_HOME path, and the Java version, respectively, # on stdout. Upon error, it must exit with a non-zero exit code. - url: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/kevin.monroe%40canonical.com-20150625222410-qfck64q46ubx5i6p/javainstaller.sh-20150311213053-4vq7369jhlvc6qy8-1/java-installer.sh - hash: 8fdff60270ea4be7bbef1e013e503057fe6efc2f4b5761edebc206a54f303023 + url: https://git.launchpad.net/bigdata-data/plain/common/noarch/java-installer.sh?id=baa0b74b86587f97b446f255deb96c8420021dd8 + hash: f7df6937bdb4dcc60de559252b4e6b65c77959f871c7ef2e59af57832d7ddfca hash_type: sha256 optional_resources: hadoop-aarch64: - url: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/kevin.monroe%40canonical.com-20150303192631-swhrf8f7q82si75t/hadoop2.4.1.tar.gz-20150303192554-7gqslr4m8ahkwiax-2/hadoop-2.4.1.tar.gz + url: https://git.launchpad.net/bigdata-data/plain/apache/aarch64/hadoop-2.4.1.tar.gz?id=c34a21c939f5fce9ab89b95d65fe2df50e7bbab0 hash: 03ad135835bfe413f85fe176259237a8 hash_type: md5 hadoop-ppc64le: - url: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/kevin.monroe%40canonical.com-20150130165209-nuz1myezjpdx7eus/hadoop2.4.1ppc64le.t-20150130165148-s8i19s002ht88gio-2/hadoop-2.4.1-ppc64le.tar.gz + url: https://git.launchpad.net/bigdata-data/plain/apache/ppc64le/hadoop-2.4.1-ppc64le.tar.gz?id=c34a21c939f5fce9ab89b95d65fe2df50e7bbab0 hash: 09942b168a3db0d183b281477d3dae9deb7b7bc4b5783ba5cda3965b62e71bd5 hash_type: sha256 hadoop-x86_64: - url: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/cory.johns%40canonical.com-20150116154822-x5osw3zfhw6e03b1/hadoop2.4.1.tar.gz-20150116154748-yfa2j12rr5m53xd3-1/hadoop-2.4.1.tar.gz + url: https://git.launchpad.net/bigdata-data/plain/apache/x86_64/hadoop-2.4.1.tar.gz?id=c34a21c939f5fce9ab89b95d65fe2df50e7bbab0 hash: a790d39baba3a597bd226042496764e0520c2336eedb28a1a3d5c48572d3b672 hash_type: sha256 === removed file 'resources/python/jujuresources-0.2.8.tar.gz' Binary files resources/python/jujuresources-0.2.8.tar.gz 2015-06-25 15:41:48 +0000 and resources/python/jujuresources-0.2.8.tar.gz 1970-01-01 00:00:00 +0000 differ === added file 'resources/python/jujuresources-0.2.9.tar.gz' Binary files resources/python/jujuresources-0.2.9.tar.gz 1970-01-01 00:00:00 +0000 and resources/python/jujuresources-0.2.9.tar.gz 2015-08-20 23:12:25 +0000 differ
-- Mailing list: https://launchpad.net/~bigdata-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~bigdata-dev More help : https://help.launchpad.net/ListHelp
