HBASE-18418 Remove apache_hbase_topology from dev-support
Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/3acb0817 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/3acb0817 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/3acb0817 Branch: refs/heads/HBASE-18410 Commit: 3acb081787a4289d86d977db26bedaf6a42172ce Parents: c16eb78 Author: Dima Spivak <dimaspi...@apache.org> Authored: Thu Jul 20 10:08:11 2017 -0700 Committer: Dima Spivak <dimaspi...@apache.org> Committed: Wed Oct 18 14:08:26 2017 -0700 ---------------------------------------------------------------------- dev-support/apache_hbase_topology/Dockerfile | 24 -- dev-support/apache_hbase_topology/README.md | 49 --- dev-support/apache_hbase_topology/__init__.py | 15 - dev-support/apache_hbase_topology/actions.py | 421 ------------------- .../apache_hbase_topology/configurations.cfg | 80 ---- dev-support/apache_hbase_topology/profile.cfg | 82 ---- dev-support/apache_hbase_topology/ssh/id_rsa | 44 -- .../apache_hbase_topology/ssh/id_rsa.pub | 18 - 8 files changed, 733 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/Dockerfile ---------------------------------------------------------------------- diff --git a/dev-support/apache_hbase_topology/Dockerfile b/dev-support/apache_hbase_topology/Dockerfile deleted file mode 100644 index 714a55c..0000000 --- a/dev-support/apache_hbase_topology/Dockerfile +++ /dev/null @@ -1,24 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -FROM debian:wheezy - -ENV TOPOLOGY_NAME=apache_hbase -ADD . /root/clusterdock/clusterdock/topologies/${TOPOLOGY_NAME} - -RUN find /root -type f -name id_rsa -exec chmod 600 {} \; - -VOLUME /root/clusterdock/clusterdock/topologies/${TOPOLOGY_NAME} -CMD ["/true"] http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/README.md ---------------------------------------------------------------------- diff --git a/dev-support/apache_hbase_topology/README.md b/dev-support/apache_hbase_topology/README.md deleted file mode 100644 index 018ee99..0000000 --- a/dev-support/apache_hbase_topology/README.md +++ /dev/null @@ -1,49 +0,0 @@ -<!--- - Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. ---> -# apache_hbase clusterdock topology - -## Overview -*clusterdock* is a framework for creating Docker-based container clusters. Unlike regular Docker -containers, which tend to run single processes and then exit once the process terminates, these -container clusters are characterized by the execution of an init process in daemon mode. As such, -the containers act more like "fat containers" or "light VMs;" entities with accessible IP addresses -which emulate standalone hosts. - -*clusterdock* relies upon the notion of a topology to define how clusters should be built into -images and then what to do with those images to start Docker container clusters. - -## Usage -The *clusterdock* framework is designed to be run out of its own container while affecting -operations on the host. To avoid problems that might result from incorrectly -formatting this framework invocation, a Bash helper script (`clusterdock.sh`) can be sourced on a -host that has Docker installed. Afterwards, running any of the binaries intended to carry -out *clusterdock* actions can be done using the `clusterdock_run` command. -``` -wget https://raw.githubusercontent.com/cloudera/clusterdock/master/clusterdock.sh -# ALWAYS INSPECT SCRIPTS FROM THE INTERNET BEFORE SOURCING THEM. -source clusterdock.sh -``` - -Since the *clusterdock* framework itself lives outside of Apache HBase, an environmental variable -is used to let the helper script know where to find an image of the *apache_hbase* topology. To -start a four-node Apache HBase cluster with default versions, you would simply run -``` -CLUSTERDOCK_TOPOLOGY_IMAGE=apache_hbase_topology_location clusterdock_run \ - ./bin/start_cluster apache_hbase --secondary-nodes='node-{2..4}' -``` http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/__init__.py ---------------------------------------------------------------------- diff --git a/dev-support/apache_hbase_topology/__init__.py b/dev-support/apache_hbase_topology/__init__.py deleted file mode 100644 index 635f0d9..0000000 --- a/dev-support/apache_hbase_topology/__init__.py +++ /dev/null @@ -1,15 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/actions.py ---------------------------------------------------------------------- diff --git a/dev-support/apache_hbase_topology/actions.py b/dev-support/apache_hbase_topology/actions.py deleted file mode 100644 index 26566e0..0000000 --- a/dev-support/apache_hbase_topology/actions.py +++ /dev/null @@ -1,421 +0,0 @@ -#!/usr/bin/env python -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""The actions module for the apache_hbase topology. The behavior to be carried out by the -build_cluster and start_cluster clusterdock scripts are to be defined through the build and -start functions, respectively. -""" - -import logging -import tarfile -from ConfigParser import ConfigParser -from os import EX_OK, listdir, makedirs, remove # pylint: disable=ungrouped-imports - # Follow convention of grouping from module imports - # after normal imports. -from os.path import exists, join -from shutil import move -from socket import getfqdn -from sys import stdout -from uuid import uuid4 - -# pylint: disable=import-error -# clusterdock topologies get access to the clusterdock package at run time, but their reference -# to clusterdock modules will confuse pylint, so we have to disable it. - -import requests -from docker import Client - -from clusterdock import Constants -from clusterdock.cluster import Cluster, Node, NodeGroup -from clusterdock.docker_utils import (build_image, get_clusterdock_container_id, - get_host_port_binding, is_image_available_locally, pull_image) -from clusterdock.utils import strip_components_from_tar, XmlConfiguration - -# We disable a couple of Pylint conventions because it assumes that module level variables must be -# named as if they're constants (which isn't the case here). -logger = logging.getLogger(__name__) # pylint: disable=invalid-name -logger.setLevel(logging.INFO) - -client = Client() # pylint: disable=invalid-name - -DEFAULT_APACHE_NAMESPACE = Constants.DEFAULT.apache_namespace # pylint: disable=no-member - -def _copy_container_folder_to_host(container_id, source_folder, destination_folder, - host_folder=None): - if not exists(destination_folder): - makedirs(destination_folder) - stream, _ = client.get_archive(container_id, source_folder) - tar_filename = join(destination_folder, 'container_folder.tar') - with open(tar_filename, 'wb') as file_descriptor: - file_descriptor.write(stream.read()) - tar = tarfile.open(name=tar_filename) - tar.extractall(path=destination_folder, members=strip_components_from_tar(tar)) - tar.close() - remove(tar_filename) - logger.info("Extracted container folder %s to %s.", source_folder, - host_folder if host_folder else destination_folder) - -def _create_configs_from_file(filename, cluster_config_dir, wildcards): - configurations = ConfigParser(allow_no_value=True) - configurations.read(filename) - - for config_file in configurations.sections(): - logger.info("Updating %s...", config_file) - # For XML configuration files, run things through XmlConfiguration. - if config_file.endswith('.xml'): - XmlConfiguration( - {item[0]: item[1].format(**wildcards) - for item in configurations.items(config_file)} - ).write_to_file(join(cluster_config_dir, config_file)) - # For everything else, recognize whether a line in the configuration should simply be - # appended to the bottom of a file or processed in some way. The presence of +++ will - # lead to the evaluation of the following string through the end of the line. - else: - lines = [] - for item in configurations.items(config_file): - if item[0].startswith('+++'): - command = item[0].lstrip('+ ').format(**wildcards) - - # Yes, we use eval here. This is potentially dangerous, but intention. - lines.append(str(eval(command))) # pylint: disable=eval-used - elif item[0] == "body": - lines.append(item[1].format(**wildcards)) - else: - lines.append(item[0].format(**wildcards)) - with open(join(cluster_config_dir, config_file), 'w') as conf: - conf.write("".join(["{0}\n".format(line) for line in lines])) - -# Keep track of some common web UI ports that we'll expose to users later (e.g. to allow a user -# to reach the HDFS NameNode web UI over the internet). -HBASE_REST_SERVER_PORT = 8080 -NAMENODE_WEB_UI_PORT = 50070 -RESOURCEMANAGER_WEB_UI_PORT = 8088 - -# When starting or building cluster, CLUSTERDOCK_VOLUME will be the root directory for persistent -# files (note that this location will itself be in a Docker container's filesystem). -CLUSTERDOCK_VOLUME = '/tmp/clusterdock' - -def start(args): - """This function will be executed when ./bin/start_cluster apache_hbase is invoked.""" - - # pylint: disable=too-many-locals - # Pylint doesn't want more than 15 local variables in a function; this one has 17. This is about - # as low as I want to go because, while I can cheat and stuff unrelated things in a dictionary, - # that won't improve readability. - - uuid = str(uuid4()) - container_cluster_config_dir = join(CLUSTERDOCK_VOLUME, uuid, 'config') - makedirs(container_cluster_config_dir) - - for mount in client.inspect_container(get_clusterdock_container_id())['Mounts']: - if mount['Destination'] == CLUSTERDOCK_VOLUME: - host_cluster_config_dir = join(mount['Source'], uuid, 'config') - break - else: - raise Exception("Could not find source of {0} mount.".format(CLUSTERDOCK_VOLUME)) - - # CLUSTERDOCK_VOLUME/uuid/config in the clusterdock container corresponds to - # host_cluster_config_dir on the Docker host. - logger.debug("Creating directory for cluster configuration files in %s...", - host_cluster_config_dir) - - # Generate the image name to use from the command line arguments passed in. - image = '/'.join( - [item - for item in [args.registry_url, args.namespace or DEFAULT_APACHE_NAMESPACE, - "clusterdock:{os}_java-{java}_hadoop-{hadoop}_hbase-{hbase}".format( - os=args.operating_system, java=args.java_version, - hadoop=args.hadoop_version, hbase=args.hbase_version - )] - if item] - ) - if args.always_pull or not is_image_available_locally(image): - pull_image(image) - - # Before starting the cluster, we create a throwaway container from which we copy - # configuration files back to the host. We also use this container to run an HBase - # command that returns the port of the HBase master web UI. Since we aren't running init here, - # we also have to manually pass in JAVA_HOME as an environmental variable. - get_hbase_web_ui_port_command = ('/hbase/bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool ' - 'hbase.master.info.port') - container_id = client.create_container(image=image, command=get_hbase_web_ui_port_command, - environment={'JAVA_HOME': '/java'})['Id'] - logger.debug("Created temporary container (id: %s) from which to copy configuration files.", - container_id) - - # Actually do the copying of Hadoop configs... - _copy_container_folder_to_host(container_id, '/hadoop/etc/hadoop', - join(container_cluster_config_dir, 'hadoop'), - join(host_cluster_config_dir, 'hadoop')) - - # ... and repeat for HBase configs. - _copy_container_folder_to_host(container_id, '/hbase/conf', - join(container_cluster_config_dir, 'hbase'), - join(host_cluster_config_dir, 'hbase')) - - logger.info("The /hbase/lib folder on containers in the cluster will be volume mounted " - "into %s...", join(host_cluster_config_dir, 'hbase-lib')) - _copy_container_folder_to_host(container_id, '/hbase/lib', - join(container_cluster_config_dir, 'hbase-lib'), - join(host_cluster_config_dir, 'hbase-lib')) - - # Every node in the cluster will have a shared volume mount from the host for Hadoop and HBase - # configuration files as well as the HBase lib folder. - shared_volumes = [{join(host_cluster_config_dir, 'hadoop'): '/hadoop/etc/hadoop'}, - {join(host_cluster_config_dir, 'hbase'): '/hbase/conf'}, - {join(host_cluster_config_dir, 'hbase-lib'): '/hbase/lib'}] - - # Get the HBase master web UI port, stripping the newline the Docker REST API gives us. - client.start(container=container_id) - if client.wait(container=container_id) == EX_OK: - hbase_master_web_ui_port = client.logs(container=container_id).rstrip() - client.remove_container(container=container_id, force=True) - else: - raise Exception('Failed to remove HBase configuration container.') - - # Create the Node objects. These hold the state of our container nodes and will be started - # at Cluster instantiation time. - primary_node = Node(hostname=args.primary_node[0], network=args.network, - image=image, ports=[NAMENODE_WEB_UI_PORT, - hbase_master_web_ui_port, - RESOURCEMANAGER_WEB_UI_PORT, - HBASE_REST_SERVER_PORT], - volumes=shared_volumes) - secondary_nodes = [] - for hostname in args.secondary_nodes: - # A list of service directories will be used to name folders on the host and, appended - # with an index, in the container, as well (e.g. /data1/node-1/dfs:/dfs1). - service_directories = ['dfs', 'yarn'] - - # Every Node will have shared_volumes to let one set of configs on the host be propagated - # to every container. If --data-directories is specified, this will be appended to allow - # containers to use multiple disks on the host. - volumes = shared_volumes[:] - if args.data_directories: - data_directories = args.data_directories.split(',') - volumes += [{join(data_directory, uuid, hostname, service_directory): - "/{0}{1}".format(service_directory, i)} - for i, data_directory in enumerate(data_directories, start=1) - for service_directory in service_directories] - secondary_nodes.append(Node(hostname=hostname, - network=args.network, - image=image, - volumes=volumes)) - - Cluster(topology='apache_hbase', - node_groups=[NodeGroup(name='primary', nodes=[primary_node]), - NodeGroup(name='secondary', nodes=secondary_nodes)], - network_name=args.network).start() - - # When creating configs, pass in a dictionary of wildcards into create_configurations_from_file - # to transform placeholders in the configurations.cfg file into real values. - _create_configs_from_file(filename=args.configurations, - cluster_config_dir=container_cluster_config_dir, - wildcards={"primary_node": args.primary_node, - "secondary_nodes": args.secondary_nodes, - "all_nodes": args.primary_node + args.secondary_nodes, - "network": args.network}) - - # After creating configurations from the configurations.cfg file, update hdfs-site.xml and - # yarn-site.xml to use the data directories passed on the command line. - if args.data_directories: - _update_config_for_data_dirs( - container_cluster_config_dir=container_cluster_config_dir, - data_directories=data_directories - ) - - if not args.dont_start_services: - _start_services(primary_node, hbase_master_web_ui_port=hbase_master_web_ui_port) - -def _update_config_for_data_dirs(container_cluster_config_dir, data_directories): - logger.info('Updating dfs.datanode.data.dir in hdfs-site.xml...') - hdfs_site_xml_filename = join(container_cluster_config_dir, 'hadoop', 'hdfs-site.xml') - hdfs_site_xml = XmlConfiguration( - properties={'dfs.datanode.data.dir': - ','.join(["/dfs{0}".format(i) - for i, _ in enumerate(data_directories, start=1)])}, - source_file=hdfs_site_xml_filename - ) - hdfs_site_xml.write_to_file(filename=hdfs_site_xml_filename) - - logger.info('Updating yarn.nodemanager.local-dirs in yarn-site.xml...') - yarn_site_xml_filename = join(container_cluster_config_dir, 'hadoop', 'yarn-site.xml') - yarn_site_xml = XmlConfiguration( - properties={'yarn.nodemanager.local-dirs': - ','.join(["/yarn{0}".format(i) - for i, _ in enumerate(data_directories, start=1)])}, - source_file=yarn_site_xml_filename - ) - yarn_site_xml.write_to_file(filename=yarn_site_xml_filename) - -def _start_services(primary_node, **kwargs): - logger.info("Formatting namenode on %s...", primary_node.fqdn) - primary_node.ssh('hdfs namenode -format') - - logger.info("Starting HDFS...") - primary_node.ssh('/hadoop/sbin/start-dfs.sh') - - logger.info("Starting YARN...") - primary_node.ssh('/hadoop/sbin/start-yarn.sh') - - logger.info('Starting HBase...') - primary_node.ssh('/hbase/bin/start-hbase.sh') - primary_node.ssh('/hbase/bin/hbase-daemon.sh start rest') - - logger.info("NameNode and HBase master are located on %s. SSH over and have fun!", - primary_node.hostname) - - logger.info("The HDFS NameNode web UI can be reached at http://%s:%s";, - getfqdn(), get_host_port_binding(primary_node.container_id, - NAMENODE_WEB_UI_PORT)) - - logger.info("The YARN ResourceManager web UI can be reached at http://%s:%s";, - getfqdn(), get_host_port_binding(primary_node.container_id, - RESOURCEMANAGER_WEB_UI_PORT)) - - logger.info("The HBase master web UI can be reached at http://%s:%s";, - getfqdn(), get_host_port_binding(primary_node.container_id, - kwargs.get('hbase_master_web_ui_port'))) - - logger.info("The HBase REST server can be reached at http://%s:%s";, - getfqdn(), get_host_port_binding(primary_node.container_id, - HBASE_REST_SERVER_PORT)) - -def build(args): - """This function will be executed when ./bin/build_cluster apache_hbase is invoked.""" - - # pylint: disable=too-many-locals - # See start function above for rationale for disabling this warning. - - container_build_dir = join(CLUSTERDOCK_VOLUME, str(uuid4())) - makedirs(container_build_dir) - - # If --hbase-git-commit is specified, we build HBase from source. - if args.hbase_git_commit: - build_hbase_commands = [ - "git clone https://github.com/apache/hbase.git {0}".format(container_build_dir), - "git -C {0} checkout {1}".format(container_build_dir, args.hbase_git_commit), - "mvn --batch-mode clean install -DskipTests assembly:single -f {0}/pom.xml".format( - container_build_dir - ) - ] - - maven_image = Constants.docker_images.maven # pylint: disable=no-member - if not is_image_available_locally(maven_image): - pull_image(maven_image) - - container_configs = { - 'command': 'bash -c "{0}"'.format(' && '.join(build_hbase_commands)), - 'image': maven_image, - 'host_config': client.create_host_config(volumes_from=get_clusterdock_container_id()) - } - - maven_container_id = client.create_container(**container_configs)['Id'] - client.start(container=maven_container_id) - for line in client.logs(container=maven_container_id, stream=True): - stdout.write(line) - stdout.flush() - - # Mimic docker run --rm by blocking on docker wait and then removing the container - # if it encountered no errors. - if client.wait(container=maven_container_id) == EX_OK: - client.remove_container(container=maven_container_id, force=True) - else: - raise Exception('Error encountered while building HBase.') - - assembly_target_dir = join(container_build_dir, 'hbase-assembly', 'target') - for a_file in listdir(assembly_target_dir): - if a_file.endswith('bin.tar.gz'): - args.hbase_tarball = join(assembly_target_dir, a_file) - break - - # Download all the binary tarballs into our temporary directory so that we can add them - # into the Docker image we're building. - filenames = [] - for tarball_location in [args.java_tarball, args.hadoop_tarball, args.hbase_tarball]: - tarball_filename = tarball_location.rsplit('/', 1)[-1] - filenames.append(tarball_filename) - - # Download tarballs given as URLs. - if container_build_dir not in tarball_location: - get_request = requests.get(tarball_location, stream=True, cookies=( - {'oraclelicense': 'accept-securebackup-cookie'} - if tarball_location == args.java_tarball - else None - )) - # Raise Exception if download failed. - get_request.raise_for_status() - logger.info("Downloading %s...", tarball_filename) - with open(join(container_build_dir, tarball_filename), 'wb') as file_descriptor: - for chunk in get_request.iter_content(1024): - file_descriptor.write(chunk) - else: - move(tarball_location, container_build_dir) - - dockerfile_contents = r""" - FROM {nodebase_image} - COPY {java_tarball} /tarballs/ - RUN mkdir /java && tar -xf /tarballs/{java_tarball} -C /java --strip-components=1 - RUN echo "JAVA_HOME=/java" >> /etc/environment - - COPY {hadoop_tarball} /tarballs/ - RUN mkdir /hadoop && tar -xf /tarballs/{hadoop_tarball} -C /hadoop --strip-components=1 - COPY {hbase_tarball} /tarballs/ - RUN mkdir /hbase && tar -xf /tarballs/{hbase_tarball} -C /hbase --strip-components=1 - - # Remove tarballs folder. - RUN rm -rf /tarballs - - # Set PATH explicitly. - RUN echo "PATH=/java/bin:/hadoop/bin:/hbase/bin/:$(echo $PATH)" >> /etc/environment - - # Add hbase user and group before copying root's SSH keys over. - RUN groupadd hbase \ - && useradd -g hbase hbase \ - && cp -R /root/.ssh ~hbase \ - && chown -R hbase:hbase ~hbase/.ssh - - # Disable requiretty in /etc/sudoers as required by HBase chaos monkey. - RUN sed -i 's/Defaults\s*requiretty/#&/' /etc/sudoers - """.format(nodebase_image='/'.join([item - for item in [args.registry_url, - args.namespace or DEFAULT_APACHE_NAMESPACE, - "clusterdock:{os}_nodebase".format( - os=args.operating_system - )] - if item]), - java_tarball=filenames[0], hadoop_tarball=filenames[1], hbase_tarball=filenames[2]) - - logger.info("Created Dockerfile: %s", dockerfile_contents) - - with open(join(container_build_dir, 'Dockerfile'), 'w') as dockerfile: - dockerfile.write(dockerfile_contents) - - image = '/'.join( - [item - for item in [args.registry_url, args.namespace or DEFAULT_APACHE_NAMESPACE, - "clusterdock:{os}_java-{java}_hadoop-{hadoop}_hbase-{hbase}".format( - os=args.operating_system, java=args.java_version, - hadoop=args.hadoop_version, hbase=args.hbase_version - )] - if item]) - - logger.info("Building image %s...", image) - build_image(dockerfile=join(container_build_dir, 'Dockerfile'), tag=image) - - logger.info("Removing build temporary directory...") - return [image] http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/configurations.cfg ---------------------------------------------------------------------- diff --git a/dev-support/apache_hbase_topology/configurations.cfg b/dev-support/apache_hbase_topology/configurations.cfg deleted file mode 100644 index f28995e..0000000 --- a/dev-support/apache_hbase_topology/configurations.cfg +++ /dev/null @@ -1,80 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# This configuration file is used by the apache-hbase clusterdock topology to populate configuration -# files for an HBase cluster. Section names denote the filenames (e.g. [hadoop/core-site.xml]) with -# the section's corresponding items denoting properties. -# -# Filenames ending with .xml will have items run through an XML parser. That is, -# -# [hbase/hbase-site.xml] -# hbase.cluster.distributed = true -# -# would lead to the creation of an hbase-site.xml file containing: -# -# <property> -# <name>hbase.cluster.distributed</name> -# <value>true</value> -# </property> -# -# Note, also, that items in non-xml files can be copied verbatim by with using the "body:" item (be -# sure to include leading whitespace must be used for the following lines). For example: -# -# [hbase/hbase-env.sh] -# body: -# COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+UseG1GC" -# COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+PrintGCDetails" -# -# would result in an hbase-env.sh file with: -# -# COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+UseG1GC" -# COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+PrintGCDetails" -# -# Two last notes: -# 1. Items starting with +++ will be eval'd with Python directly. -# 2. As defined in action.py, some wildcards are processed at cluster start time (e.g. {network}). - -[hadoop/slaves] -+++ '\n'.join(["{{0}}.{network}".format(node) for node in {secondary_nodes}]) - -[hadoop/core-site.xml] -fs.default.name = hdfs://{primary_node[0]}.{network}:8020 - -[hadoop/mapred-site.xml] -mapreduce.framework.name = yarn - -[hadoop/yarn-site.xml] -yarn.resourcemanager.hostname = {primary_node[0]}.{network} -yarn.nodemanager.aux-services = mapreduce_shuffle -yarn.nodemanager.aux-services.mapreduce_shuffle.class = org.apache.hadoop.mapred.ShuffleHandler -yarn.nodemanager.vmem-check-enabled = false - -[hbase/regionservers] -+++ '\n'.join(["{{0}}.{network}".format(node) for node in {secondary_nodes}]) - -[hbase/backup-masters] -{secondary_nodes[0]}.{network} - -[hbase/hbase-site.xml] -hbase.cluster.distributed = true -hbase.rootdir = hdfs://{primary_node[0]}.{network}/hbase -hbase.zookeeper.quorum = {primary_node[0]}.{network} -hbase.zookeeper.property.dataDir = /usr/local/zookeeper - -# For now, set service users for Chaos Monkey to be root. -hbase.it.clustermanager.hadoop.hdfs.user = root -hbase.it.clustermanager.zookeeper.user = root -hbase.it.clustermanager.hbase.user = root http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/profile.cfg ---------------------------------------------------------------------- diff --git a/dev-support/apache_hbase_topology/profile.cfg b/dev-support/apache_hbase_topology/profile.cfg deleted file mode 100644 index e6937ff..0000000 --- a/dev-support/apache_hbase_topology/profile.cfg +++ /dev/null @@ -1,82 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# This configuration file is used to define some general properties of the apache-hbase clusterdock -# topology, as well as its command line. - -[general] -name = Apache HBase -description = An Apache HBase cluster with 1 primary node and n-1 secondary nodes - - -[node_groups] -# Define node groups and specify which of each to start during the build process. -primary-node = node-1 -secondary-nodes = node-2 - - -[build] -arg.java-tarball = http://download.oracle.com/otn-pub/java/jdk/8u91-b14/jdk-8u91-linux-x64.tar.gz -arg.java-tarball.help = The URL (or filename) of the Java tarball to install on the cluster -arg.java-tarball.metavar = tarball - -arg.java-version = 8u91 -arg.java-version.help = The label to use when identifying the version of Java -arg.java-version.metavar = ver - -arg.hadoop-tarball = https://archive.apache.org/dist/hadoop/core/hadoop-2.7.2/hadoop-2.7.2.tar.gz -arg.hadoop-tarball.help = The URL (or filename) of the Hadoop tarball to install on the cluster -arg.hadoop-tarball.metavar = tarball - -arg.hadoop-version = 2.7.2 -arg.hadoop-version.help = The label to use when identifying the version of Hadoop -arg.hadoop-version.metavar = ver - -arg.hbase-tarball -arg.hbase-tarball.help = The URL (or filename) of the HBase tarball to install on the cluster -arg.hbase-tarball.metavar = tarball - -arg.hbase-version -arg.hbase-version.help = The label to use when identifying the version of HBase -arg.hbase-version.metavar = ver - -arg.hbase-git-commit -arg.hbase-git-commit.help = The git commit to checkout when building the image -arg.hbase-git-commit.metavar = commit - -[start] -arg.java-version = 8u91 -arg.java-version.help = The Java version on the cluster -arg.java-version.metavar = ver - -arg.hadoop-version = 2.7.2 -arg.hadoop-version.help = The Hadoop version on the cluster -arg.hadoop-version.metavar = ver - -arg.hbase-version = master -arg.hbase-version.help = The HBase version on the cluster -arg.hbase-version.metavar = ver - -arg.configurations = /root/clusterdock/clusterdock/topologies/apache_hbase/configurations.cfg -arg.configurations.help = Location of a configurations.cfg file to apply to the cluster -arg.configurations.metavar = path - -arg.data-directories -arg.data-directories.help = A comma-separated list of host directories on which to save data (i.e. for HDFS and YARN) -arg.data-directories.metavar = dir - -arg.dont-start-services = False -arg.dont-start-services.help = Don't start Hadoop and HBase services as part of cluster start http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/ssh/id_rsa ---------------------------------------------------------------------- diff --git a/dev-support/apache_hbase_topology/ssh/id_rsa b/dev-support/apache_hbase_topology/ssh/id_rsa deleted file mode 100644 index 7c9d31e..0000000 --- a/dev-support/apache_hbase_topology/ssh/id_rsa +++ /dev/null @@ -1,44 +0,0 @@ -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ ------BEGIN RSA PRIVATE KEY----- -MIIEpAIBAAKCAQEAtj/yZZNF+bv26eqWqsx+vFehSlxBJp/QhIrFWKpjHcpQJ29o -6hJN9moU3Goft2C5w6FoZdjC1TWlzhxUnRS8xeksFnW3ItvkjySLA1Iq6jchYxNd -fZ3HwTdH0rubM1uJ/CnkaijoxBqGBmPSL0TxfqcteaJ8APhslwl0WWJ6b+tBCHDV -pTLATebtggCAfhKmSuAYmn3QIqJ7DFoSGkwhkxpUHuuVCZxUH3CIxLIw+6npArr/ -S5gtFo50oi6FXPmvv6mJg6yLqq3VlKcQh6d/COaJopHn+nLed2kECESUlpTruMpr -6IcGESgz4hnkmhop8oTY42sQJhPPF2Ahq9a3aQIDAQABAoIBADSbxPb5SkvKrH3d -j9yB51uq2A5FDzF9FI4OGOV9WdsxmW2oxVo8KnElMhxmLf2bWERWhXJQ3fz53YDf -wLUPVWaz5lwdYt4XJ6UCYXZ185lkjKiy4FvwfccSlBMKwMRUekJmPV8/q+Ff3qxd -iEDI4AU1cPUZqD4HeCEpQ4LB4KIJhCdLkCgoWxxaCwwuB6DnwB4P4VLeAfsm2NEX -k9dld87q/miOmuw9QsmSv9wYiQqoPdV5Qj7KYqHBAa6syqUfFni3Ibmw1WBzMydp -8YyP9HvrzDBMnPPzkmp6od0fAgGafIlkIxz/9sCKOSISnuuqahbNAJK/rIiJzLY3 -Pi49M+ECgYEA2vCFObmM/hpKUPNxG841nQScFfeMg1q1z1krEmfjqDTELXyq9AOS -fGiVTxfWagOGoAWIwg3ZfgGEmxsKrOSxkFvabWdhN1Wn98Zf8fJG8YAwLYg8JOgf -gZ5pkxFW4FwrAeFDyJyKvFJVrbDw1PM41yvTmRzf3NjcaqJrBE2fgKUCgYEA1RmF -XjfMlBoMLZ4pXS1zF91WgOF4SNdJJj9RCGzqqdy+DzPKNAZVa0HBhaIHCZXL3Hcv -zqgEb6RSMysyVYjZPwfSwevsSuxpfriVpYux5MN3AEXX5Ysv51WWStWgt9+iQlfo -xAdxxukOa++PZ4Z+TIIEDAFS47rnKEQUh+ZNfHUCgYEA0amTa3wtcQlsMalvn9kR -rpRDhSXTAddUVIRnovCqKuKdG5JPg+4H0eu1UFDbnBpUSdoC5RKuPOTnQEHdL0Sy -ZjQQMMTXbE4y1Cy8pM4G8i539KKKNi20PkSdhaENOT4KUXqPlwWSNlYChprzhnqE -7EmkEPR9zNg//D4djbloDaECgYANOJIfsFKO9ba/tcpXL5SubFsLj/GIg2LUbqU2 -YpuEgl+ATfRDmgj+qIu7ILxTCeol+XcL2Ty9OHKpHgr3Z5Ai6vdWdK6qT1SUOht+ -s9YLnVzqtWqZoTMNpS+34N0hy0wj1ZRpZRTYBGmSpMA+6gc38/EQVZyw6E2jH+Yu -MEmqaQKBgQDGh9uXCl/WjhBOF9VLrX/Aeaa2Mzrh21Ic1dw6aWrE4EW6k1LvSP36 -evrvrs2jQuzRMGH6DKX8ImnVEWjK+gZfgf2MuyDSW7KYR5zxkdZtRkotF6X0fu6N -8uLa7CN8UmS4FiAMLwNbTJ6zA6ohny7r+AiOqNGlP9vBFMhpGs3NFg== ------END RSA PRIVATE KEY----- http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/ssh/id_rsa.pub ---------------------------------------------------------------------- diff --git a/dev-support/apache_hbase_topology/ssh/id_rsa.pub b/dev-support/apache_hbase_topology/ssh/id_rsa.pub deleted file mode 100644 index 0dca44a..0000000 --- a/dev-support/apache_hbase_topology/ssh/id_rsa.pub +++ /dev/null @@ -1,18 +0,0 @@ -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ -ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2P/Jlk0X5u/bp6paqzH68V6FKXEEmn9CEisVYqmMdylAnb2jqEk32ahTcah+3YLnDoWhl2MLVNaXOHFSdFLzF6SwWdbci2+SPJIsDUirqNyFjE119ncfBN0fSu5szW4n8KeRqKOjEGoYGY9IvRPF+py15onwA+GyXCXRZYnpv60EIcNWlMsBN5u2CAIB+EqZK4BiafdAionsMWhIaTCGTGlQe65UJnFQfcIjEsjD7qekCuv9LmC0WjnSiLoVc+a+/qYmDrIuqrdWUpxCHp38I5omikef6ct53aQQIRJSWlOu4ymvohwYRKDPiGeSaGinyhNjjaxAmE88XYCGr1rdp clusterdock
