ACCUMULO-4528 Accumulo scripts improvements * Accumulo now determines hostname in Java * Minimized use of environment variables * Consolidated scripts by moving code in service.sh and cluster.sh to accumulo-service and accumulo-cluster. * Example host files (i.e master, tservers) are no longer included in tarball distribution. User can create them using 'accumulo-cluster create-config' * Moved check-tservers from distribution to contrib/scripts in repo. * NUMA and multiple tservers are no longer configured in scripts but could be NUMA commands could be added using ACCUMULO_JAVA_PREFIX env variable. * Removed accumulo-watcher script as restarting services should not be handled by Accumulo scripts. Script coud still be used in future using ACCUMULO_JAVA_PREFIX. * Made several updates to accumulo-env.sh * Moved create-config and build-native from accumulo script to new accumulo-util script which now includes scripts from contrib dir
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/9a3cc5c4 Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/9a3cc5c4 Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/9a3cc5c4 Branch: refs/heads/master Commit: 9a3cc5c445293e8d4219cc066757a72a972804b4 Parents: 00b1535 Author: Mike Walch <mwa...@apache.org> Authored: Thu Feb 2 12:15:17 2017 -0500 Committer: Mike Walch <mwa...@apache.org> Committed: Fri Feb 17 13:47:06 2017 -0500 ---------------------------------------------------------------------- INSTALL.md | 42 +- assemble/bin/accumulo | 610 +++-------------- assemble/bin/accumulo-cluster | 300 +++++++- assemble/bin/accumulo-service | 95 ++- assemble/bin/accumulo-util | 686 +++++++++++++++++++ assemble/conf/examples/gc | 16 - .../conf/examples/generic_logger.properties | 6 +- assemble/conf/examples/generic_logger.xml | 6 +- assemble/conf/examples/masters | 16 - assemble/conf/examples/monitor | 16 - .../conf/examples/templates/accumulo-env.sh | 64 ++ .../conf/examples/templates/accumulo-site.xml | 175 +++++ assemble/conf/examples/tracers | 16 - assemble/conf/examples/tservers | 16 - assemble/contrib/bootstrap-hdfs.sh | 91 --- assemble/contrib/check-tservers | 199 ------ assemble/contrib/gen-monitor-cert.sh | 85 --- assemble/contrib/tool.sh | 93 --- assemble/libexec/accumulo-watcher.sh | 141 ---- assemble/libexec/cluster.sh | 325 --------- assemble/libexec/load-env.sh | 155 ----- assemble/libexec/service.sh | 239 ------- assemble/libexec/templates/accumulo-env.sh | 90 --- assemble/libexec/templates/accumulo-site.xml | 175 ----- assemble/pom.xml | 7 +- assemble/src/main/assemblies/component.xml | 50 +- .../main/scripts/generate-example-configs.sh | 2 +- contrib/scripts/check-tservers.py | 199 ++++++ .../accumulo/core/conf/ConfigurationDocGen.java | 2 +- .../org/apache/accumulo/core/conf/Property.java | 2 +- .../main/asciidoc/chapters/administration.txt | 187 ++--- docs/src/main/asciidoc/chapters/clients.txt | 4 +- docs/src/main/asciidoc/chapters/kerberos.txt | 14 +- docs/src/main/asciidoc/chapters/replication.txt | 4 +- docs/src/main/asciidoc/chapters/ssl.txt | 4 +- .../main/asciidoc/chapters/troubleshooting.txt | 7 +- .../apache/accumulo/server/init/Initialize.java | 2 +- .../java/org/apache/accumulo/start/Main.java | 2 - 38 files changed, 1691 insertions(+), 2452 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/INSTALL.md ---------------------------------------------------------------------- diff --git a/INSTALL.md b/INSTALL.md index b614991..6e924c4 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -25,11 +25,13 @@ source code. Unpack as follows. tar xzf <some dir>/accumulo-X.Y.Z-bin.tar.gz cd accumulo-X.Y.Z -There are three scripts in the the `bin/` directory that are used to manage Accumulo: +There are four scripts in the `bin` directory of the tarball distribution that are used +to manage Accumulo: 1. `accumulo` - Runs Accumulo command-line tools and starts Accumulo processes 2. `accumulo-service` - Runs Accumulo processes as services 3. `accumulo-cluster` - Manages Accumulo cluster on a single node or several nodes +4. `accumulo-util` - Accumulo utilities for creating configuration, native libraries, etc. These scripts will be used in the remaining instructions to configure and run Accumulo. For convenience, consider adding `accumulo-X.Y.Z/bin/` to your shell's path. @@ -40,13 +42,13 @@ Accumulo has some optional native code that improves its performance and stability. Before configuring Accumulo, attempt to build this native code with the following command. - accumulo build-native + accumulo-util build-native If the command fails, its OK to continue with setup and resolve the issue later. Run the command below to create configuration for Accumulo in `conf/`: - accumulo create-config + accumulo-util create-config The script will ask you questions about your set up. Below are some suggestions: @@ -57,7 +59,7 @@ The script will ask you questions about your set up. Below are some suggestions: processes like Hadoop, Zookeeper, and the Accumulo client code. If Accumulo worker processes are swapped out and unresponsive, they may be killed. -After the `create-config` command is run, the `conf/` directory will contain +After the `accumulo-util create-config` command is run, the `conf/` directory will contain `accumulo-env.sh`, `accumulo-site.xml`, and few a additional files. These files require a few edits before starting Accumulo. @@ -65,7 +67,7 @@ a few edits before starting Accumulo. Accumulo coordination and worker processes can only communicate with each other if they share the same secret key. To change the secret key set -`instance.secret` in `conf/accumulo-site.xml`. Changing this secret key from +`instance.secret` in `accumulo-site.xml`. Changing this secret key from the default is highly recommended. ### Dependencies @@ -76,19 +78,19 @@ When configuring Accumulo the following information about these dependencies must be provided. * **Location of Zookeepers** : Provide this by setting `instance.zookeeper.host` - in `conf/accumulo-site.xml`. + in `accumulo-site.xml`. * **Where to store data** : Provide this by setting `instance.volumes` in - `conf/accumulo-site.xml`. If your namenode is running at 192.168.1.9:9000 + `accumulo-site.xml`. If your namenode is running at 192.168.1.9:9000 and you want to store data in `/accumulo` in HDFS, then set `instance.volumes` to `hdfs://192.168.1.9:9000/accumulo`. * **Location of Zoookeeper and Hadoop jars** : Setting `ZOOKEEPER_HOME` and - `HADOOP_PREFIX` in `conf/accumulo-env.sh` will help Accumulo find these - jars. + `HADOOP_PREFIX` in `accumulo-env.sh` will help Accumulo find these jars + when using the default setting for `general.classpaths` in accumulo-site.xml. If Accumulo has problems later on finding jars, then run `bin/accumulo classpath` to print out info about where Accumulo is finding jars. If the settings mentioned above are correct, then inspect `general.classpaths` in -`conf/accumulo-site.xml`. +`accumulo-site.xml`. ## Initialization @@ -141,19 +143,19 @@ changed and the next section should be skipped. #### Multi-node configuration If you are running an Accumulo cluster on multiple nodes, the following files -should be configured with a newline seperated list of node names: +in `conf/` should be configured with a newline seperated list of node names: - * `conf/masters` : Accumulo primary coordinating process. Must specify one - node. Can specify a few for fault tolerance. - * `conf/gc` : Accumulo garbage collector. Must specify one node. Can - specify a few for fault tolerance. - * `conf/monitor` : Node where Accumulo monitoring web server is run. - * `conf/tservers`: Accumulo worker processes. List all of the nodes where - tablet servers should run in this file. - * `conf/tracers` : Optional capability. Can specify zero or more nodes. + * `masters` : Accumulo primary coordinating process. Must specify one node. Can + specify a few for fault tolerance. + * `gc` : Accumulo garbage collector. Must specify one node. Can specify a + few for fault tolerance. + * `monitor` : Node where Accumulo monitoring web server is run. + * `tservers`: Accumulo worker processes. List all of the nodes where tablet servers + should run in this file. + * `tracers` : Optional capability. Can specify zero or more nodes. The Accumulo, Hadoop, and Zookeeper software should be present at the same -location on every node. Also the files in the `conf` directory must be copied +location on every node. Also the files in the `conf` directory must be copied to every node. There are many ways to replicate the software and configuration, two possible tools that can help replicate software and/or config are [pdcp][5] and [prsync][6]. http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/bin/accumulo ---------------------------------------------------------------------- diff --git a/assemble/bin/accumulo b/assemble/bin/accumulo index f725952..f530e3c 100755 --- a/assemble/bin/accumulo +++ b/assemble/bin/accumulo @@ -15,436 +15,21 @@ # See the License for the specific language governing permissions and # limitations under the License. -function build_native() { - if [[ -f "$basedir"/conf/accumulo-env.sh ]]; then - source "$basedir"/libexec/load-env.sh - fi - - native_tarball="$basedir/lib/accumulo-native.tar.gz" - final_native_target="$basedir/lib/native" - - if [[ ! -f $native_tarball ]]; then - echo "Could not find native code artifact: ${native_tarball}"; - exit 1 - fi - - # Make the destination for the native library - mkdir -p "${final_native_target}" || exit 1 - - # Make a directory for us to unpack the native source into - TMP_DIR=$(mktemp -d /tmp/accumulo-native.XXXX) || exit 1 - - # Unpack the tarball to our temp directory - if ! tar xf "${native_tarball}" -C "${TMP_DIR}" - then - echo "Failed to unpack native tarball to ${TMP_DIR}" - exit 1 - fi - - # Move to the first (only) directory in our unpacked tarball - native_dir=$(find "${TMP_DIR}" -maxdepth 1 -mindepth 1 -type d) - - cd "${native_dir}" || exit 1 - - # Make the native library - export USERFLAGS="$*" - make || { echo 'Make failed!'; exit 1; } - - # "install" the artifact - cp libaccumulo.* "${final_native_target}" || exit 1 - - # Clean up our temp directory - rm -rf "${TMP_DIR}" - - echo "Successfully installed native library" -} - -function create_config_usage() { - cat <<EOF -Usage: accumulo create-config [-options] -where options include (long options not available on all platforms): - -d, --dir Alternate directory to setup config files - -s, --size Supported sizes: '1GB' '2GB' '3GB' '512MB' - -n, --native Configure to use native libraries - -j, --jvm Configure to use the jvm - -o, --overwrite Overwrite the default config directory - -v, --version Specify the Apache Hadoop version supported versions: '1' '2' - -k, --kerberos Configure for use with Kerberos - -h, --help Print this help message -EOF -} - -function create_config() { - TEMPLATE_CONF_DIR="$basedir/libexec/templates" - CONF_DIR="${ACCUMULO_CONF_DIR:-$basedir/conf}" - ACCUMULO_SITE=accumulo-site.xml - ACCUMULO_ENV=accumulo-env.sh - - SIZE= - TYPE= - HADOOP_VERSION= - OVERWRITE="0" - BASE_DIR= - KERBEROS= - - #Execute getopt - if [[ $(uname -s) == "Linux" ]]; then - args=$(getopt -o "b:d:s:njokv:h" -l "basedir:,dir:,size:,native,jvm,overwrite,kerberos,version:,help" -q -- "$@") - else # Darwin, BSD - args=$(getopt b:d:s:njokv:h "$@") - fi - - #Bad arguments - if [[ $? != 0 ]]; then - create_config_usage 1>&2 +function verify_env_dir() { + property=$1 + directory=$2 + if [[ -z "$directory" ]]; then + echo "$property is not set. Please make sure it's set globally or in $conf/accumulo-env.sh" exit 1 fi - eval set -- "${args[@]}" - - for i - do - case "$i" in - -b|--basedir) #Hidden option used to set general.maven.project.basedir for developers - BASE_DIR=$2; shift - shift;; - -d|--dir) - CONF_DIR=$2; shift - shift;; - -s|--size) - SIZE=$2; shift - shift;; - -n|--native) - TYPE=native - shift;; - -j|--jvm) - TYPE=jvm - shift;; - -o|--overwrite) - OVERWRITE=1 - shift;; - -v|--version) - HADOOP_VERSION=$2; shift - shift;; - -k|--kerberos) - KERBEROS="true" - shift;; - -h|--help) - create_config_usage - exit 0 - shift;; - --) - shift - break;; - esac - done - - while [[ "${OVERWRITE}" = "0" ]]; do - if [[ -e "${CONF_DIR}/${ACCUMULO_ENV}" || -e "${CONF_DIR}/${ACCUMULO_SITE}" ]]; then - echo "Warning your current config files in ${CONF_DIR} will be overwritten!" - echo - echo "How would you like to proceed?:" - select CHOICE in 'Continue with overwrite' 'Specify new conf dir'; do - if [[ "${CHOICE}" = 'Specify new conf dir' ]]; then - echo -n "Please specifiy new conf directory: " - read CONF_DIR - elif [[ "${CHOICE}" = 'Continue with overwrite' ]]; then - OVERWRITE=1 - fi - break - done - else - OVERWRITE=1 - fi - done - echo "Copying configuration files to: ${CONF_DIR}" - - #Native 1GB - native_1GB_tServer="-Xmx128m -Xms128m" - _1GB_master="-Xmx128m -Xms128m" - _1GB_monitor="-Xmx64m -Xms64m" - _1GB_gc="-Xmx64m -Xms64m" - _1GB_other="-Xmx128m -Xms64m" - _1GB_shell="${_1GB_other}" - - _1GB_memoryMapMax="256M" - native_1GB_nativeEnabled="true" - _1GB_cacheDataSize="15M" - _1GB_cacheIndexSize="40M" - _1GB_sortBufferSize="50M" - _1GB_waLogMaxSize="256M" - - #Native 2GB - native_2GB_tServer="-Xmx256m -Xms256m" - _2GB_master="-Xmx256m -Xms256m" - _2GB_monitor="-Xmx128m -Xms64m" - _2GB_gc="-Xmx128m -Xms128m" - _2GB_other="-Xmx256m -Xms64m" - _2GB_shell="${_2GB_other}" - - _2GB_memoryMapMax="512M" - native_2GB_nativeEnabled="true" - _2GB_cacheDataSize="30M" - _2GB_cacheIndexSize="80M" - _2GB_sortBufferSize="50M" - _2GB_waLogMaxSize="512M" - - #Native 3GB - native_3GB_tServer="-Xmx1g -Xms1g -XX:NewSize=500m -XX:MaxNewSize=500m" - _3GB_master="-Xmx1g -Xms1g" - _3GB_monitor="-Xmx1g -Xms256m" - _3GB_gc="-Xmx256m -Xms256m" - _3GB_other="-Xmx1g -Xms256m" - _3GB_shell="${_3GB_other}" - - _3GB_memoryMapMax="1G" - native_3GB_nativeEnabled="true" - _3GB_cacheDataSize="128M" - _3GB_cacheIndexSize="128M" - _3GB_sortBufferSize="200M" - _3GB_waLogMaxSize="1G" - - #Native 512MB - native_512MB_tServer="-Xmx48m -Xms48m" - _512MB_master="-Xmx128m -Xms128m" - _512MB_monitor="-Xmx64m -Xms64m" - _512MB_gc="-Xmx64m -Xms64m" - _512MB_other="-Xmx128m -Xms64m" - _512MB_shell="${_512MB_other}" - - _512MB_memoryMapMax="80M" - native_512MB_nativeEnabled="true" - _512MB_cacheDataSize="7M" - _512MB_cacheIndexSize="20M" - _512MB_sortBufferSize="50M" - _512MB_waLogMaxSize="100M" - - #JVM 1GB - jvm_1GB_tServer="-Xmx384m -Xms384m" - - jvm_1GB_nativeEnabled="false" - - #JVM 2GB - jvm_2GB_tServer="-Xmx768m -Xms768m" - - jvm_2GB_nativeEnabled="false" - - #JVM 3GB - jvm_3GB_tServer="-Xmx2g -Xms2g -XX:NewSize=1G -XX:MaxNewSize=1G" - - jvm_3GB_nativeEnabled="false" - - #JVM 512MB - jvm_512MB_tServer="-Xmx128m -Xms128m" - - jvm_512MB_nativeEnabled="false" - - - if [[ -z "${SIZE}" ]]; then - echo "Choose the heap configuration:" - select DIRNAME in 1GB 2GB 3GB 512MB; do - echo "Using '${DIRNAME}' configuration" - SIZE=${DIRNAME} - break - done - elif [[ "${SIZE}" != "1GB" && "${SIZE}" != "2GB" && "${SIZE}" != "3GB" && "${SIZE}" != "512MB" ]]; then - echo "Invalid memory size" - echo "Supported sizes: '1GB' '2GB' '3GB' '512MB'" - exit 1 - fi - - if [[ -z "${TYPE}" ]]; then - echo - echo "Choose the Accumulo memory-map type:" - select TYPENAME in Java Native; do - if [[ "${TYPENAME}" == "Native" ]]; then - TYPE="native" - echo "Don't forget to build the native libraries using the command 'bin/accumulo build-native'" - elif [[ "${TYPENAME}" == "Java" ]]; then - TYPE="jvm" - fi - echo "Using '${TYPE}' configuration" - echo - break - done - fi - - if [[ -z "${HADOOP_VERSION}" ]]; then - echo - echo "Choose the Apache Hadoop version:" - select HADOOP in 'Hadoop 2' 'HDP 2.0/2.1' 'HDP 2.2' 'IOP 4.1'; do - if [ "${HADOOP}" == "Hadoop 2" ]; then - HADOOP_VERSION="2" - elif [ "${HADOOP}" == "HDP 2.0/2.1" ]; then - HADOOP_VERSION="HDP2" - elif [ "${HADOOP}" == "HDP 2.2" ]; then - HADOOP_VERSION="HDP2.2" - elif [ "${HADOOP}" == "IOP 4.1" ]; then - HADOOP_VERSION="IOP4.1" - fi - echo "Using Hadoop version '${HADOOP_VERSION}' configuration" - echo - break - done - elif [[ "${HADOOP_VERSION}" != "2" && "${HADOOP_VERSION}" != "HDP2" && "${HADOOP_VERSION}" != "HDP2.2" ]]; then - echo "Invalid Hadoop version" - echo "Supported Hadoop versions: '2', 'HDP2', 'HDP2.2'" + if [[ ! -d "$directory" ]]; then + echo "$property=$directory is not a valid directory. Please make sure it's set correctly globally or in $conf/accumulo-env.sh" exit 1 fi - - TRACE_USER="root" - - if [[ ! -z "${KERBEROS}" ]]; then - echo - read -p "Enter server's Kerberos principal: " PRINCIPAL - read -p "Enter server's Kerberos keytab: " KEYTAB - TRACE_USER="${PRINCIPAL}" - fi - - for var in SIZE TYPE HADOOP_VERSION; do - if [[ -z ${!var} ]]; then - echo "Invalid $var configuration" - exit 1 - fi - done - - TSERVER="${TYPE}_${SIZE}_tServer" - MASTER="_${SIZE}_master" - MONITOR="_${SIZE}_monitor" - GC="_${SIZE}_gc" - SHELL="_${SIZE}_shell" - OTHER="_${SIZE}_other" - - MEMORY_MAP_MAX="_${SIZE}_memoryMapMax" - NATIVE="${TYPE}_${SIZE}_nativeEnabled" - CACHE_DATA_SIZE="_${SIZE}_cacheDataSize" - CACHE_INDEX_SIZE="_${SIZE}_cacheIndexSize" - SORT_BUFFER_SIZE="_${SIZE}_sortBufferSize" - WAL_MAX_SIZE="_${SIZE}_waLogMaxSize" - - MAVEN_PROJ_BASEDIR="" - - if [[ ! -z "${BASE_DIR}" ]]; then - MAVEN_PROJ_BASEDIR="\n <property>\n <name>general.maven.project.basedir</name>\n <value>${BASE_DIR}</value>\n </property>\n" - fi - - mkdir -p "${CONF_DIR}" && cp "${TEMPLATE_CONF_DIR}"/* "${CONF_DIR}"/ - - if [[ -f "${CONF_DIR}/examples/client.conf" ]]; then - cp "${CONF_DIR}"/examples/client.conf "${CONF_DIR}"/ - fi - - #Configure accumulo-env.sh - sed -e "s/\${tServerHigh_tServerLow}/${!TSERVER}/" \ - -e "s/\${masterHigh_masterLow}/${!MASTER}/" \ - -e "s/\${monitorHigh_monitorLow}/${!MONITOR}/" \ - -e "s/\${gcHigh_gcLow}/${!GC}/" \ - -e "s/\${shellHigh_shellLow}/${!SHELL}/" \ - -e "s/\${otherHigh_otherLow}/${!OTHER}/" \ - "${TEMPLATE_CONF_DIR}/$ACCUMULO_ENV" > "${CONF_DIR}/$ACCUMULO_ENV" - - #Configure accumulo-site.xml - sed -e "s/\${memMapMax}/${!MEMORY_MAP_MAX}/" \ - -e "s/\${nativeEnabled}/${!NATIVE}/" \ - -e "s/\${cacheDataSize}/${!CACHE_DATA_SIZE}/" \ - -e "s/\${cacheIndexSize}/${!CACHE_INDEX_SIZE}/" \ - -e "s/\${sortBufferSize}/${!SORT_BUFFER_SIZE}/" \ - -e "s/\${waLogMaxSize}/${!WAL_MAX_SIZE}/" \ - -e "s=\${traceUser}=${TRACE_USER}=" \ - -e "s=\${mvnProjBaseDir}=${MAVEN_PROJ_BASEDIR}=" "${TEMPLATE_CONF_DIR}/$ACCUMULO_SITE" > "${CONF_DIR}/$ACCUMULO_SITE" - - # If we're not using kerberos, filter out the krb properties - if [[ -z "${KERBEROS}" ]]; then - sed -e 's/<!-- Kerberos requirements -->/<!-- Kerberos requirements --><!--/' \ - -e 's/<!-- End Kerberos requirements -->/--><!-- End Kerberos requirements -->/' \ - "${CONF_DIR}/$ACCUMULO_SITE" > temp - mv temp "${CONF_DIR}/$ACCUMULO_SITE" - else - # Make the substitutions - sed -e "s!\${keytab}!${KEYTAB}!" \ - -e "s!\${principal}!${PRINCIPAL}!" \ - "${CONF_DIR}/${ACCUMULO_SITE}" > temp - mv temp "${CONF_DIR}/${ACCUMULO_SITE}" - fi - - # Configure hadoop version - if [[ "${HADOOP_VERSION}" == "2" ]]; then - sed -e 's/<!-- HDP 2.0 requirements -->/<!-- HDP 2.0 requirements --><!--/' \ - -e 's/<!-- End HDP 2.0 requirements -->/--><!-- End HDP 2.0 requirements -->/' \ - "${CONF_DIR}/$ACCUMULO_SITE" > temp - mv temp "${CONF_DIR}/$ACCUMULO_SITE" - sed -e 's/<!-- HDP 2.2 requirements -->/<!-- HDP 2.2 requirements --><!--/' \ - -e 's/<!-- End HDP 2.2 requirements -->/--><!-- End HDP 2.2 requirements -->/' \ - "${CONF_DIR}/$ACCUMULO_SITE" > temp - mv temp "${CONF_DIR}/$ACCUMULO_SITE" - sed -e 's/<!-- IOP 4.1 requirements -->/<!-- IOP 4.1 requirements --><!--/' \ - -e 's/<!-- End IOP 4.1 requirements -->/--><!-- End IOP 4.1 requirements -->/' \ - "${CONF_DIR}/$ACCUMULO_SITE" > temp - mv temp "${CONF_DIR}/$ACCUMULO_SITE" - elif [[ "${HADOOP_VERSION}" == "HDP2" ]]; then - sed -e 's/<!-- Hadoop 2 requirements -->/<!-- Hadoop 2 requirements --><!--/' \ - -e 's/<!-- End Hadoop 2 requirements -->/--><!-- End Hadoop 2 requirements -->/' \ - "${CONF_DIR}/$ACCUMULO_SITE" > temp - mv temp "${CONF_DIR}/$ACCUMULO_SITE" - sed -e 's/<!-- HDP 2.2 requirements -->/<!-- HDP 2.2 requirements --><!--/' \ - -e 's/<!-- End HDP 2.2 requirements -->/--><!-- End HDP 2.2 requirements -->/' \ - "${CONF_DIR}/$ACCUMULO_SITE" > temp - mv temp "${CONF_DIR}/$ACCUMULO_SITE" - sed -e 's/<!-- IOP 4.1 requirements -->/<!-- IOP 4.1 requirements --><!--/' \ - -e 's/<!-- End IOP 4.1 requirements -->/--><!-- End IOP 4.1 requirements -->/' \ - "${CONF_DIR}/$ACCUMULO_SITE" > temp - mv temp "${CONF_DIR}/$ACCUMULO_SITE" - elif [[ "${HADOOP_VERSION}" == "HDP2.2" ]]; then - sed -e 's/<!-- Hadoop 2 requirements -->/<!-- Hadoop 2 requirements --><!--/' \ - -e 's/<!-- End Hadoop 2 requirements -->/--><!-- End Hadoop 2 requirements -->/' \ - "${CONF_DIR}/$ACCUMULO_SITE" > temp - mv temp "${CONF_DIR}/$ACCUMULO_SITE" - sed -e 's/<!-- HDP 2.0 requirements -->/<!-- HDP 2.0 requirements --><!--/' \ - -e 's/<!-- End HDP 2.0 requirements -->/--><!-- End HDP 2.0 requirements -->/' \ - "${CONF_DIR}/$ACCUMULO_SITE" > temp - mv temp "${CONF_DIR}/$ACCUMULO_SITE" - sed -e 's/<!-- IOP 4.1 requirements -->/<!-- IOP 4.1 requirements --><!--/' \ - -e 's/<!-- End IOP 4.1 requirements -->/--><!-- End IOP 4.1 requirements -->/' \ - "${CONF_DIR}/$ACCUMULO_SITE" > temp - mv temp "${CONF_DIR}/$ACCUMULO_SITE" - elif [[ "${HADOOP_VERSION}" == "IOP4.1" ]]; then - sed -e 's/<!-- Hadoop 2 requirements -->/<!-- Hadoop 2 requirements --><!--/' \ - -e 's/<!-- End Hadoop 2 requirements -->/--><!-- End Hadoop 2 requirements -->/' \ - "${CONF_DIR}/$ACCUMULO_SITE" > temp - mv temp "${CONF_DIR}/$ACCUMULO_SITE" - sed -e 's/<!-- HDP 2.0 requirements -->/<!-- HDP 2.0 requirements --><!--/' \ - -e 's/<!-- End HDP 2.0 requirements -->/--><!-- End HDP 2.0 requirements -->/' \ - "${CONF_DIR}/$ACCUMULO_SITE" > temp - mv temp "${CONF_DIR}/$ACCUMULO_SITE" - sed -e 's/<!-- HDP 2.2 requirements -->/<!-- HDP 2.2 requirements --><!--/' \ - -e 's/<!-- End HDP 2.2 requirements -->/--><!-- End HDP 2.2 requirements -->/' \ - "${CONF_DIR}/$ACCUMULO_SITE" > temp - mv temp "${CONF_DIR}/$ACCUMULO_SITE" - fi - - #Additional setup steps for native configuration. - if [[ ${TYPE} == native ]]; then - if [[ $(uname) == Linux ]]; then - if [[ -z $HADOOP_PREFIX ]]; then - echo "WARNING: HADOOP_PREFIX not set, cannot automatically configure LD_LIBRARY_PATH to include Hadoop native libraries" - else - NATIVE_LIB=$(readlink -ef "$(dirname "$(for x in $(find "$HADOOP_PREFIX" -name libhadoop.so); do ld "$x" 2>/dev/null && echo "$x" && break; done)" 2>>/dev/null)" 2>>/dev/null) - if [[ -z $NATIVE_LIB ]]; then - echo -e "WARNING: The Hadoop native libraries could not be found for your sytem in: $HADOOP_PREFIX" - else - sed "/# Should the monitor/ i export LD_LIBRARY_PATH=${NATIVE_LIB}:\${LD_LIBRARY_PATH}" "${CONF_DIR}/$ACCUMULO_ENV" > temp - mv temp "${CONF_DIR}/$ACCUMULO_ENV" - echo -e "Added ${NATIVE_LIB} to the LD_LIBRARY_PATH" - fi - fi - fi - echo -e "Please remember to compile the Accumulo native libraries using the command 'bin/accumulo build-native' and to set the LD_LIBRARY_PATH variable in the ${CONF_DIR}/accumulo-env.sh script if needed." - fi - echo "Setup complete" } function main() { - # Start: Resolve Script Directory SOURCE="${BASH_SOURCE[0]}" while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink bin="$( cd -P "$( dirname "${SOURCE}" )" && pwd )" @@ -453,129 +38,108 @@ function main() { done bin="$( cd -P "$( dirname "${SOURCE}" )" && pwd )" basedir=$( cd -P "${bin}"/.. && pwd ) - # Stop: Resolve Script Directory + conf="${basedir}/conf" + lib="${basedir}/lib" - if [[ "$1" == "create-config" ]]; then - create_config "${@:2}" - exit 0 - elif [[ "$1" == "build-native" ]]; then - build_native "${@:2}" - exit 0 - fi + # Might be needed by accumulo-env.sh, accumulo-site.xml, and other Accumulo configuration + export ACCUMULO_HOME="$basedir" + export ACCUMULO_CONF_DIR="$conf" - source "$basedir"/libexec/load-env.sh + export ACCUMULO_CMD="$1" - # ACCUMULO_XTRAJARS is where all of the commandline -add items go into for reading by accumulo. - # It also holds the JAR run with the jar command and, if possible, any items in the JAR manifest's Class-Path. - if [[ "$1" = "-add" ]]; then - export ACCUMULO_XTRAJARS="$2" - shift 2 - else - export ACCUMULO_XTRAJARS="" + if [[ -z $conf || ! -d $conf ]]; then + echo "$conf is not a valid directory. Please make sure it exists" + exit 1 fi - if [[ "$1" = "jar" && -f "$2" ]]; then - if [[ $2 =~ ^/ ]]; then - jardir="$(dirname "$2")" - jarfile="$2" - else - jardir="$(pwd)" - jarfile="${jardir}/${2}" - fi - if jar tf "$jarfile" | grep -q META-INF/MANIFEST.MF ; then - cp="$(unzip -p "$jarfile" META-INF/MANIFEST.MF | grep ^Class-Path: | sed 's/^Class-Path: *//')" - if [[ -n "$cp" ]] ; then - for j in $cp; do - if [[ "$j" != "Class-Path:" ]] ; then - ACCUMULO_XTRAJARS="${jardir}/${j},$ACCUMULO_XTRAJARS" - fi - done - fi - fi - ACCUMULO_XTRAJARS="${jarfile},$ACCUMULO_XTRAJARS" + if [[ ! -f $conf/accumulo-env.sh || ! -f $conf/accumulo-site.xml ]]; then + echo "The configuration files 'accumulo-env.sh' & 'accumulo-site.xml' must exist in $conf" + echo "Copy them from $conf/examples and edit them for your environment." + exit 1 fi + source "$conf/accumulo-env.sh" - # Set up -D switches for JAAS and Kerberos if files exist - if [[ -f ${ACCUMULO_JAAS_CONF} ]]; then - ACCUMULO_GENERAL_OPTS="${ACCUMULO_GENERAL_OPTS} -Djava.security.auth.login.config=${ACCUMULO_JAAS_CONF}" - fi - if [[ -f ${ACCUMULO_KRB5_CONF} ]]; then - ACCUMULO_GENERAL_OPTS="${ACCUMULO_GENERAL_OPTS} -Djava.security.krb5.conf=${ACCUMULO_KRB5_CONF}" - fi + # Verify setting in accumulo-env.sh - # Add appropriate options for process type - case "$1" in - master) export ACCUMULO_OPTS="${ACCUMULO_GENERAL_OPTS} ${ACCUMULO_MASTER_OPTS}" ;; - gc) export ACCUMULO_OPTS="${ACCUMULO_GENERAL_OPTS} ${ACCUMULO_GC_OPTS}" ;; - tserver*) export ACCUMULO_OPTS="${ACCUMULO_GENERAL_OPTS} ${ACCUMULO_TSERVER_OPTS}" ;; - monitor) export ACCUMULO_OPTS="${ACCUMULO_GENERAL_OPTS} ${ACCUMULO_MONITOR_OPTS}" ;; - shell) export ACCUMULO_OPTS="${ACCUMULO_GENERAL_OPTS} ${ACCUMULO_SHELL_OPTS}" ;; - *) export ACCUMULO_OPTS="${ACCUMULO_GENERAL_OPTS} ${ACCUMULO_OTHER_OPTS}" ;; - esac + : "${JAVA_OPTS:?"variable is not set in accumulo-env.sh"}" + : "${ACCUMULO_LOG_DIR:?"variable is not set in accumulo-env.sh"}" + mkdir -p "${ACCUMULO_LOG_DIR}" 2>/dev/null + verify_env_dir "ACCUMULO_LOG_DIR" "${ACCUMULO_LOG_DIR}" + verify_env_dir "HADOOP_PREFIX" "${HADOOP_PREFIX}" + verify_env_dir "HADOOP_CONF_DIR" "${HADOOP_CONF_DIR}" + verify_env_dir "ZOOKEEPER_HOME" "${ZOOKEEPER_HOME}" + : "${MALLOC_ARENA_MAX:?"variable is not set in accumulo-env.sh"}" - LOG4J_JAR=$(find -H "${HADOOP_PREFIX}/lib" "${HADOOP_PREFIX}"/share/hadoop/common/lib -name 'log4j*.jar' -print 2>/dev/null | head -1) - SLF4J_JARS="${ACCUMULO_LIB_DIR}/slf4j-api.jar:${ACCUMULO_LIB_DIR}/slf4j-log4j12.jar" + export HADOOP_HOME=$HADOOP_PREFIX + export HADOOP_HOME_WARN_SUPPRESS=true - # The `find` command could fail for environmental reasons or bad configuration - # Avoid trying to run Accumulo when we can't find the jar - if [[ -z "${LOG4J_JAR}" && -z "${CLASSPATH}" ]]; then - echo "Could not locate Log4j jar in Hadoop installation at ${HADOOP_PREFIX}" - exit 1 + if [ -f "${conf}/jaas.conf" ]; then + export JAVA_OPTS=("${JAVA_OPTS[@]}" "-Djava.security.auth.login.config=${conf}/jaas.conf") + fi + if [ -f "${conf}/krb5.conf" ]; then + export JAVA_OPTS=("${JAVA_OPTS[@]}" "-Djava.security.krb5.conf=${conf}/krb5.conf") fi - if [[ ! "$1" =~ ^(gc|master|monitor|tserver|tracer)$ ]]; then - if [[ -f ${ACCUMULO_CONF_DIR}/log4j.properties ]]; then - export ACCUMULO_OPTS="${ACCUMULO_OPTS} -Dlog4j.configuration=file:${ACCUMULO_CONF_DIR}/log4j.properties" + if [[ ! "$ACCUMULO_CMD" =~ ^(gc|master|monitor|tserver|tracer)$ ]]; then + if [[ -f ${conf}/log4j.properties ]]; then + export JAVA_OPTS=("${JAVA_OPTS[@]}" "-Dlog4j.configuration=file:${conf}/log4j.properties") else - export ACCUMULO_OPTS="${ACCUMULO_OPTS} -Dlog4j.configuration=file:${ACCUMULO_CONF_DIR}/examples/log4j.properties" + export JAVA_OPTS=("${JAVA_OPTS[@]}" "-Dlog4j.configuration=file:${conf}/examples/log4j.properties") fi fi - CLASSPATH="${ACCUMULO_LIB_DIR}/accumulo-start.jar:${ACCUMULO_CONF_DIR}:${SLF4J_JARS}:${LOG4J_JAR}:${CLASSPATH}" - - # This is default for hadoop 2.x; - # for another distribution, specify (DY)LD_LIBRARY_PATH - # explicitly in ${ACCUMULO_CONF_DIR}/accumulo-env.sh - # usually something like: - # ${HADOOP_PREFIX}/lib/native/${PLATFORM} + # This is default for hadoop 2.x; for another distribution, specify (DY)LD_LIBRARY_PATH explicitly above if [ -e "${HADOOP_PREFIX}/lib/native/libhadoop.so" ]; then - LIB_PATH="${HADOOP_PREFIX}/lib/native" - LD_LIBRARY_PATH="${LIB_PATH}:${LD_LIBRARY_PATH}" # For Linux - DYLD_LIBRARY_PATH="${LIB_PATH}:${DYLD_LIBRARY_PATH}" # For Mac + export LD_LIBRARY_PATH="${HADOOP_PREFIX}/lib/native:${LD_LIBRARY_PATH}" # For Linux + export DYLD_LIBRARY_PATH="${HADOOP_PREFIX}/lib/native:${DYLD_LIBRARY_PATH}" # For Mac + fi + # ACCUMULO_XTRAJARS is where all of the commandline -add items go into for reading by accumulo. + # It also holds the JAR run with the jar command and, if possible, any items in the JAR manifest's Class-Path. + if [[ "$ACCUMULO_CMD" = "-add" ]]; then + export ACCUMULO_XTRAJARS="$2" + shift 2 + else + export ACCUMULO_XTRAJARS="" + fi + if [[ "$ACCUMULO_CMD" = "jar" && -f "$2" ]]; then + if [[ $2 =~ ^/ ]]; then + jardir="$(dirname "$2")" + jarfile="$2" + else + jardir="$(pwd)" + jarfile="${jardir}/${2}" + fi + if jar tf "$jarfile" | grep -q META-INF/MANIFEST.MF ; then + cp="$(unzip -p "$jarfile" META-INF/MANIFEST.MF | grep ^Class-Path: | sed 's/^Class-Path: *//')" + if [[ -n "$cp" ]] ; then + for j in $cp; do + if [[ "$j" != "Class-Path:" ]] ; then + ACCUMULO_XTRAJARS="${jardir}/${j},$ACCUMULO_XTRAJARS" + fi + done + fi + fi + ACCUMULO_XTRAJARS="${jarfile},$ACCUMULO_XTRAJARS" fi - # Export the variables just in case they are not exported - # This makes them available to java - export JAVA_HOME HADOOP_PREFIX ZOOKEEPER_HOME LD_LIBRARY_PATH DYLD_LIBRARY_PATH - - # Strip the instance from $1 - APP=$1 - # Avoid setting an instance unless it's necessary to ensure consistency in filenames - INSTANCE="" - # Avoid setting a pointless system property - INSTANCE_OPT="" - if [[ "$1" =~ ^tserver-[1-9][0-9]*$ ]]; then - APP="$(echo "$1" | cut -d'-' -f1)" - # Appending the trailing underscore to make single-tserver deploys look how they did - INSTANCE="$(echo "$1" | cut -d'-' -f2)_" - - #Rewrite the input arguments - set -- "$APP" "${@:2}" - - # The extra system property we'll pass to the java cmd - INSTANCE_OPT="-Daccumulo.service.instance=${INSTANCE}" + if [ -x "$JAVA_HOME/bin/java" ]; then + JAVA="$JAVA_HOME/bin/java" + else + JAVA=$(which java) + fi + if [ ! -x "$JAVA" ]; then + echo "Could not find any executable java binary. Please set java on your PATH or set JAVA_HOME" + exit 1 fi - # app isn't used anywhere, but it makes the process easier to spot when ps/top/snmp truncate the command line - JAVA="${JAVA_HOME}/bin/java" + if [ -n "$ACCUMULO_JAVA_PREFIX" ]; then + JAVA=($ACCUMULO_JAVA_PREFIX $JAVA) + fi + + CLASSPATH="${lib}/accumulo-start.jar:${conf}:${lib}/slf4j-api.jar:${lib}/slf4j-log4j12.jar:${lib}/log4j.jar:${CLASSPATH}" - ACCUMULO_OPTS_ARRAY=(${ACCUMULO_OPTS}) - exec "$JAVA" "-Dapp=$1" \ - $INSTANCE_OPT \ - "${ACCUMULO_OPTS_ARRAY[@]}" \ + exec "${JAVA[@]}" "-Dapp=$ACCUMULO_CMD" \ + "${JAVA_OPTS[@]}" \ -classpath "${CLASSPATH}" \ - -XX:OnOutOfMemoryError="${ACCUMULO_KILL_CMD:-kill -9 %p}" \ - -XX:-OmitStackTraceInFastThrow \ - -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl \ -Dhadoop.home.dir="${HADOOP_PREFIX}" \ -Dzookeeper.home.dir="${ZOOKEEPER_HOME}" \ org.apache.accumulo.start.Main \ http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/bin/accumulo-cluster ---------------------------------------------------------------------- diff --git a/assemble/bin/accumulo-cluster b/assemble/bin/accumulo-cluster index a8c0362..4f457a5 100755 --- a/assemble/bin/accumulo-cluster +++ b/assemble/bin/accumulo-cluster @@ -20,10 +20,14 @@ function print_usage { Usage: accumulo-cluster <command> (<argument> ...) Commands: - create-config Creates cluster config - start Starts Accumulo cluster - stop Stops Accumulo cluster - + create-config Creates cluster config + start Starts Accumulo cluster + stop Stops Accumulo cluster + start-non-tservers Starts all services except tservers + start-tservers Starts all tservers on cluster + stop-tservers Stops all tservers on cluster + start-here Starts all services on this node + stop-here Stops all services on this node EOF exit 1 } @@ -34,28 +38,248 @@ function invalid_args { exit 1 } -function verify_dir { - if [[ ! -d $1 ]]; then - echo "Directory does not exist: $1" +function verify_config { + if [[ -f ${conf}/slaves ]]; then + echo "ERROR: A 'slaves' file was found in ${conf}/" + echo "Accumulo now reads tablet server hosts from 'tservers' and requires that the 'slaves' file not be present to reduce confusion." + echo "Please rename the 'slaves' file to 'tservers' or remove it if both exist." exit 1 fi -} -function verify_file { - if [[ ! -f $1 ]]; then - echo "File does not exist: $1" + if [[ ! -f ${conf}/tservers ]]; then + echo "ERROR: A 'tservers' file was not found at ${conf}/tservers" + echo "Please make sure it exists and is configured with tablet server hosts." exit 1 fi + + unset master1 + if [[ -f "${conf}/masters" ]]; then + master1=$(egrep -v '(^#|^\s*$)' "${conf}/masters" | head -1) + fi + + if [[ -z "${monitor}" ]] ; then + monitor=$master1 + if [[ -f "${conf}/monitor" ]]; then + monitor=$(egrep -v '(^#|^\s*$)' "${conf}/monitor" | head -1) + fi + if [[ -z "${monitor}" ]] ; then + echo "Could not infer a Monitor role. You need to either define \"${conf}/monitor\"," + echo "or make sure \"${conf}/masters\" is non-empty." + exit 1 + fi + fi + if [[ ! -f "${conf}/tracers" ]]; then + if [[ -z "${master1}" ]] ; then + echo "Could not find a master node to use as a default for the tracer role." + echo "Either set up \"${conf}/tracers\" or make sure \"${conf}/masters\" is non-empty." + exit 1 + else + echo "$master1" > "${conf}/tracers" + fi + fi + if [[ ! -f "${conf}/gc" ]]; then + if [[ -z "${master1}" ]] ; then + echo "Could not infer a GC role. You need to either set up \"${conf}/gc\" or make sure \"${conf}/masters\" is non-empty." + exit 1 + else + echo "$master1" > "${conf}/gc" + fi + fi +} + +function get_ip() { + ip_addr=$(ip addr | grep 'state UP' -A2 | tail -n1 | awk '{print $2}' | cut -f1 -d'/') + if [[ $? != 0 ]]; then + ip_addr=$(getent ahosts "$(hostname -f)" | grep DGRAM | cut -f 1 -d ' ') + fi + echo "$ip_addr" +} + +function start_service() { + host="$1" + service="$2" + + if [[ $host == "localhost" || $host == $(hostname -f) || $host == $(hostname -s) || $host == $(get_ip) ]]; then + "${bin}/accumulo-service" "$service" start + else + $SSH "$host" "bash -c '${bin}/accumulo-service \"$service\" start'" + fi } -function copy_file { - verify_file "$1" - verify_dir "$2" - if ! cp "$1" "$2" +function start_tservers() { + echo -n "Starting tablet servers ..." + count=1 + for server in $(egrep -v '(^#|^\s*$)' "${conf}/tservers"); do + echo -n "." + start_service "$server" tserver & + if (( ++count % 72 == 0 )) ; + then + echo + wait + fi + done + echo " done" +} + +function start_all() { + unset DISPLAY + + start_service "$monitor" monitor + + if [ "$1" != "--no-tservers" ]; then + start_tservers + fi + + for host in $(egrep -v '(^#|^\s*$)' "${conf}/masters"); do + start_service "$host" master + done + + for host in $(egrep -v '(^#|^\s*$)' "${conf}/gc"); do + start_service "$host" gc + done + + for host in $(egrep -v '(^#|^\s*$)' "${conf}/tracers"); do + start_service "$host" tracer + done +} + +function start_here() { + + local_hosts="$(hostname -a 2> /dev/null) $(hostname) localhost 127.0.0.1 $(get_ip)" + for host in $local_hosts; do + if grep -q "^${host}\$" "${conf}/tservers"; then + start_service "$host" tserver + break + fi + done + + for host in $local_hosts; do + if grep -q "^${host}\$" "${conf}/masters"; then + start_service "$host" master + break + fi + done + + for host in $local_hosts; do + if grep -q "^${host}\$" "${conf}/gc"; then + start_service "$host" gc + break + fi + done + + for host in $local_hosts; do + if [ "$host" == "$monitor" ]; then + start_service "$host" monitor + break + fi + done + + for host in $local_hosts; do + if grep -q "^${host}\$" "${conf}/tracers"; then + start_service "$host" tracer + break + fi + done +} + +function end_service() { + host="$1" + service="$2" + end_cmd="$3" + if [[ $host == localhost || $host = "$(hostname -s)" || $host = "$(hostname -f)" || $host = $(get_ip) ]] ; then + "${bin}/accumulo-service" "$service" "$end_cmd" + else + $SSH "$host" "bash -c '${bin}/accumulo-service \"$service\" \"$end_cmd\"'" + fi +} + +function stop_service() { + end_service "$1" "$2" "stop" +} + +function kill_service() { + end_service "$1" "$2" "kill" +} + +function stop_tservers() { + tserver_hosts=$(egrep -v '(^#|^\s*$)' "${conf}/tservers") + + echo "Stopping unresponsive tablet servers (if any)..." + for host in ${tserver_hosts}; do + stop_service "$host" tserver & + done + + sleep 10 + + echo "Stopping unresponsive tablet servers hard (if any)..." + for host in ${tserver_hosts}; do + kill_service "$host" tserver & + done + + echo "Cleaning tablet server entries from zookeeper" + ${accumulo_cmd} org.apache.accumulo.server.util.ZooZap -tservers +} + +function stop_all() { + echo "Stopping accumulo services..." + if ! ${accumulo_cmd} admin stopAll then - echo "Failed to copy file '$1' to '$2'" - exit 1 + echo "Invalid password or unable to connect to the master" + echo "Initiating forced shutdown in 15 seconds (Ctrl-C to abort)" + sleep 10 + echo "Initiating forced shutdown in 5 seconds (Ctrl-C to abort)" + else + echo "Accumulo shut down cleanly" + echo "Utilities and unresponsive servers will shut down in 5 seconds (Ctrl-C to abort)" + fi + + sleep 5 + + # Look for processes not killed by 'admin stopAll' + for end_cmd in "stop" "kill" ; do + for master in $(grep -v '^#' "${conf}/masters"); do + end_service "$master" master $end_cmd + done + + for gc in $(grep -v '^#' "${conf}/gc"); do + end_service "$gc" gc $end_cmd + done + + end_service "$monitor" monitor $end_cmd + + for tracer in $(egrep -v '(^#|^\s*$)' "${conf}/tracers"); do + end_service "$tracer" tracer $end_cmd + done + done + + # stop tserver still running + stop_tservers + + echo "Cleaning all server entries in ZooKeeper" + ${accumulo_cmd} org.apache.accumulo.server.util.ZooZap -master -tservers -tracers --site-file "${conf}/accumulo-site.xml" +} + +function stop_here() { + # Determine hostname without errors to user + hosts_to_check=($(hostname -a 2> /dev/null | head -1) $(hostname -f)) + + if egrep -q localhost\|127.0.0.1 "${conf}/tservers"; then + ${accumulo_cmd} admin stop localhost + else + for host in "${hosts_to_check[@]}"; do + if grep -q "$host" "${conf}/tservers"; then + ${accumulo_cmd} admin stop "$host" + fi + done fi + + for host in "${hosts_to_check[@]}"; do + for end_cmd in "stop" "kill"; do + for svc in tserver gc master monitor tracer; do + end_service "$host" $svc $end_cmd + done + done + done } function main() { @@ -73,24 +297,46 @@ function main() { done bin="$( cd -P "$( dirname "${SOURCE}" )" && pwd )" basedir=$( cd -P "${bin}"/.. && pwd ) + conf="${basedir}/conf" - source "$basedir"/libexec/load-env.sh - - verify_file "${ACCUMULO_LIBEXEC_DIR}/cluster.sh" + accumulo_cmd="${bin}/accumulo" + SSH='ssh -qnf -o ConnectTimeout=2' case "$1" in create-config) - copy_file "${ACCUMULO_CONF_DIR}/examples/gc" "${ACCUMULO_CONF_DIR}" - copy_file "${ACCUMULO_CONF_DIR}/examples/masters" "${ACCUMULO_CONF_DIR}" - copy_file "${ACCUMULO_CONF_DIR}/examples/monitor" "${ACCUMULO_CONF_DIR}" - copy_file "${ACCUMULO_CONF_DIR}/examples/tservers" "${ACCUMULO_CONF_DIR}" - copy_file "${ACCUMULO_CONF_DIR}/examples/tracers" "${ACCUMULO_CONF_DIR}" + echo "localhost" > "$conf/gc" + echo "localhost" > "$conf/masters" + echo "localhost" > "$conf/monitor" + echo "localhost" > "$conf/tracers" + echo "localhost" > "$conf/tservers" ;; start) - "${ACCUMULO_LIBEXEC_DIR}"/cluster.sh start-all + verify_config + start_all ;; stop) - "${ACCUMULO_LIBEXEC_DIR}"/cluster.sh stop-all + verify_config + stop_all + ;; + start-non-tservers) + verify_config + start_all --no-tservers + ;; + start-tservers) + verify_config + start_tservers + ;; + start-here) + verify_config + start_here + ;; + stop-tservers) + verify_config + stop_tservers + ;; + stop-here) + verify_config + stop_here ;; *) invalid_args "'$1' is an invalid <command>" http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/bin/accumulo-service ---------------------------------------------------------------------- diff --git a/assemble/bin/accumulo-service b/assemble/bin/accumulo-service index bf01df9..c9b0a17 100755 --- a/assemble/bin/accumulo-service +++ b/assemble/bin/accumulo-service @@ -42,7 +42,10 @@ function invalid_args { } function get_host { - host="$(hostname -s)" + host="$(hostname -f)" + if [ "$1" == "-s" ]; then + host="$(hostname -s)" + fi if [[ -z "$host" ]]; then host=$(ip addr | grep 'state UP' -A2 | tail -n1 | awk '{print $2}' | cut -f1 -d'/') if [[ $? != 0 ]]; then @@ -52,8 +55,76 @@ function get_host { echo "$host" } -function main() { +function rotate_log () { + logfile="$1" + max_retained="5" + if [ -f "$logfile" ]; then + while [[ "$max_retained" -gt 1 ]]; do + prev=$(( max_retained - 1)) + [ -f "$logfile.$prev" ] && mv -f "$logfile.$prev" "$logfile.$max_retained" + max_retained=$prev + done + mv -f "$logfile" "$logfile.$max_retained"; + fi +} + +function start_service() { + host="$(get_host -s)" + + if [ -f "$pid_file" ]; then + pid=$(cat "$pid_file") + if kill -0 "$pid" 2>/dev/null; then + echo "$host : $service already running (${pid})" + exit 0 + fi + fi + echo "Starting $service on $host" + + if [[ $service == "master" ]]; then + "${bin}/accumulo" org.apache.accumulo.master.state.SetGoalState NORMAL + fi + + outfile="${ACCUMULO_LOG_DIR}/${service}_${host}.out" + errfile="${ACCUMULO_LOG_DIR}/${service}_${host}.err" + rotate_log "$outfile" + rotate_log "$errfile" + + export ACCUMULO_MONITOR_BIND_ALL=${ACCUMULO_MONITOR_BIND_ALL:-"true"} + extra_args="" + if [[ $service == "monitor" && $ACCUMULO_MONITOR_BIND_ALL == "true" ]]; then + extra_args=('--address' '0.0.0.0') + fi + + nohup "${bin}/accumulo" "$service" "${extra_args[@]}" >"$outfile" 2>"$errfile" < /dev/null & + echo "$!" > "${pid_file}" + # Check the max open files limit and selectively warn + max_files_open=$(ulimit -n) + if [[ -n $max_files_open ]] ; then + max_files_recommended=32768 + if (( max_files_open < max_files_recommended )); then + echo "WARN : Max open files on $host is $max_files_open, recommend $max_files_recommended" >&2 + fi + fi +} + +function stop_service() { + if [ -f "$pid_file" ]; then + echo "Stopping $service on $(get_host -s)"; + kill -s TERM "$(cat "$pid_file")" 2>/dev/null + rm -f "${pid_file}" 2>/dev/null + fi +} + +function kill_service() { + if [ -f "$pid_file" ]; then + echo "Killing $service on $(get_host -s)"; + kill -s KILL "$(cat "$pid_file")" 2>/dev/null + rm -f "${pid_file}" 2>/dev/null + fi +} + +function main() { if [[ -z $1 ]]; then invalid_args "<service> cannot be empty" fi @@ -67,10 +138,18 @@ function main() { done bin="$( cd -P "$( dirname "${SOURCE}" )" && pwd )" basedir=$( cd -P "${bin}"/.. && pwd ) + conf="${basedir}/conf" + + if [ -f "${conf}/accumulo-env.sh" ]; then + source "${conf}/accumulo-env.sh" + fi + ACCUMULO_LOG_DIR="${ACCUMULO_LOG_DIR:-${basedir}/logs}" - source "$basedir"/libexec/load-env.sh + mkdir -p "$ACCUMULO_LOG_DIR" 2>/dev/null + mkdir -p "${basedir}/run" 2>/dev/null - service=$1 + service="$1" + pid_file="${basedir}/run/accumulo-${service}.pid" case "$service" in gc|master|monitor|tserver|tracer) if [[ -z $2 ]]; then @@ -78,13 +157,13 @@ function main() { fi case "$2" in start) - "${ACCUMULO_LIBEXEC_DIR}/service.sh" start "$(get_host)" "$1" + start_service ;; stop) - "${ACCUMULO_LIBEXEC_DIR}/service.sh" stop "$(get_host)" "$service" TERM + stop_service ;; kill) - "${ACCUMULO_LIBEXEC_DIR}/service.sh" stop "$(get_host)" "$service" KILL + kill_service ;; *) invalid_args "'$2' is an invalid <command>" @@ -92,7 +171,7 @@ function main() { esac ;; *) - invalid_args "'$1' is an invalid <service>" + invalid_args "'$service' is an invalid <service>" ;; esac } http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/bin/accumulo-util ---------------------------------------------------------------------- diff --git a/assemble/bin/accumulo-util b/assemble/bin/accumulo-util new file mode 100755 index 0000000..55e89bd --- /dev/null +++ b/assemble/bin/accumulo-util @@ -0,0 +1,686 @@ +#! /usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +function print_usage { + cat <<EOF +Usage: accumulo-util <command> (<argument> ...) + +Commands: + create-config Creates Accumulo configuration + build-native Builds Accumulo native libraries + hadoop-jar Runs 'hadoop jar' command with Accumulo jars + gen-monitor-cert Generates Accumulo monitor certficate + load-jars-hdfs Loads Accumulo jars in lib/ to HDFS for VFS classloader + +EOF + exit 1 +} + +function create_config_usage() { + cat <<EOF +Usage: accumulo-util create-config [-options] +where options include (long options not available on all platforms): + -d, --dir Alternate directory to setup config files + -s, --size Supported sizes: '1GB' '2GB' '3GB' '512MB' + -n, --native Configure to use native libraries + -j, --jvm Configure to use the jvm + -o, --overwrite Overwrite the default config directory + -v, --version Specify the Apache Hadoop version supported versions: '1' '2' + -k, --kerberos Configure for use with Kerberos + -h, --help Print this help message +EOF +} + +function create_config() { + TEMPLATE_CONF_DIR="${conf}/examples/templates" + CONF_DIR="${conf}" + ACCUMULO_SITE=accumulo-site.xml + ACCUMULO_ENV=accumulo-env.sh + + SIZE= + TYPE= + HADOOP_VERSION= + OVERWRITE="0" + BASE_DIR= + KERBEROS= + + #Execute getopt + if [[ $(uname -s) == "Linux" ]]; then + args=$(getopt -o "b:d:s:njokv:h" -l "basedir:,dir:,size:,native,jvm,overwrite,kerberos,version:,help" -q -- "$@") + else # Darwin, BSD + args=$(getopt b:d:s:njokv:h "$@") + fi + + #Bad arguments + if [[ $? != 0 ]]; then + create_config_usage 1>&2 + exit 1 + fi + eval set -- "${args[@]}" + + for i + do + case "$i" in + -b|--basedir) #Hidden option used to set general.maven.project.basedir for developers + BASE_DIR=$2; shift + shift;; + -d|--dir) + CONF_DIR=$2; shift + shift;; + -s|--size) + SIZE=$2; shift + shift;; + -n|--native) + TYPE=native + shift;; + -j|--jvm) + TYPE=jvm + shift;; + -o|--overwrite) + OVERWRITE=1 + shift;; + -v|--version) + HADOOP_VERSION=$2; shift + shift;; + -k|--kerberos) + KERBEROS="true" + shift;; + -h|--help) + create_config_usage + exit 0 + shift;; + --) + shift + break;; + esac + done + + while [[ "${OVERWRITE}" = "0" ]]; do + if [[ -e "${CONF_DIR}/${ACCUMULO_ENV}" || -e "${CONF_DIR}/${ACCUMULO_SITE}" ]]; then + echo "Warning your current config files in ${CONF_DIR} will be overwritten!" + echo + echo "How would you like to proceed?:" + select CHOICE in 'Continue with overwrite' 'Specify new conf dir'; do + if [[ "${CHOICE}" = 'Specify new conf dir' ]]; then + echo -n "Please specifiy new conf directory: " + read CONF_DIR + elif [[ "${CHOICE}" = 'Continue with overwrite' ]]; then + OVERWRITE=1 + fi + break + done + else + OVERWRITE=1 + fi + done + echo "Copying configuration files to: ${CONF_DIR}" + + #Native 1GB + native_1GB_tServer="'-Xmx128m' '-Xms128m'" + _1GB_master="'-Xmx128m' '-Xms128m'" + _1GB_monitor="'-Xmx64m' '-Xms64m'" + _1GB_gc="'-Xmx64m' '-Xms64m'" + _1GB_other="'-Xmx128m' '-Xms64m'" + _1GB_shell="${_1GB_other}" + + _1GB_memoryMapMax="256M" + native_1GB_nativeEnabled="true" + _1GB_cacheDataSize="15M" + _1GB_cacheIndexSize="40M" + _1GB_sortBufferSize="50M" + _1GB_waLogMaxSize="256M" + + #Native 2GB + native_2GB_tServer="'-Xmx256m' '-Xms256m'" + _2GB_master="'-Xmx256m' '-Xms256m'" + _2GB_monitor="'-Xmx128m' '-Xms64m'" + _2GB_gc="'-Xmx128m' '-Xms128m'" + _2GB_other="'-Xmx256m' '-Xms64m'" + _2GB_shell="${_2GB_other}" + + _2GB_memoryMapMax="512M" + native_2GB_nativeEnabled="true" + _2GB_cacheDataSize="30M" + _2GB_cacheIndexSize="80M" + _2GB_sortBufferSize="50M" + _2GB_waLogMaxSize="512M" + + #Native 3GB + native_3GB_tServer="'-Xmx1g' '-Xms1g' '-XX:NewSize=500m' '-XX:MaxNewSize=500m'" + _3GB_master="'-Xmx1g' '-Xms1g'" + _3GB_monitor="'-Xmx1g' '-Xms256m'" + _3GB_gc="'-Xmx256m' '-Xms256m'" + _3GB_other="'-Xmx1g' '-Xms256m'" + _3GB_shell="${_3GB_other}" + + _3GB_memoryMapMax="1G" + native_3GB_nativeEnabled="true" + _3GB_cacheDataSize="128M" + _3GB_cacheIndexSize="128M" + _3GB_sortBufferSize="200M" + _3GB_waLogMaxSize="1G" + + #Native 512MB + native_512MB_tServer="'-Xmx48m' '-Xms48m'" + _512MB_master="'-Xmx128m' '-Xms128m'" + _512MB_monitor="'-Xmx64m' '-Xms64m'" + _512MB_gc="'-Xmx64m' '-Xms64m'" + _512MB_other="'-Xmx128m' '-Xms64m'" + _512MB_shell="${_512MB_other}" + + _512MB_memoryMapMax="80M" + native_512MB_nativeEnabled="true" + _512MB_cacheDataSize="7M" + _512MB_cacheIndexSize="20M" + _512MB_sortBufferSize="50M" + _512MB_waLogMaxSize="100M" + + #JVM 1GB + jvm_1GB_tServer="'-Xmx384m' '-Xms384m'" + + jvm_1GB_nativeEnabled="false" + + #JVM 2GB + jvm_2GB_tServer="'-Xmx768m' '-Xms768m'" + + jvm_2GB_nativeEnabled="false" + + #JVM 3GB + jvm_3GB_tServer="'-Xmx2g' '-Xms2g' '-XX:NewSize=1G' '-XX:MaxNewSize=1G'" + + jvm_3GB_nativeEnabled="false" + + #JVM 512MB + jvm_512MB_tServer="'-Xmx128m' '-Xms128m'" + + jvm_512MB_nativeEnabled="false" + + + if [[ -z "${SIZE}" ]]; then + echo "Choose the heap configuration:" + select DIRNAME in 1GB 2GB 3GB 512MB; do + echo "Using '${DIRNAME}' configuration" + SIZE=${DIRNAME} + break + done + elif [[ "${SIZE}" != "1GB" && "${SIZE}" != "2GB" && "${SIZE}" != "3GB" && "${SIZE}" != "512MB" ]]; then + echo "Invalid memory size" + echo "Supported sizes: '1GB' '2GB' '3GB' '512MB'" + exit 1 + fi + + if [[ -z "${TYPE}" ]]; then + echo + echo "Choose the Accumulo memory-map type:" + select TYPENAME in Java Native; do + if [[ "${TYPENAME}" == "Native" ]]; then + TYPE="native" + echo "Don't forget to build the native libraries using the command 'accumulo-util build-native'" + elif [[ "${TYPENAME}" == "Java" ]]; then + TYPE="jvm" + fi + echo "Using '${TYPE}' configuration" + echo + break + done + fi + + if [[ -z "${HADOOP_VERSION}" ]]; then + echo + echo "Choose the Apache Hadoop version:" + select HADOOP in 'Hadoop 2' 'HDP 2.0/2.1' 'HDP 2.2' 'IOP 4.1'; do + if [ "${HADOOP}" == "Hadoop 2" ]; then + HADOOP_VERSION="2" + elif [ "${HADOOP}" == "HDP 2.0/2.1" ]; then + HADOOP_VERSION="HDP2" + elif [ "${HADOOP}" == "HDP 2.2" ]; then + HADOOP_VERSION="HDP2.2" + elif [ "${HADOOP}" == "IOP 4.1" ]; then + HADOOP_VERSION="IOP4.1" + fi + echo "Using Hadoop version '${HADOOP_VERSION}' configuration" + echo + break + done + elif [[ "${HADOOP_VERSION}" != "2" && "${HADOOP_VERSION}" != "HDP2" && "${HADOOP_VERSION}" != "HDP2.2" ]]; then + echo "Invalid Hadoop version" + echo "Supported Hadoop versions: '2', 'HDP2', 'HDP2.2'" + exit 1 + fi + + TRACE_USER="root" + + if [[ ! -z "${KERBEROS}" ]]; then + echo + read -p "Enter server's Kerberos principal: " PRINCIPAL + read -p "Enter server's Kerberos keytab: " KEYTAB + TRACE_USER="${PRINCIPAL}" + fi + + for var in SIZE TYPE HADOOP_VERSION; do + if [[ -z ${!var} ]]; then + echo "Invalid $var configuration" + exit 1 + fi + done + + TSERVER="${TYPE}_${SIZE}_tServer" + MASTER="_${SIZE}_master" + MONITOR="_${SIZE}_monitor" + GC="_${SIZE}_gc" + SHELL="_${SIZE}_shell" + OTHER="_${SIZE}_other" + + MEMORY_MAP_MAX="_${SIZE}_memoryMapMax" + NATIVE="${TYPE}_${SIZE}_nativeEnabled" + CACHE_DATA_SIZE="_${SIZE}_cacheDataSize" + CACHE_INDEX_SIZE="_${SIZE}_cacheIndexSize" + SORT_BUFFER_SIZE="_${SIZE}_sortBufferSize" + WAL_MAX_SIZE="_${SIZE}_waLogMaxSize" + + MAVEN_PROJ_BASEDIR="" + + if [[ ! -z "${BASE_DIR}" ]]; then + MAVEN_PROJ_BASEDIR="\n <property>\n <name>general.maven.project.basedir</name>\n <value>${BASE_DIR}</value>\n </property>\n" + fi + + mkdir -p "${CONF_DIR}" && cp "${TEMPLATE_CONF_DIR}"/* "${CONF_DIR}"/ + + if [[ -f "${CONF_DIR}/examples/client.conf" ]]; then + cp "${CONF_DIR}"/examples/client.conf "${CONF_DIR}"/ + fi + + #Configure accumulo-env.sh + sed -e "s/\${tServerHigh_tServerLow}/${!TSERVER}/" \ + -e "s/\${masterHigh_masterLow}/${!MASTER}/" \ + -e "s/\${monitorHigh_monitorLow}/${!MONITOR}/" \ + -e "s/\${gcHigh_gcLow}/${!GC}/" \ + -e "s/\${shellHigh_shellLow}/${!SHELL}/" \ + -e "s/\${otherHigh_otherLow}/${!OTHER}/" \ + "${TEMPLATE_CONF_DIR}/$ACCUMULO_ENV" > "${CONF_DIR}/$ACCUMULO_ENV" + + #Configure accumulo-site.xml + sed -e "s/\${memMapMax}/${!MEMORY_MAP_MAX}/" \ + -e "s/\${nativeEnabled}/${!NATIVE}/" \ + -e "s/\${cacheDataSize}/${!CACHE_DATA_SIZE}/" \ + -e "s/\${cacheIndexSize}/${!CACHE_INDEX_SIZE}/" \ + -e "s/\${sortBufferSize}/${!SORT_BUFFER_SIZE}/" \ + -e "s/\${waLogMaxSize}/${!WAL_MAX_SIZE}/" \ + -e "s=\${traceUser}=${TRACE_USER}=" \ + -e "s=\${mvnProjBaseDir}=${MAVEN_PROJ_BASEDIR}=" "${TEMPLATE_CONF_DIR}/$ACCUMULO_SITE" > "${CONF_DIR}/$ACCUMULO_SITE" + + # If we're not using kerberos, filter out the krb properties + if [[ -z "${KERBEROS}" ]]; then + sed -e 's/<!-- Kerberos requirements -->/<!-- Kerberos requirements --><!--/' \ + -e 's/<!-- End Kerberos requirements -->/--><!-- End Kerberos requirements -->/' \ + "${CONF_DIR}/$ACCUMULO_SITE" > temp + mv temp "${CONF_DIR}/$ACCUMULO_SITE" + else + # Make the substitutions + sed -e "s!\${keytab}!${KEYTAB}!" \ + -e "s!\${principal}!${PRINCIPAL}!" \ + "${CONF_DIR}/${ACCUMULO_SITE}" > temp + mv temp "${CONF_DIR}/${ACCUMULO_SITE}" + fi + + # Configure hadoop version + if [[ "${HADOOP_VERSION}" == "2" ]]; then + sed -e 's/<!-- HDP 2.0 requirements -->/<!-- HDP 2.0 requirements --><!--/' \ + -e 's/<!-- End HDP 2.0 requirements -->/--><!-- End HDP 2.0 requirements -->/' \ + "${CONF_DIR}/$ACCUMULO_SITE" > temp + mv temp "${CONF_DIR}/$ACCUMULO_SITE" + sed -e 's/<!-- HDP 2.2 requirements -->/<!-- HDP 2.2 requirements --><!--/' \ + -e 's/<!-- End HDP 2.2 requirements -->/--><!-- End HDP 2.2 requirements -->/' \ + "${CONF_DIR}/$ACCUMULO_SITE" > temp + mv temp "${CONF_DIR}/$ACCUMULO_SITE" + sed -e 's/<!-- IOP 4.1 requirements -->/<!-- IOP 4.1 requirements --><!--/' \ + -e 's/<!-- End IOP 4.1 requirements -->/--><!-- End IOP 4.1 requirements -->/' \ + "${CONF_DIR}/$ACCUMULO_SITE" > temp + mv temp "${CONF_DIR}/$ACCUMULO_SITE" + elif [[ "${HADOOP_VERSION}" == "HDP2" ]]; then + sed -e 's/<!-- Hadoop 2 requirements -->/<!-- Hadoop 2 requirements --><!--/' \ + -e 's/<!-- End Hadoop 2 requirements -->/--><!-- End Hadoop 2 requirements -->/' \ + "${CONF_DIR}/$ACCUMULO_SITE" > temp + mv temp "${CONF_DIR}/$ACCUMULO_SITE" + sed -e 's/<!-- HDP 2.2 requirements -->/<!-- HDP 2.2 requirements --><!--/' \ + -e 's/<!-- End HDP 2.2 requirements -->/--><!-- End HDP 2.2 requirements -->/' \ + "${CONF_DIR}/$ACCUMULO_SITE" > temp + mv temp "${CONF_DIR}/$ACCUMULO_SITE" + sed -e 's/<!-- IOP 4.1 requirements -->/<!-- IOP 4.1 requirements --><!--/' \ + -e 's/<!-- End IOP 4.1 requirements -->/--><!-- End IOP 4.1 requirements -->/' \ + "${CONF_DIR}/$ACCUMULO_SITE" > temp + mv temp "${CONF_DIR}/$ACCUMULO_SITE" + elif [[ "${HADOOP_VERSION}" == "HDP2.2" ]]; then + sed -e 's/<!-- Hadoop 2 requirements -->/<!-- Hadoop 2 requirements --><!--/' \ + -e 's/<!-- End Hadoop 2 requirements -->/--><!-- End Hadoop 2 requirements -->/' \ + "${CONF_DIR}/$ACCUMULO_SITE" > temp + mv temp "${CONF_DIR}/$ACCUMULO_SITE" + sed -e 's/<!-- HDP 2.0 requirements -->/<!-- HDP 2.0 requirements --><!--/' \ + -e 's/<!-- End HDP 2.0 requirements -->/--><!-- End HDP 2.0 requirements -->/' \ + "${CONF_DIR}/$ACCUMULO_SITE" > temp + mv temp "${CONF_DIR}/$ACCUMULO_SITE" + sed -e 's/<!-- IOP 4.1 requirements -->/<!-- IOP 4.1 requirements --><!--/' \ + -e 's/<!-- End IOP 4.1 requirements -->/--><!-- End IOP 4.1 requirements -->/' \ + "${CONF_DIR}/$ACCUMULO_SITE" > temp + mv temp "${CONF_DIR}/$ACCUMULO_SITE" + elif [[ "${HADOOP_VERSION}" == "IOP4.1" ]]; then + sed -e 's/<!-- Hadoop 2 requirements -->/<!-- Hadoop 2 requirements --><!--/' \ + -e 's/<!-- End Hadoop 2 requirements -->/--><!-- End Hadoop 2 requirements -->/' \ + "${CONF_DIR}/$ACCUMULO_SITE" > temp + mv temp "${CONF_DIR}/$ACCUMULO_SITE" + sed -e 's/<!-- HDP 2.0 requirements -->/<!-- HDP 2.0 requirements --><!--/' \ + -e 's/<!-- End HDP 2.0 requirements -->/--><!-- End HDP 2.0 requirements -->/' \ + "${CONF_DIR}/$ACCUMULO_SITE" > temp + mv temp "${CONF_DIR}/$ACCUMULO_SITE" + sed -e 's/<!-- HDP 2.2 requirements -->/<!-- HDP 2.2 requirements --><!--/' \ + -e 's/<!-- End HDP 2.2 requirements -->/--><!-- End HDP 2.2 requirements -->/' \ + "${CONF_DIR}/$ACCUMULO_SITE" > temp + mv temp "${CONF_DIR}/$ACCUMULO_SITE" + fi + + #Additional setup steps for native configuration. + if [[ ${TYPE} == native ]]; then + if [[ $(uname) == Linux ]]; then + if [[ -z $HADOOP_PREFIX ]]; then + echo "WARNING: HADOOP_PREFIX not set, cannot automatically configure LD_LIBRARY_PATH to include Hadoop native libraries" + else + NATIVE_LIB=$(readlink -ef "$(dirname "$(for x in $(find "$HADOOP_PREFIX" -name libhadoop.so); do ld "$x" 2>/dev/null && echo "$x" && break; done)" 2>>/dev/null)" 2>>/dev/null) + if [[ -z $NATIVE_LIB ]]; then + echo -e "WARNING: The Hadoop native libraries could not be found for your sytem in: $HADOOP_PREFIX" + else + sed "/# Should the monitor/ i export LD_LIBRARY_PATH=${NATIVE_LIB}:\${LD_LIBRARY_PATH}" "${CONF_DIR}/$ACCUMULO_ENV" > temp + mv temp "${CONF_DIR}/$ACCUMULO_ENV" + echo -e "Added ${NATIVE_LIB} to the LD_LIBRARY_PATH" + fi + fi + fi + echo -e "Please remember to compile the Accumulo native libraries using the command 'accumulo-util build-native' and to set the LD_LIBRARY_PATH variable in the ${CONF_DIR}/accumulo-env.sh if needed." + fi + + echo "Setup complete" +} + +function build_native() { + native_tarball="$basedir/lib/accumulo-native.tar.gz" + final_native_target="$basedir/lib/native" + + if [[ ! -f $native_tarball ]]; then + echo "Could not find native code artifact: ${native_tarball}"; + exit 1 + fi + + # Make the destination for the native library + mkdir -p "${final_native_target}" || exit 1 + + # Make a directory for us to unpack the native source into + TMP_DIR=$(mktemp -d /tmp/accumulo-native.XXXX) || exit 1 + + # Unpack the tarball to our temp directory + if ! tar xf "${native_tarball}" -C "${TMP_DIR}" + then + echo "Failed to unpack native tarball to ${TMP_DIR}" + exit 1 + fi + + # Move to the first (only) directory in our unpacked tarball + native_dir=$(find "${TMP_DIR}" -maxdepth 1 -mindepth 1 -type d) + + cd "${native_dir}" || exit 1 + + # Make the native library + export USERFLAGS="$*" + make || { echo 'Make failed!'; exit 1; } + + # "install" the artifact + cp libaccumulo.* "${final_native_target}" || exit 1 + + # Clean up our temp directory + rm -rf "${TMP_DIR}" + + echo "Successfully installed native library" +} + +function gen_monitor_cert() { + if [[ -z "$JAVA_HOME" || -d "$JAVA_HOME" ]]; then + echo "JAVA_HOME=${JAVA_HOME} must be set and exist" + exit 1 + fi + + ALIAS="default" + KEYPASS=$(LC_CTYPE=C tr -dc '#-~' < /dev/urandom | tr -d '<>&' | head -c 20) + STOREPASS=$(LC_CTYPE=C tr -dc '#-~' < /dev/urandom | tr -d '<>&' | head -c 20) + KEYSTOREPATH="${conf}/keystore.jks" + TRUSTSTOREPATH="${conf}/conf/cacerts.jks" + CERTPATH="${conf}/server.cer" + + if [[ -e "$KEYSTOREPATH" ]]; then + rm -i "$KEYSTOREPATH" + if [[ -e "$KEYSTOREPATH" ]]; then + echo "KeyStore already exists, exiting" + exit 1 + fi + fi + if [[ -e "$TRUSTSTOREPATH" ]]; then + rm -i "$TRUSTSTOREPATH" + if [[ -e "$TRUSTSTOREPATH" ]]; then + echo "TrustStore already exists, exiting" + exit 2 + fi + fi + if [[ -e "$CERTPATH" ]]; then + rm -i "$CERTPATH" + if [[ -e "$CERTPATH" ]]; then + echo "Certificate already exists, exiting" + exit 3 + fi + fi + + "${JAVA_HOME}/bin/keytool" -genkey -alias "$ALIAS" -keyalg RSA -keypass "$KEYPASS" -storepass "$KEYPASS" -keystore "$KEYSTOREPATH" + "${JAVA_HOME}/bin/keytool" -export -alias "$ALIAS" -storepass "$KEYPASS" -file "$CERTPATH" -keystore "$KEYSTOREPATH" + "${JAVA_HOME}/bin/keytool" -import -v -trustcacerts -alias "$ALIAS" -file "$CERTPATH" -keystore "$TRUSTSTOREPATH" -storepass "$STOREPASS" <<< "yes" + + echo + echo "keystore and truststore generated. now add the following to accumulo-site.xml:" + echo + echo " <property>" + echo " <name>monitor.ssl.keyStore</name>" + echo " <value>$KEYSTOREPATH</value>" + echo " </property>" + echo " <property>" + echo " <name>monitor.ssl.keyStorePassword</name>" + echo " <value>$KEYPASS</value>" + echo " </property>" + echo " <property>" + echo " <name>monitor.ssl.trustStore</name>" + echo " <value>$TRUSTSTOREPATH</value>" + echo " </property>" + echo " <property>" + echo " <name>monitor.ssl.trustStorePassword</name>" + echo " <value>$STOREPASS</value>" + echo " </property>" + echo +} + +function load_jars_hdfs() { + export ACCUMULO_HOME="$basedir" + + if [ -f "${conf}/accumulo-env.sh" ]; then + source "$conf/accumulo-env.sh" + fi + if [ -z "$HADOOP_PREFIX" ]; then + echo "HADOOP_PREFIX is not set. Please make sure it's set globally or in $conf/accumulo-env.sh" + exit 1 + fi + + # Find the system context directory in HDFS + SYSTEM_CONTEXT_HDFS_DIR=$(grep -A1 "general.vfs.classpaths" "$conf/accumulo-site.xml" | tail -1 | perl -pe 's/\s+<value>//; s/<\/value>//; s/,.+$//; s|[^/]+$||; print $ARGV[1]') + + if [ -z "$SYSTEM_CONTEXT_HDFS_DIR" ] + then + echo "Your accumulo-site.xml file is not set up for the HDFS Classloader. Please add the following to your accumulo-site.xml file where ##CLASSPATH## is one of the following formats:" + echo "A single directory: hdfs://host:port/directory/" + echo "A single directory with a regex: hdfs://host:port/directory/.*.jar" + echo "Multiple directories: hdfs://host:port/directory/.*.jar,hdfs://host:port/directory2/" + echo "" + echo "<property>" + echo " <name>general.vfs.classpaths</name>" + echo " <value>##CLASSPATH##</value>" + echo " <description>location of the jars for the default (system) context</description>" + echo "</property>" + exit 1 + fi + + # Create the system context directy in HDFS if it does not exist + "$HADOOP_PREFIX/bin/hadoop" fs -ls "$SYSTEM_CONTEXT_HDFS_DIR" > /dev/null + if [[ $? != 0 ]]; then + "$HADOOP_PREFIX/bin/hadoop" fs -mkdir "$SYSTEM_CONTEXT_HDFS_DIR" > /dev/null + if [[ $? != 0 ]]; then + echo "Unable to create classpath directory at $SYSTEM_CONTEXT_HDFS_DIR" + exit 1 + fi + fi + + # Replicate to all tservers to avoid network contention on startup + TSERVERS=${conf}/tservers + NUM_TSERVERS=$(egrep -v '(^#|^\s*$)' "$TSERVERS" | wc -l) + + #let each datanode service around 50 clients + REP=$(( NUM_TSERVERS / 50 )) + (( REP < 3 )) && REP=3 + + # Copy all jars in lib to the system context directory + "$HADOOP_PREFIX/bin/hadoop" fs -moveFromLocal "$lib"/*.jar "$SYSTEM_CONTEXT_HDFS_DIR" > /dev/null + "$HADOOP_PREFIX/bin/hadoop" fs -setrep -R $REP "$SYSTEM_CONTEXT_HDFS_DIR" > /dev/null + + # We need some of the jars in lib, copy them back out and remove them from the system context dir + "$HADOOP_PREFIX/bin/hadoop" fs -copyToLocal "$SYSTEM_CONTEXT_HDFS_DIR/commons-vfs2.jar" "$lib/." > /dev/null + "$HADOOP_PREFIX/bin/hadoop" fs -rm "$SYSTEM_CONTEXT_HDFS_DIR/commons-vfs2.jar" > /dev/null + "$HADOOP_PREFIX/bin/hadoop" fs -copyToLocal "$SYSTEM_CONTEXT_HDFS_DIR/accumulo-start.jar" "$lib/." > /dev/null + "$HADOOP_PREFIX/bin/hadoop" fs -rm "$SYSTEM_CONTEXT_HDFS_DIR/accumulo-start.jar" > /dev/null + "$HADOOP_PREFIX/bin/hadoop" fs -copyToLocal "$SYSTEM_CONTEXT_HDFS_DIR/slf4j*.jar" "$lib/." > /dev/null + "$HADOOP_PREFIX/bin/hadoop" fs -rm "$SYSTEM_CONTEXT_HDFS_DIR/slf4j*.jar" > /dev/null + for f in $(grep -v '^#' "${conf}/tservers") + do + rsync -ra --delete "$ACCUMULO_HOME" "$(dirname "$ACCUMULO_HOME")" + done +} + +function hadoop_jar() { + if [ -f "${conf}/accumulo-env.sh" ]; then + source "$conf/accumulo-env.sh" + fi + if [ -z "$HADOOP_PREFIX" ]; then + echo "HADOOP_PREFIX is not set. Please make sure it's set globally or in $conf/accumulo-env.sh" + exit 1 + fi + if [ -z "$ZOOKEEPER_HOME" ]; then + echo "ZOOKEEPER_HOME is not set. Please make sure it's set globally or in $conf/accumulo-env.sh" + exit 1 + fi + + ZOOKEEPER_CMD="ls -1 $ZOOKEEPER_HOME/zookeeper-[0-9]*[^csn].jar " + if [[ $(eval "$ZOOKEEPER_CMD" | wc -l) -ne 1 ]] ; then + echo "Not exactly one zookeeper jar in $ZOOKEEPER_HOME" + exit 1 + fi + ZOOKEEPER_LIB=$(eval "$ZOOKEEPER_CMD") + + CORE_LIB="${lib}/accumulo-core.jar" + FATE_LIB="${lib}/accumulo-fate.jar" + THRIFT_LIB="${lib}/libthrift.jar" + JCOMMANDER_LIB="${lib}/jcommander.jar" + COMMONS_VFS_LIB="${lib}/commons-vfs2.jar" + GUAVA_LIB="${lib}/guava.jar" + HTRACE_LIB="${lib}/htrace-core.jar" + + USERJARS=" " + for arg in "$@"; do + if [ "$arg" != "-libjars" -a -z "$TOOLJAR" ]; then + TOOLJAR="$arg" + shift + elif [ "$arg" != "-libjars" -a -z "$CLASSNAME" ]; then + CLASSNAME="$arg" + shift + elif [ -z "$USERJARS" ]; then + USERJARS=$(echo "$arg" | tr "," " ") + shift + elif [ "$arg" = "-libjars" ]; then + USERJARS="" + shift + else + break + fi + done + + LIB_JARS="$THRIFT_LIB,$CORE_LIB,$FATE_LIB,$ZOOKEEPER_LIB,$JCOMMANDER_LIB,$COMMONS_VFS_LIB,$GUAVA_LIB,$HTRACE_LIB" + H_JARS="$THRIFT_LIB:$CORE_LIB:$FATE_LIB:$ZOOKEEPER_LIB:$JCOMMANDER_LIB:$COMMONS_VFS_LIB:$GUAVA_LIB:$HTRACE_LIB" + + for jar in $USERJARS; do + LIB_JARS="$LIB_JARS,$jar" + H_JARS="$H_JARS:$jar" + done + export HADOOP_CLASSPATH="$H_JARS:$HADOOP_CLASSPATH" + + if [[ -z "$CLASSNAME" || -z "$TOOLJAR" ]]; then + echo "Usage: accumulo-util hadoop-jar path/to/myTool.jar my.tool.class.Name [-libjars my1.jar,my2.jar]" 1>&2 + exit 1 + fi + + #echo USERJARS=$USERJARS + #echo CLASSNAME=$CLASSNAME + #echo HADOOP_CLASSPATH=$HADOOP_CLASSPATH + #echo exec "$HADOOP_PREFIX/bin/hadoop" jar "$TOOLJAR" "$CLASSNAME" -libjars \"$LIB_JARS\" $ARGS + exec "$HADOOP_PREFIX/bin/hadoop" jar "$TOOLJAR" "$CLASSNAME" -libjars "$LIB_JARS" "$@" +} + +function main() { + SOURCE="${BASH_SOURCE[0]}" + while [ -h "${SOURCE}" ]; do + bin="$( cd -P "$( dirname "${SOURCE}" )" && pwd )" + SOURCE="$(readlink "${SOURCE}")" + [[ "${SOURCE}" != /* ]] && SOURCE="${bin}/${SOURCE}" + done + bin="$( cd -P "$( dirname "${SOURCE}" )" && pwd )" + basedir=$( cd -P "${bin}"/.. && pwd ) + conf="${basedir}/conf" + lib="${basedir}/lib" + + case "$1" in + create-config) + create_config "${@:2}" + ;; + build-native) + build_native "${@:2}" + ;; + hadoop-jar) + hadoop_jar "${@:2}" + ;; + gen-monitor-cert) + gen_monitor_cert + ;; + load-jars-hdfs) + load_jars_hdfs + ;; + *) + echo -e "'$1' is an invalid <command>\n" + print_usage 1>&2 + exit 1 + ;; + esac +} + +main "$@" http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/conf/examples/gc ---------------------------------------------------------------------- diff --git a/assemble/conf/examples/gc b/assemble/conf/examples/gc deleted file mode 100644 index 63fb8bb..0000000 --- a/assemble/conf/examples/gc +++ /dev/null @@ -1,16 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -localhost http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/conf/examples/generic_logger.properties ---------------------------------------------------------------------- diff --git a/assemble/conf/examples/generic_logger.properties b/assemble/conf/examples/generic_logger.properties index ff589d9..b4db23a 100644 --- a/assemble/conf/examples/generic_logger.properties +++ b/assemble/conf/examples/generic_logger.properties @@ -15,7 +15,7 @@ # Write out everything at the DEBUG level to the debug log log4j.appender.A2=org.apache.log4j.RollingFileAppender -log4j.appender.A2.File=${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}_${accumulo.service.instance}${org.apache.accumulo.core.ip.localhost.hostname}_fromprops.debug.log +log4j.appender.A2.File=${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}${accumulo.service.instance}_${org.apache.accumulo.core.ip.localhost.hostname}_fromprops.debug.log log4j.appender.A2.MaxFileSize=1000MB log4j.appender.A2.MaxBackupIndex=10 log4j.appender.A2.Threshold=DEBUG @@ -24,7 +24,7 @@ log4j.appender.A2.layout.ConversionPattern=%d{ISO8601} [%-8c{2}] %-5p: %m%n # Write out INFO and higher to the regular log log4j.appender.A3=org.apache.log4j.RollingFileAppender -log4j.appender.A3.File=${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}_${accumulo.service.instance}${org.apache.accumulo.core.ip.localhost.hostname}_fromprops.log +log4j.appender.A3.File=${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}${accumulo.service.instance}_${org.apache.accumulo.core.ip.localhost.hostname}_fromprops.log log4j.appender.A3.MaxFileSize=1000MB log4j.appender.A3.MaxBackupIndex=10 log4j.appender.A3.Threshold=INFO @@ -36,7 +36,7 @@ log4j.appender.A3.layout.ConversionPattern=%d{ISO8601} [%-8c{2}] %-5p: %m%n log4j.appender.ASYNC=org.apache.accumulo.core.util.AsyncSocketAppender log4j.appender.ASYNC.RemoteHost=${org.apache.accumulo.core.host.log} log4j.appender.ASYNC.Port=${org.apache.accumulo.core.host.log.port} -log4j.appender.ASYNC.Application=${org.apache.accumulo.core.application}:${org.apache.accumulo.core.ip.localhost.hostname} +log4j.appender.ASYNC.Application=${org.apache.accumulo.core.application}${accumulo.service.instance}:${org.apache.accumulo.core.ip.localhost.hostname} log4j.appender.ASYNC.Threshold=WARN # Log accumulo events to the debug, normal and remote logs. http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/conf/examples/generic_logger.xml ---------------------------------------------------------------------- diff --git a/assemble/conf/examples/generic_logger.xml b/assemble/conf/examples/generic_logger.xml index a71effe..23a1598 100644 --- a/assemble/conf/examples/generic_logger.xml +++ b/assemble/conf/examples/generic_logger.xml @@ -20,7 +20,7 @@ <!-- Write out everything at the DEBUG level to the debug log --> <appender name="A2" class="org.apache.log4j.RollingFileAppender"> - <param name="File" value="${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}_${accumulo.service.instance}${org.apache.accumulo.core.ip.localhost.hostname}.debug.log"/> + <param name="File" value="${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}${accumulo.service.instance}_${org.apache.accumulo.core.ip.localhost.hostname}.debug.log"/> <param name="MaxFileSize" value="1000MB"/> <param name="MaxBackupIndex" value="10"/> <param name="Threshold" value="DEBUG"/> @@ -31,7 +31,7 @@ <!-- Write out INFO and higher to the regular log --> <appender name="A3" class="org.apache.log4j.RollingFileAppender"> - <param name="File" value="${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}_${accumulo.service.instance}${org.apache.accumulo.core.ip.localhost.hostname}.log"/> + <param name="File" value="${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}${accumulo.service.instance}_${org.apache.accumulo.core.ip.localhost.hostname}.log"/> <param name="MaxFileSize" value="1000MB"/> <param name="MaxBackupIndex" value="10"/> <param name="Threshold" value="INFO"/> @@ -44,7 +44,7 @@ <appender name="N1" class="org.apache.log4j.net.SocketAppender"> <param name="remoteHost" value="${org.apache.accumulo.core.host.log}"/> <param name="port" value="${org.apache.accumulo.core.host.log.port}"/> - <param name="application" value="${org.apache.accumulo.core.application}:${org.apache.accumulo.core.ip.localhost.hostname}"/> + <param name="application" value="${org.apache.accumulo.core.application}${accumulo.service.instance}:${org.apache.accumulo.core.ip.localhost.hostname}"/> <param name="Threshold" value="WARN"/> </appender> http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/conf/examples/masters ---------------------------------------------------------------------- diff --git a/assemble/conf/examples/masters b/assemble/conf/examples/masters deleted file mode 100644 index 63fb8bb..0000000 --- a/assemble/conf/examples/masters +++ /dev/null @@ -1,16 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -localhost http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/conf/examples/monitor ---------------------------------------------------------------------- diff --git a/assemble/conf/examples/monitor b/assemble/conf/examples/monitor deleted file mode 100644 index 63fb8bb..0000000 --- a/assemble/conf/examples/monitor +++ /dev/null @@ -1,16 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -localhost http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/conf/examples/templates/accumulo-env.sh ---------------------------------------------------------------------- diff --git a/assemble/conf/examples/templates/accumulo-env.sh b/assemble/conf/examples/templates/accumulo-env.sh new file mode 100644 index 0000000..e6a360c --- /dev/null +++ b/assemble/conf/examples/templates/accumulo-env.sh @@ -0,0 +1,64 @@ +#! /usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +## Before accumulo-env.sh is loaded, these environment variables are set and can be used in this file: + +# ACCUMULO_CMD - Command that is being called such as tserver, master, etc. +# ACCUMULO_HOME - Root directory of Accumulo installation +# ACCUMULO_CONF_DIR - Directory containing Accumulo configuration + +############################ +# Variables that must be set +############################ + +## JVM options set for all processes. Extra options can be passed in by setting ACCUMULO_JAVA_OPTS to an array of options. +JAVA_OPTS=("${ACCUMULO_JAVA_OPTS[@]}" '-XX:+UseConcMarkSweepGC' '-XX:CMSInitiatingOccupancyFraction=75' '-XX:+CMSClassUnloadingEnabled' +'-XX:OnOutOfMemoryError=kill -9 %p' '-XX:-OmitStackTraceInFastThrow' '-Djava.net.preferIPv4Stack=true' +'-Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl') + +## JVM options set for individual applications +case "$ACCUMULO_CMD" in +master) JAVA_OPTS=("${JAVA_OPTS[@]}" ${masterHigh_masterLow}) ;; +gc) JAVA_OPTS=("${JAVA_OPTS[@]}" ${gcHigh_gcLow}) ;; +tserver) JAVA_OPTS=("${JAVA_OPTS[@]}" ${tServerHigh_tServerLow}) ;; +monitor) JAVA_OPTS=("${JAVA_OPTS[@]}" ${monitorHigh_monitorLow}) ;; +shell) JAVA_OPTS=("${JAVA_OPTS[@]}" ${shellHigh_shellLow}) ;; +*) JAVA_OPTS=("${JAVA_OPTS[@]}" ${otherHigh_otherLow}) ;; +esac +export JAVA_OPTS + +## Accumulo logs directory. Referenced by logger config. +export ACCUMULO_LOG_DIR="${ACCUMULO_LOG_DIR:-$ACCUMULO_HOME/logs}" +## Hadoop installation +export HADOOP_PREFIX="${HADOOP_PREFIX:-/path/to/hadoop}" +## Hadoop configuration +export HADOOP_CONF_DIR="${HADOOP_CONF_DIR:-${HADOOP_PREFIX}/etc/hadoop}" +## Zookeeper installation +export ZOOKEEPER_HOME="${ZOOKEEPER_HOME:-/path/to/zookeeper}" +## See HADOOP-7154 and ACCUMULO-847 +export MALLOC_ARENA_MAX=${MALLOC_ARENA_MAX:-1} + +############################################### +# Variables that are optional. Uncomment to set +############################################### + +## Specifies command that will be placed before calls to Java in accumulo script +# export ACCUMULO_JAVA_PREFIX="" +## Optionally look for hadoop and accumulo native libraries for your platform in additional +## directories. (Use DYLD_LIBRARY_PATH on Mac OS X.) May not be necessary for Hadoop 2.x or +## using an RPM that installs to the correct system library directory. +# export LD_LIBRARY_PATH=${HADOOP_PREFIX}/lib/native/${PLATFORM}:${LD_LIBRARY_PATH}
