stym06 commented on issue #3747: URL: https://github.com/apache/hudi/issues/3747#issuecomment-940130218
1. Apache Hive 2. Attaching the file (Have altered the provided sh file to include all Hive libraries as it was not working with Hive 3.1.2) 3. I will check this and revert. ###hudi-cli_run_sync.sh ``` #!/usr/bin/env bash # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. function error_exit { echo "$1" >&2 ## Send message to stderr. Exclude >&2 if you don't want it that way. exit "${2:-1}" ## Return a code specified by $2 or 1 by default. } if [ -z "${HADOOP_HOME}" ]; then error_exit "Please make sure the environment variable HADOOP_HOME is setup" fi if [ -z "${HIVE_HOME}" ]; then error_exit "Please make sure the environment variable HIVE_HOME is setup" fi DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" #Ensure we pick the right jar even for hive11 builds HUDI_HIVE_UBER_JAR=`ls -c $DIR/hudi-hive-sync-*.jar | grep -v source | head -1` if [ -z "$HADOOP_CONF_DIR" ]; then echo "setting hadoop conf dir" HADOOP_CONF_DIR="${HADOOP_HOME}/etc/hadoop" fi ## Include only specific packages from HIVE_HOME/lib to avoid version mismatches HIVE_EXEC=`ls ${HIVE_HOME}/lib/hive-exec-*.jar | tr '\n' ':'` HIVE_SERVICE=`ls ${HIVE_HOME}/lib/hive-service-*.jar | grep -v rpc | tr '\n' ':'` HIVE_METASTORE=`ls ${HIVE_HOME}/lib/hive-metastore-*.jar | tr '\n' ':'` HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | tr '\n' ':'` if [ -z "${HIVE_JDBC}" ]; then HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | grep -v handler | tr '\n' ':'` fi HIVE_JACKSON=`ls ${HIVE_HOME}/lib/jackson-*.jar | tr '\n' ':'` HIVE_JARS=$HIVE_METASTORE:$HIVE_SERVICE:$HIVE_EXEC:$HIVE_JDBC:$HIVE_JACKSON HIVE_CALCITE=`ls ${HIVE_HOME}/lib/calcite-*.jar | tr '\n' ':'` if [ -n "$HIVE_CALCITE" ]; then HIVE_JARS=$HIVE_JARS:$HIVE_CALCITE fi HIVE_LIBFB303=`ls ${HIVE_HOME}/lib/libfb303-*.jar | tr '\n' ':'` if [ -n "$HIVE_LIBFB303" ]; then HIVE_JARS=$HIVE_JARS:$HIVE_LIBFB303 fi HIVE_DATANUCLEUS=`ls ${HIVE_HOME}/lib/datanucleus-*.jar | tr '\n' ':'` HIVE_HIKARI=`ls ${HIVE_HOME}/lib/HikariCP-2.6.1.jar | tr '\n' ':'` HIVE_JARS=$HIVE_JARS:$HIVE_DATANUCLEUS:$HIVE_HIKARI HUDI_MR_BUNDLE=`ls ${HIVE_HOME}/lib/hudi-hadoop-*.jar | tr '\n' ':'` HIVE_JARS=$HIVE_JARS:$HUDI_MR_BUNDLE HADOOP_HIVE_JARS=${HIVE_JARS}:${HADOOP_HOME}/share/hadoop/common/*:${HADOOP_HOME}/share/hadoop/mapreduce/*:${HADOOP_HOME}/share/hadoop/hdfs/*:${HADOOP_HOME}/share/hadoop/common/lib/*:${HADOOP_HOME}/share/hadoop/hdfs/lib/* echo "Running Command : java -cp ${HADOOP_HIVE_JARS}:${HADOOP_CONF_DIR}:$HUDI_HIVE_UBER_JAR org.apache.hudi.hive.HiveSyncTool $@" java -cp $HUDI_HIVE_UBER_JAR:${HADOOP_HIVE_JARS}:${HADOOP_CONF_DIR}:${HIVE_HOME}/lib/*: org.apache.hudi.hive.HiveSyncTool "$@" ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
