曹勇 created KYLIN-5085:
-------------------------
Summary: Something wrong with Hive CLI or Beeline, please execute
Hive CLI or Beeline CLI in terminal to find the root cause.
Key: KYLIN-5085
URL: https://issues.apache.org/jira/browse/KYLIN-5085
Project: Kylin
Issue Type: Bug
Components: Client - CLI, Environment
Affects Versions: v4.0.0
Environment: hive-2.3.9
kylin-4.0.0
spark-3.1.2-bin-hadoop3.2
zookeeper-3.7.0
Reporter: 曹勇
Attachments: image-2021-09-07-20-40-15-503.png
when I reduce hadoop version to hadoop-3.2.2 , run ./kylin.sh start
the command export :
{code:java}
[hadoop@node3 logs]$ ../bin/kylin.sh start
Retrieving hadoop conf dir...
...................................................[PASS]
KYLIN_HOME is set to /home/hadoop/kylin-4.0.0
Checking hive
...................................................[PASS]
Checking hadoop shell
...................................................[PASS]
Checking hdfs working dir
...................................................[PASS]Checking environment
finished successfully. To check again, run 'bin/check-env.sh' manually.
Retrieving hive dependency...
Something wrong with Hive CLI or Beeline, please execute Hive CLI or Beeline
CLI in terminal to find the root cause.
{code}
Then I set `set -exv` in the $KYLIN_HOME/bin/find-hive-dependency.sh
!image-2021-09-07-20-40-15-503.png!
And the script print:
{code:java}
[hadoop@node3 bin]$ ./kylin.sh start
Retrieving hadoop conf dir...
...................................................[PASS]
KYLIN_HOME is set to /home/hadoop/kylin-4.0.0
Checking hive
...................................................[PASS]
Checking hadoop shell
...................................................[PASS]
Checking hdfs working dir
...................................................[PASS]Checking environment
finished successfully. To check again, run 'bin/check-env.sh' manually.
source ${KYLIN_HOME:-"$(cd -P -- "$(dirname -- "$0")" && pwd
-P)/../"}/bin/header.sh
++ source /home/hadoop/kylin-4.0.0/bin/header.sh
#!/bin/bash#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
## source meverbose=${verbose:-""}
+++ verbose=while getopts ":v" opt; do
case $opt in
v)
echo "Turn on verbose mode." >&2
verbose=true
;;
\?)
echo "Invalid option: -$OPTARG" >&2
;;
esac
done
+++ getopts :v optif [[ "$dir" == "" ]]
then
dir=$(cd -P -- "$(dirname -- "$0")" && pwd -P) # set KYLIN_HOME
with consideration for multiple instances that are on the same node
KYLIN_HOME=${KYLIN_HOME:-"${dir}/../"}
export KYLIN_HOME=`cd "$KYLIN_HOME"; pwd`
dir="$KYLIN_HOME/bin" function quit {
echo "$@"
exit 1
} function verbose {
if [[ -n "$verbose" ]]; then
echo "$@"
fi
} function setColor() {
echo -e "\033[$1m$2\033[0m"
} # set JAVA
if [[ "${JAVA}" == "" ]]; then
if [[ -z "$JAVA_HOME" ]]; then
JAVA_VERSION=`java -version 2>&1 | awk -F\" '/version/ {print $2}'`
if [[ $JAVA_VERSION ]] && [[ "$JAVA_VERSION" > "1.8" ]]; then
JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java))))
else
quit "Java 1.8 or above is required."
fi
[[ -z "$JAVA_HOME" ]] && quit "Please set JAVA_HOME"
export JAVA_HOME
fi
export JAVA=$JAVA_HOME/bin/java
[[ -e "${JAVA}" ]] || quit "${JAVA} does not exist. Please set
JAVA_HOME correctly."
verbose "java is ${JAVA}"
fi
fi
+++ [[ /home/hadoop/kylin-4.0.0/bin == '' ]]## ${dir} assigned to
$KYLIN_HOME/bin in header.sh
source ${dir}/load-hive-conf.sh
++ source /home/hadoop/kylin-4.0.0/bin/load-hive-conf.sh
#!/bin/bash#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#source ${KYLIN_HOME:-"$(cd -P -- "$(dirname -- "$0")" && pwd
-P)/../"}/bin/header.sh
+++ source /home/hadoop/kylin-4.0.0/bin/header.sh
#!/bin/bash#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
## source meverbose=${verbose:-""}
++++ verbose=while getopts ":v" opt; do
case $opt in
v)
echo "Turn on verbose mode." >&2
verbose=true
;;
\?)
echo "Invalid option: -$OPTARG" >&2
;;
esac
done
++++ getopts :v optif [[ "$dir" == "" ]]
then
dir=$(cd -P -- "$(dirname -- "$0")" && pwd -P) # set KYLIN_HOME
with consideration for multiple instances that are on the same node
KYLIN_HOME=${KYLIN_HOME:-"${dir}/../"}
export KYLIN_HOME=`cd "$KYLIN_HOME"; pwd`
dir="$KYLIN_HOME/bin" function quit {
echo "$@"
exit 1
} function verbose {
if [[ -n "$verbose" ]]; then
echo "$@"
fi
} function setColor() {
echo -e "\033[$1m$2\033[0m"
} # set JAVA
if [[ "${JAVA}" == "" ]]; then
if [[ -z "$JAVA_HOME" ]]; then
JAVA_VERSION=`java -version 2>&1 | awk -F\" '/version/ {print $2}'`
if [[ $JAVA_VERSION ]] && [[ "$JAVA_VERSION" > "1.8" ]]; then
JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java))))
else
quit "Java 1.8 or above is required."
fi
[[ -z "$JAVA_HOME" ]] && quit "Please set JAVA_HOME"
export JAVA_HOME
fi
export JAVA=$JAVA_HOME/bin/java
[[ -e "${JAVA}" ]] || quit "${JAVA} does not exist. Please set
JAVA_HOME correctly."
verbose "java is ${JAVA}"
fi
fi
++++ [[ /home/hadoop/kylin-4.0.0/bin == '' ]]# source me
hive_conf_dir="${KYLIN_HOME}/conf/kylin_hive_conf.xml"
+++ hive_conf_dir=/home/hadoop/kylin-4.0.0/conf/kylin_hive_conf.xmlnames=(`sed
-n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n 's|<name>\(.*\)</name>|\1|p' |
sed 's/ \+//g'`)
+++ names=(`sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n
's|<name>\(.*\)</name>|\1|p' | sed 's/ \+//g'`)
sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n 's|<name>\(.*\)</name>|\1|p'
| sed 's/ \+//g'
++++ sed -n 's|<name>\(.*\)</name>|\1|p'
++++ sed -n '/<!--/,/-->/!p' /home/hadoop/kylin-4.0.0/conf/kylin_hive_conf.xml
++++ sed 's/ \+//g'
values=(`sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n
's|<value>\(.*\)</value>|\1|p' | sed 's/ \+//g'`)
+++ values=(`sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n
's|<value>\(.*\)</value>|\1|p' | sed 's/ \+//g'`)
sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n
's|<value>\(.*\)</value>|\1|p' | sed 's/ \+//g'
++++ sed -n 's|<value>\(.*\)</value>|\1|p'
++++ sed -n '/<!--/,/-->/!p' /home/hadoop/kylin-4.0.0/conf/kylin_hive_conf.xml
++++ sed 's/ \+//g'len_names=${#names[@]}
+++ len_names=9
len_values=${#values[@]}
+++ len_values=9[[ $len_names == $len_values ]] || quit "Current hive conf
file: ${hive_conf_dir} has inconsistent key value pairs."
+++ [[ 9 == 9 ]]for((i=0;i<$len_names;i++))
do
hive_conf_properties=${hive_conf_properties}" --hiveconf
${names[$i]}=${values[$i]} "
done
+++ (( i=0 ))
+++ (( i<9 ))
+++ hive_conf_properties=' --hiveconf dfs.replication=2 '
+++ (( i++ ))
+++ (( i<9 ))
+++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
hive.exec.compress.output=true '
+++ (( i++ ))
+++ (( i<9 ))
+++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true '
+++ (( i++ ))
+++ (( i<9 ))
+++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true
--hiveconf hive.auto.convert.join.noconditionaltask=true '
+++ (( i++ ))
+++ (( i<9 ))
+++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true
--hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf
hive.auto.convert.join.noconditionaltask.size=100000000 '
+++ (( i++ ))
+++ (( i<9 ))
+++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true
--hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf
hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf
mapreduce.job.split.metainfo.maxsize=-1 '
+++ (( i++ ))
+++ (( i<9 ))
+++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true
--hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf
hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf
mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.stats.autogather=true '
+++ (( i++ ))
+++ (( i<9 ))
+++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true
--hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf
hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf
mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.stats.autogather=true
--hiveconf hive.merge.mapfiles=false '
+++ (( i++ ))
+++ (( i<9 ))
+++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf
hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true
--hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf
hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf
mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.stats.autogather=true
--hiveconf hive.merge.mapfiles=false --hiveconf hive.merge.mapredfiles=false '
+++ (( i++ ))
+++ (( i<9 ))echo Retrieving hive dependency...
++ echo Retrieving hive dependency...
Retrieving hive dependency...client_mode=`bash
${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client`
bash ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client
+++ bash /home/hadoop/kylin-4.0.0/bin/get-properties.sh kylin.source.hive.client
++ client_mode=
hive_env=
++ hive_env=if [ "${client_mode}" == "beeline" ]
then
beeline_shell=`$KYLIN_HOME/bin/get-properties.sh
kylin.source.hive.beeline-shell`
beeline_params=`bash ${KYLIN_HOME}/bin/get-properties.sh
kylin.source.hive.beeline-params`
hive_env=`${beeline_shell} ${hive_conf_properties} ${beeline_params}
--outputformat=dsv -e "set;" 2>&1 | grep --text 'env:CLASSPATH' `
else
source ${dir}/check-hive-usability.sh
hive_env=`hive ${hive_conf_properties} -e set 2>&1 | grep 'env:CLASSPATH'`
fi
++ '[' '' == beeline ']'
++ source /home/hadoop/kylin-4.0.0/bin/check-hive-usability.sh
#!/bin/bash#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#source ${KYLIN_HOME:-"$(cd -P -- "$(dirname -- "$0")" && pwd
-P)/../"}/bin/header.sh
+++ source /home/hadoop/kylin-4.0.0/bin/header.sh
#!/bin/bash#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
## source meverbose=${verbose:-""}
++++ verbose=while getopts ":v" opt; do
case $opt in
v)
echo "Turn on verbose mode." >&2
verbose=true
;;
\?)
echo "Invalid option: -$OPTARG" >&2
;;
esac
done
++++ getopts :v optif [[ "$dir" == "" ]]
then
dir=$(cd -P -- "$(dirname -- "$0")" && pwd -P) # set KYLIN_HOME
with consideration for multiple instances that are on the same node
KYLIN_HOME=${KYLIN_HOME:-"${dir}/../"}
export KYLIN_HOME=`cd "$KYLIN_HOME"; pwd`
dir="$KYLIN_HOME/bin" function quit {
echo "$@"
exit 1
} function verbose {
if [[ -n "$verbose" ]]; then
echo "$@"
fi
} function setColor() {
echo -e "\033[$1m$2\033[0m"
} # set JAVA
if [[ "${JAVA}" == "" ]]; then
if [[ -z "$JAVA_HOME" ]]; then
JAVA_VERSION=`java -version 2>&1 | awk -F\" '/version/ {print $2}'`
if [[ $JAVA_VERSION ]] && [[ "$JAVA_VERSION" > "1.8" ]]; then
JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java))))
else
quit "Java 1.8 or above is required."
fi
[[ -z "$JAVA_HOME" ]] && quit "Please set JAVA_HOME"
export JAVA_HOME
fi
export JAVA=$JAVA_HOME/bin/java
[[ -e "${JAVA}" ]] || quit "${JAVA} does not exist. Please set
JAVA_HOME correctly."
verbose "java is ${JAVA}"
fi
fi
++++ [[ /home/hadoop/kylin-4.0.0/bin == '' ]]hive -e "select 1" > /dev/null
2>&1 &
pid=$!
+++ pid=25366((timeLeft = 60))
+++ (( timeLeft = 60 ))while ((timeLeft > 0)); do
sleep 5
ps -p $pid > /dev/null || break
((timeLeft -= 5))
done
+++ hive -e 'select 1'
+++ (( timeLeft > 0 ))
+++ sleep 5
+++ ps -p 25366
+++ breakif ps -p $pid > /dev/null
then
kill $pid
sleep 5 # still alive, use kill -9
if ps -p $pid > /dev/null
then
kill -9 $pid
fi quit "ERROR: Check hive's usability failed, please check the status
of your cluster"
fi
+++ ps -p 25366
hive ${hive_conf_properties} -e set 2>&1 | grep 'env:CLASSPATH'
+++ hive --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true
--hiveconf hive.auto.convert.join=true --hiveconf
hive.auto.convert.join.noconditionaltask=true --hiveconf
hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf
mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.stats.autogather=true
--hiveconf hive.merge.mapfiles=false --hiveconf hive.merge.mapredfiles=false -e
set
+++ grep env:CLASSPATH
++ hive_env=
{code}
I can't really tell what the problem is .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
