-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26591/
-----------------------------------------------------------
(Updated Oct. 11, 2014, 3:23 a.m.)
Review request for Ambari, Andrew Onischuk, Dmytro Sen, Hitesh Shah, and Sumit
Mohanty.
Bugs: AMBARI-7739
https://issues.apache.org/jira/browse/AMBARI-7739
Repository: ambari
Description
-------
Create an Ambari 1.7.0 cluster with HDP 2.2.1, and install HDFS, Tez, MR, Hive,
Zookeeper.
After switching the hive.execution.engine from "mr" to "tez", attempting to run
the hive shell generates an error.
Logging initialized using configuration in
file:/etc/hive/conf.server/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:457)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already
shutdown
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:598)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:119)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:454)
... 8 more
***********************************************************
# This needs two +1s in order to be committed to branch-1.7.0
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
62e3cd6
ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml
93ad95d
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-env.xml
PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-site.xml
c1e8fa6
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml
dfedb6b
Diff: https://reviews.apache.org/r/26591/diff/
Testing
-------
Deployed an HDP 2.2.1 cluster and switched the execution engine to tez.
Then changed the following properties to coincide with the changed files.
HDFS => hadoop-env =
{
...
if [[ -d "/usr/hdp/current/tez-client" && -d "/etc/tez/conf/" ]]; then
# When using versioned RPMs, the tez-client will be a symlink to the current
folder of tez in HDP.
export
HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/tez-client/:/usr/hdp/current/tez-client/lib/:/etc/tez/conf/
fi
# Setting path to hdfs command line
export HADOOP_LIBEXEC_DIR={{hadoop_libexec_dir}}
# Mostly required for hadoop 2.0
export
JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
...
}
Hive => hive-env =
{
...
if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}
elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog
fi
...
}
Tez => tez.lib.uris = hdfs:///apps/tez/tez.tar.gz
After restarting the services, ran the following commands to simulate the
behavior of the scripts during the installation of Hadoop.
export TEZ_CONF_DIR=/etc/tez/conf
export TEZ_JARS=/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*
export HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS:$HADOOP_CLASSPATH
Then ran
su - hive
hive
And was able to get the hive shell to start.
To test this on a brand new cluster install, would like for this to be
committed first.
Also ran unit tests,
----------------------------------------------------------------------
Total run:641
Total errors:0
Total failures:0
OK
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:21.678s
[INFO] Finished at: Fri Oct 10 19:39:55 PDT 2014
[INFO] Final Memory: 63M/1540M
[INFO] ------------------------------------------------------------------------
Thanks,
Alejandro Fernandez
