[
https://issues.apache.org/jira/browse/AMBARI-21607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114416#comment-16114416
]
Di Li edited comment on AMBARI-21607 at 8/4/17 2:10 PM:
--------------------------------------------------------
Hello Eric and Attila,
_hbase_lib_ was removed in BI 4.2.5. We should not use it post migration. The
following section in hive-env from IOP 4.2 should be removed during EU.
{quote}# Set HIVE_AUX_JARS_PATH
export HIVE_AUX_JARS_PATH={{hbase_lib}}/hbase-client.jar,\
{{hbase_lib}}/hbase-common.jar,\
{{hbase_lib}}/hbase-hadoop2-compat.jar,\
{{hbase_lib}}/hbase-prefix-tree.jar,\
{{hbase_lib}}/hbase-protocol.jar,\
{{hbase_lib}}/hbase-server.jar,\
{{hbase_lib}}/htrace-core-3.1.0-incubating.jar,\
${HIVE_AUX_JARS_PATH}
{quote}
Right around the time of the IOP/HDP annoucement, Tim, myself, and a dev from
our Hive team were looking at a fix and our thinking was the following
{quote}<service name="HIVE">
<component name="HIVE_SERVER">
<changes>
<definition xsi:type="configure"
id="biginsights_4_2_5_update_hive_env">
<type>hive-env</type>
<replace key="content" find="{{hbase_lib}}/hbase-client.jar,"
replace-with=""/>
<replace key="content" find="{{hbase_lib}}/hbase-common.jar,\"
replace-with="\"/>
<replace key="content"
find="{{hbase_lib}}/hbase-hadoop2-compat.jar,\" replace-with="\"/>
<replace key="content" find="{{hbase_lib}}/hbase-prefix-tree.jar,\"
replace-with="\"/>
<replace key="content" find="{{hbase_lib}}/hbase-protocol.jar,\"
replace-with="\"/>
<replace key="content" find="{{hbase_lib}}/hbase-server.jar,\"
replace-with="\"/>
<replace key="content"
find="{{hbase_lib}}/htrace-core-3.1.0-incubating.jar,\" replace-with="\"/>
</definition>{quote}
So the end result is like the following
{quote}export HIVE_AUX_JARS_PATH=\
\
\
\
\
\
\
${HIVE_AUX_JARS_PATH}{quote}
*IMPORTANT*: Please note that for a BigSQL cluster to be migrated, hive-env has
the following at the end of the file that *must be retained during EU*. -- the
section only exists on a BigSQL cluster.
{quote}# Allow Hive to read Big SQL HBase tables
if [ -d "/usr/ibmpacks/current/bigsql/bigsql/lib/java" ]; then
export HIVE_AUX_JARS_PATH=\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/biga-io.jar,\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/biga-hbase.jar,\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/commoncatalog.jar,\
/usr/ibmpacks/current/bigsql/hive/lib/hive-hbase-handler.jar,\
${HIVE_AUX_JARS_PATH}
fi{quote}
P.S
I have an IOP4.2.5/HDP 2.6.2 migrated cluster and I am able to open the Hive
shell after I switch (as the migration was done before Attila's EU config fix)
the engine to MR.
[hive@loch4 ~]$ hive
2017-08-04 06:40:32,485 WARN [main] conf.HiveConf: HiveConf of name
hive.optimize.mapjoin.mapreduce does not exist
Logging initialized using configuration in
file:/etc/hive/2.6.2.0-129/0/hive-log4j.properties
hive>
was (Author: dili):
Hello Eric and Attila,
_hbase_lib_ was removed in BI 4.2.5. We should not use it post migration. The
following section in hive-env from IOP 4.2 should be removed during EU.
{quote}# Set HIVE_AUX_JARS_PATH
export HIVE_AUX_JARS_PATH={{hbase_lib}}/hbase-client.jar,\
{{hbase_lib}}/hbase-common.jar,\
{{hbase_lib}}/hbase-hadoop2-compat.jar,\
{{hbase_lib}}/hbase-prefix-tree.jar,\
{{hbase_lib}}/hbase-protocol.jar,\
{{hbase_lib}}/hbase-server.jar,\
{{hbase_lib}}/htrace-core-3.1.0-incubating.jar,\
${HIVE_AUX_JARS_PATH}
{quote}
Right around the time of the IOP/HDP annoucement, Tim, myself, and a dev from
our Hive team were looking at a fix and our thinking was the following
{{<service name="HIVE">
<component name="HIVE_SERVER">
<changes>
<definition xsi:type="configure"
id="biginsights_4_2_5_update_hive_env">
<type>hive-env</type>
<replace key="content" find="{{hbase_lib}}/hbase-client.jar,"
replace-with=""/>
<replace key="content" find="{{hbase_lib}}/hbase-common.jar,\"
replace-with="\"/>
<replace key="content"
find="{{hbase_lib}}/hbase-hadoop2-compat.jar,\" replace-with="\"/>
<replace key="content" find="{{hbase_lib}}/hbase-prefix-tree.jar,\"
replace-with="\"/>
<replace key="content" find="{{hbase_lib}}/hbase-protocol.jar,\"
replace-with="\"/>
<replace key="content" find="{{hbase_lib}}/hbase-server.jar,\"
replace-with="\"/>
<replace key="content"
find="{{hbase_lib}}/htrace-core-3.1.0-incubating.jar,\" replace-with="\"/>
</definition>}}
So the end result is like the following
{{export HIVE_AUX_JARS_PATH=\
\
\
\
\
\
\
${HIVE_AUX_JARS_PATH}}}
*IMPORTANT*: Please note that for a BigSQL cluster to be migrated, hive-env has
the following at the end of the file that *must be retained during EU*. -- the
section only exists on a BigSQL cluster.
{{# Allow Hive to read Big SQL HBase tables
if [ -d "/usr/ibmpacks/current/bigsql/bigsql/lib/java" ]; then
export HIVE_AUX_JARS_PATH=\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/biga-io.jar,\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/biga-hbase.jar,\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/commoncatalog.jar,\
/usr/ibmpacks/current/bigsql/hive/lib/hive-hbase-handler.jar,\
${HIVE_AUX_JARS_PATH}
fi}}
P.S
I have an IOP4.2.5/HDP 2.6.2 migrated cluster and I am able to open the Hive
shell after I switch (as the migration was done before Attila's EU config fix)
the engine to MR.
[hive@loch4 ~]$ hive
2017-08-04 06:40:32,485 WARN [main] conf.HiveConf: HiveConf of name
hive.optimize.mapjoin.mapreduce does not exist
Logging initialized using configuration in
file:/etc/hive/2.6.2.0-129/0/hive-log4j.properties
hive>
> Hive can not write to session directory after upgrade
> -----------------------------------------------------
>
> Key: AMBARI-21607
> URL: https://issues.apache.org/jira/browse/AMBARI-21607
> Project: Ambari
> Issue Type: Bug
> Components: stacks
> Affects Versions: 2.5.2
> Reporter: Eric Yang
> Assignee: Doroszlai, Attila
> Fix For: 2.5.2
>
>
> After upgrade, start hive shell. This error message is shown:
> {code}
> Exception in thread "main" java.lang.RuntimeException: java.io.IOException:
> Previous writer likely failed to write
> hdfs://eyang-1.openstacklocal:8020/tmp/hive/hive/_tez_session_dir/6a2cab98-2e71-4223-bc9d-6301b3c6a5d5/hbase-client.jar.
> Failing because I am unlikely to write too.
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:560)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> Caused by: java.io.IOException: Previous writer likely failed to write
> hdfs://eyang-1.openstacklocal:8020/tmp/hive/hive/_tez_session_dir/6a2cab98-2e71-4223-bc9d-6301b3c6a5d5/hbase-client.jar.
> Failing because I am unlikely to write too.
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1002)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:882)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:809)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:258)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:157)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:557)
> ... 8 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
