[
https://issues.apache.org/jira/browse/TEZ-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4228:
------------------------------
Description:
While testing a 0.10.0 staging artifact, I found that there multiple issues of
TEZ-3860/TEZ-4223, which should be fixed before 0.10.0 release. The issues came
up while hive precommit tests:
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-1311/11/tests
The [failing hive
test|https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/mapjoin_addjar.q]
uses ADD JAR functionality:
{code}
ADD JAR
${system:maven.local.repository}/org/apache/hive/hive-it-test-serde/${system:hive.version}/hive-it-test-serde-${system:hive.version}.jar;
{code}
1. There are code paths, where Configuration object's classloader is used, so
if we add resources to TezClassLoader, we should point Configuration objects to
use them, otherwise, we'll face exception as this: [^syslog_bad.log] .
The changes in TezUtils solved the problem for container mode as
[^syslog_good.log]
2. There are codepaths, which use thread context classloader, so calling
TezClassLoader.setupTezClassLoader early in TezChild solves issues like this:
[^syslog_bad_context_classloader.log]
3. The Configuration issue is the same in Hive LLAP, which is solved in 1),
example failure reproduced locally is: [^hive_llap.log].
4. The context classloader problem is also present in LLAP daemons, which is
being solved by HIVE-24108
was:
While testing a 0.10.0 staging artifact, I found that there multiple issues of
TEZ-3860/TEZ-4223, which should be fixed before 0.10.0 release. The issues came
up while hive precommit tests:
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-1311/11/tests
The [failing hive
test|https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/mapjoin_addjar.q]
uses ADD JAR functionality:
{code}
ADD JAR
${system:maven.local.repository}/org/apache/hive/hive-it-test-serde/${system:hive.version}/hive-it-test-serde-${system:hive.version}.jar;
{code}
1. There are code paths, where Configuration object's classloader is used, so
if we add resources to TezClassLoader, we should point Configuration objects to
use them, otherwise, we'll face exception as this: [^syslog_bad.log] .
The changes in TezUtils solved the problem for container mode as
[^syslog_good.log]
2. There are codepaths, which use thread context classloader, so calling
TezClassLoader.setupTezClassLoader early in TezChild solves issues like this:
[^syslog_bad_context_classloader.log]
3. The configuration Configuration objects have their own classloader, which
should be overridden with TezClassLoader, otherwise they won't find newly added
classes, issue in [^hive_llap.log] . This one is being solved within TezUtils
methods.
> TezClassLoader should be used in TezChild and for Configuration objects
> -----------------------------------------------------------------------
>
> Key: TEZ-4228
> URL: https://issues.apache.org/jira/browse/TEZ-4228
> Project: Apache Tez
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Blocker
> Labels: 0.10_blocker
> Fix For: 0.10.0
>
> Attachments: TEZ-4228.01.patch, hive_llap.log, syslog_bad.log,
> syslog_bad_context_classloader.log, syslog_good.log
>
>
> While testing a 0.10.0 staging artifact, I found that there multiple issues
> of TEZ-3860/TEZ-4223, which should be fixed before 0.10.0 release. The issues
> came up while hive precommit tests:
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-1311/11/tests
> The [failing hive
> test|https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/mapjoin_addjar.q]
> uses ADD JAR functionality:
> {code}
> ADD JAR
> ${system:maven.local.repository}/org/apache/hive/hive-it-test-serde/${system:hive.version}/hive-it-test-serde-${system:hive.version}.jar;
> {code}
> 1. There are code paths, where Configuration object's classloader is used, so
> if we add resources to TezClassLoader, we should point Configuration objects
> to use them, otherwise, we'll face exception as this: [^syslog_bad.log] .
> The changes in TezUtils solved the problem for container mode as
> [^syslog_good.log]
> 2. There are codepaths, which use thread context classloader, so calling
> TezClassLoader.setupTezClassLoader early in TezChild solves issues like this:
> [^syslog_bad_context_classloader.log]
> 3. The Configuration issue is the same in Hive LLAP, which is solved in 1),
> example failure reproduced locally is: [^hive_llap.log].
> 4. The context classloader problem is also present in LLAP daemons, which is
> being solved by HIVE-24108
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
