[
https://issues.apache.org/jira/browse/TEZ-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4228:
------------------------------
Description:
While testing a 0.10.0 staging artifact, I found that there multiple issues of
TEZ-3860/TEZ-4223, which should be fixed before 0.10.0 release. The issues came
up while hive precommit tests:
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-1311/11/tests
The [failing hive
test|https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/mapjoin_addjar.q]
uses ADD JAR functionality:
{code}
ADD JAR
${system:maven.local.repository}/org/apache/hive/hive-it-test-serde/${system:hive.version}/hive-it-test-serde-${system:hive.version}.jar;
{code}
1. There are code paths, where Configuration object's classloader is used, so
if we add resources to TezClassLoader, we should point Configuration objects to
use them, otherwise, we'll face exception as this: [^syslog_bad.log] .
The changes in TezUtils solved the problem for container mode as
[^syslog_good.log]
2. There are codepaths, which use thread context classloader, so calling
TezClassLoader.setupTezClassLoader early in TezChild solves issues like this:
[^syslog_bad_context_classloader.log]
3. The configuration Configuration objects have their own classloader, which
should be overridden with TezClassLoader, otherwise they won't find newly added
classes, issue in [^hive_llap.log] . This one is being solved within TezUtils
methods.
was:
While testing a 0.10.0 staging artifact, I found that there are 2 issues of
TEZ-3860/TEZ-4223, which should be fixed before 0.10.0 release. The issues came
up while hive precommit tests:
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-1311/11/tests
The [failing hive
test|https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/mapjoin_addjar.q]
uses ADD JAR functionality:
{code}
ADD JAR
${system:maven.local.repository}/org/apache/hive/hive-it-test-serde/${system:hive.version}/hive-it-test-serde-${system:hive.version}.jar;
{code}
1. TezClassLoader static initialization is not deterministic, or at least it
seems like LogicalIOProcessorRuntimeTask.run can run before it. See "tez
classloader" log messages in [^syslog_bad.log] . The changes in TezChild
solved the problem as [^syslog_good.log]
2. Configuration objects have their own classloader, which should be overridden
with TezClassLoader, otherwise they won't find newly added classes, issue in
[^hive_llap.log] . This one is being solved within TezUtils methods.
> TezClassLoader should be used in TezChild and for Configuration objects
> -----------------------------------------------------------------------
>
> Key: TEZ-4228
> URL: https://issues.apache.org/jira/browse/TEZ-4228
> Project: Apache Tez
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Blocker
> Labels: 0.10_blocker
> Fix For: 0.10.0
>
> Attachments: TEZ-4228.01.patch, hive_llap.log, syslog_bad.log,
> syslog_bad_context_classloader.log, syslog_good.log
>
>
> While testing a 0.10.0 staging artifact, I found that there multiple issues
> of TEZ-3860/TEZ-4223, which should be fixed before 0.10.0 release. The issues
> came up while hive precommit tests:
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-1311/11/tests
> The [failing hive
> test|https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/mapjoin_addjar.q]
> uses ADD JAR functionality:
> {code}
> ADD JAR
> ${system:maven.local.repository}/org/apache/hive/hive-it-test-serde/${system:hive.version}/hive-it-test-serde-${system:hive.version}.jar;
> {code}
> 1. There are code paths, where Configuration object's classloader is used, so
> if we add resources to TezClassLoader, we should point Configuration objects
> to use them, otherwise, we'll face exception as this: [^syslog_bad.log] .
> The changes in TezUtils solved the problem for container mode as
> [^syslog_good.log]
> 2. There are codepaths, which use thread context classloader, so calling
> TezClassLoader.setupTezClassLoader early in TezChild solves issues like this:
> [^syslog_bad_context_classloader.log]
> 3. The configuration Configuration objects have their own classloader, which
> should be overridden with TezClassLoader, otherwise they won't find newly
> added classes, issue in [^hive_llap.log] . This one is being solved within
> TezUtils methods.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
