[
https://issues.apache.org/jira/browse/HIVE-24108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220600#comment-17220600
]
László Bodor edited comment on HIVE-24108 at 10/26/20, 10:29 AM:
-----------------------------------------------------------------
according to an earlier slack topic, I found that ADD JAR is not supposed to be
supported in LLAP, so I'm about to change the first patch and use
TezClassloader only in FunctionLocalizer/AddToClassPathAction
the only test which failed with the new TezClassLoader is mapjoin_addjar.q,
which is an "ADD JAR" test on TestMiniLlapLocalCliDriver, and that's invalid in
my opinion, I'm moving it to TestMiniTezCliDriver
I think the reason why mapjoin_addjar.q is worked with
TestMiniLlapLocalCliDriver (before this TezClassloader stuff and tez 0.10) is
that it runs the llap daemon in the same JVM (and somehow also had the custom
TestSerDe class on the classpath), but that doesn't reflect a real-life
scenario (confirmed with TestMiniLlapCliDriver where the daemon runs in
separate JVM and it fails as expected)
cc: [~harishjp]: what do you think about [^HIVE-24108.02.patch] ? it doesn't
use the global TezClassloader...I'm testing it in the scope of HIVE-23930
was (Author: abstractdog):
according to an earlier slack topic, I found that ADD JAR is not supposed to be
supported in LLAP, so I'm about to change the first patch and use
TezClassloader only in FunctionLocalizer/AddToClassPathAction
the only test which failed with the new TezClassLoader is mapjoin_addjar.q,
which is an "ADD JAR" test on TestMiniLlapLocalCliDriver, and that's invalid in
my opinion, I'm moving it to TestMiniTezCliDriver
I think the reason why mapjoin_addjar.q is worked with
TestMiniLlapLocalCliDriver (before this TezClassloader stuff and tez 0.10) is
that it runs the llap daemon in the same JVM (and somehow also had the custom
TestSerDe class on the classpath), but that doesn't reflect a real-life
scenario (confirmed with TestMiniLlapCliDriver where the daemon runs in
separate JVM and it fails as expected)
> AddToClassPathAction should use TezClassLoader
> ----------------------------------------------
>
> Key: HIVE-24108
> URL: https://issues.apache.org/jira/browse/HIVE-24108
> Project: Hive
> Issue Type: Improvement
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Attachments: HIVE-24108.01.patch, HIVE-24108.02.patch,
> hive_log_llap.log
>
>
> TEZ-4228 fixes an issue from tez side, which is about to use TezClassLoader
> instead of the system classloader. However, there are some codepaths, e.g. in
> [^hive_log_llap.log] which shows that the system class loader is used. As
> thread context classloaders are inherited, the easier solution is to
> early-initialize TezClassLoader in LlapDaemon, and let all threads use that
> as context class loader, so this solution is more like TEZ-4223 for llap
> daemons.
> {code}
> 2020-09-02T00:18:20,242 ERROR [TezTR-93696_1_1_1_0_0] tez.TezProcessor:
> java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:351)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.serde2.TestSerDe
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:332)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:427)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:288)
> ... 16 more
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.serde2.TestSerDe
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:79)
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:100)
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:313)
> ... 18 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.serde2.TestSerDe
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:76)
> ... 21 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
