[
https://issues.apache.org/jira/browse/TEZ-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17174188#comment-17174188
]
Harish JP commented on TEZ-4223:
--------------------------------
[~abstractdog], I do not think it will fix issues where you want to replace a
jar and run new DAG with new jar. Classes which were loaded previously will
continue to be loaded by the same classloader and will not be replaced. If new
classes are added via new jars this should be able to pick it up.
TEZ-3860 is already committed, this is a patch on top of that. I'm not sure I
understand the cleanup part.
> Adding new jars or resources after the first DAG runs does not work.
> --------------------------------------------------------------------
>
> Key: TEZ-4223
> URL: https://issues.apache.org/jira/browse/TEZ-4223
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Harish JP
> Assignee: Harish JP
> Priority: Major
> Attachments: TEZ-4223.01.patch
>
>
> If we executed DAG which needs additional jars after the first DAG is run, we
> get ClassNotFoundException.
>
>
> {noformat}
> 2020-08-03 13:57:14,776 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|:
> Added additional resources :
> [[file:/dataroot/ycloud/yarn/nm/usercache/hive/appcache/application_1596442677646_0012/container_1596442677646_0012_01_000001/commons-pool-1.5.4.jar,
>
> file:/dataroot/ycloud/yarn/nm/usercache/hive/appcache/application_1596442677646_0012/container_1596442677646_0012_01_000001/postgresql-42.2.8.jar,
>
> file:/dataroot/ycloud/yarn/nm/usercache/hive/appcache/application_1596442677646_0012/container_1596442677646_0012_01_000001/hive-jdbc-handler-3.1.3000.7.2.2.0-73.jar,
>
> file:/dataroot/ycloud/yarn/nm/usercache/hive/appcache/application_1596442677646_0012/container_1596442677646_0012_01_000001/mssql-jdbc-6.2.1.jre7.jar,
>
> file:/dataroot/ycloud/yarn/nm/usercache/hive/appcache/application_1596442677646_0012/container_1596442677646_0012_01_000001/commons-dbcp-1.4.jar]]
> to classpath
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
> class: org.apache.hive.storage.jdbc.JdbcInputFormat
> Serialization trace:
> inputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
> aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
> ...
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hive.storage.jdbc.JdbcInputFormat
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:398)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)
> ... 46 more{noformat}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
