[
https://issues.apache.org/jira/browse/APEXCORE-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263558#comment-15263558
]
ASF GitHub Bot commented on APEXCORE-304:
-----------------------------------------
Github user chinmaykolhatkar commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/311#discussion_r61535209
--- Diff:
engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java ---
@@ -2386,7 +2381,7 @@ public static void write(DAG dag, OutputStream os)
throws IOException
public static LogicalPlan read(InputStream is) throws IOException,
ClassNotFoundException
{
- return (LogicalPlan)new ObjectInputStream(is).readObject();
+ return (LogicalPlan)new
ClassLoaderObjectInputStream(Thread.currentThread().getContextClassLoader(),
is).readObject();
--- End diff --
This is required for Local Mode case.
Lets say a jar path is set to LIBRARY_JARS and this jar contains a POJO
that needs to be set to TUPLE_CLASS attr. Now, one can do that in populateDAG
method, that would cause a problem in LocalMode as follows:
1. populateDAG sets TUPLE_CLASS attr with new POJO class - Works.
2. Local mode serializes the LogicalPlan - Works.
3. Local mode deserialized the LogicalPlan - Fails.
3rd step fails becuase The ObjectInputStream deserializer uses System
ClassLoader which does that have that POJO. ClassLoaderObjectInputStream is an
overridden version of ObjectInputStream which allows to set the custom
classloader.
By this time of callstack, Thread.currentThread().getClassLoader() will
have a classloader which is super of all.
As previously, there was no functionality of LIBRARY_JARS, I guess this was
not tested and so did not surface as an issue.
For cluster mode, there is no difference. i.e. There is only 1 classloader
i.e. System.Classloader and that has new jars as well.
i.e. For cluster mode, Thread.currentThread().getClassLoader() = System
classloader
i.e. ClassLoaderObjectInputStream will behave as if ObjectInputStream.
> Ability to add jars to classpath in populateDAG
> -----------------------------------------------
>
> Key: APEXCORE-304
> URL: https://issues.apache.org/jira/browse/APEXCORE-304
> Project: Apache Apex Core
> Issue Type: Improvement
> Reporter: Chinmay Kolhatkar
> Assignee: Chinmay Kolhatkar
>
> This will have following functionality:
> 1) In populateDAG one would be allowed to add given local jar path to
> classpath of the application.
> 2) Optionally delete the given jar file after copying to HDFS is done.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
