[
https://issues.apache.org/jira/browse/APEXCORE-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119088#comment-15119088
]
ASF GitHub Bot commented on APEXCORE-304:
-----------------------------------------
Github user chinmaykolhatkar commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/208#discussion_r50972253
--- Diff: engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java
---
@@ -318,6 +333,21 @@ public StramLocalCluster(LogicalPlan dag) throws
IOException, ClassNotFoundExcep
}
}
+ private void addExtraJarsToClasspath(Set<String> jars)
+ throws InvocationTargetException, IllegalAccessException,
MalformedURLException, NoSuchMethodException
+ {
+ List<URL> jarUrls = new LinkedList<>();
+ for (String jarPath : jars) {
+ File file = new File(jarPath);
+ URL url = file.toURI().toURL();
+ jarUrls.add(url);
+ }
+
+ ClassLoader prevCl = Thread.currentThread().getContextClassLoader();
+ ClassLoader urlCl = URLClassLoader.newInstance(jarUrls.toArray(new
URL[jarUrls.size()]), prevCl);
+ Thread.currentThread().setContextClassLoader(urlCl);
--- End diff --
You're right. There was a little modification required.
So here is how I've addressed things for various run modes:
**1. For cluster mode:**
The jar gets copied to HDFS lib location and the hdfs jar path is added to
the classpath.
This is how it was done originally in this pull request and is unchanged
since.
**2. For local mode via dtcli (-local):**
Idea is to add the extra jars to same context loader as the one which loads
operator classes. Because of this, the thread that gets spawned for
operator/container is set with a context loader having both extra jars and the
operator code. Because of this the extra jars are available by all ways. I'm
not sure to write a unit test for -local mode of dtcli. But I wrote a sample
test application and ran in local mode with dtcli. The operator setup tried to
load the classes by following APIs and it ran successfully:
*Class.forName(<class>)*
*Class.forName(<class>, init,
Thread.currentThread().getContextClassLoader())*
*Thread.currentThread().getContextClassLoader().loadClass(classToLoad);*
**3. For local mode via LocalMode.runApp API (i.e. ApplicationTest):**
I've added the unit test in which operator loads the class via following
APIs successfully:
*Class.forName(<class>, init,
Thread.currentThread().getContextClassLoader())*
*Thread.currentThread().getContextClassLoader().loadClass(classToLoad)*
Class.forName will not work here as Class.forName find classloader of
caller class and then tries to load the class from caller's loader or its
parent. As the operators are already loaded by System classloader, there is no
clean way to modify the class loader to add another path to it.
If its URL class loader, then via reflection can be done. But not sure if
its clean again.
Hence, operator will have to use Thread context OR StramUtils.classForName
method to load the class.
> Ability to add jars to classpath in populateDAG
> -----------------------------------------------
>
> Key: APEXCORE-304
> URL: https://issues.apache.org/jira/browse/APEXCORE-304
> Project: Apache Apex Core
> Issue Type: Improvement
> Reporter: Chinmay Kolhatkar
> Assignee: Chinmay Kolhatkar
>
> This will have following functionality:
> 1) In populateDAG one would be allowed to add given local jar path to
> classpath of the application.
> 2) Optionally delete the given jar file after copying to HDFS is done.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
