Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19643#discussion_r151307924
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1838,12 +1852,21 @@ class SparkContext(config: SparkConf) extends
Logging {
case _ => path
}
}
+
if (key != null) {
val timestamp = System.currentTimeMillis
if (addedJars.putIfAbsent(key, timestamp).isEmpty) {
logInfo(s"Added JAR $path at $key with timestamp $timestamp")
postEnvironmentUpdate()
}
+
+ if (addToCurrentClassLoader) {
+ Utils.getContextOrSparkClassLoader match {
+ case cl: MutableURLClassLoader =>
cl.addURL(Utils.resolveURI(path).toURL)
--- End diff --
I'm not sure does it support remote jars on HTTPS or Hadoop
FileSystemsï¼In the executor side, we handle this explicitly by downloading
jars to local and add to classpath, but here looks like we don't have such
logic. I'm not sure how this `URLClassLoader` communicate with Hadoop or Https
without certificates.
The `addJar` is just adding jars to fileserver, so that executor could
fetch them from driver and add to classpath. It will not affect driver's
classpath. If we support adding jars to current driver's classloader, then how
do we leverage this newly added jars?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
