Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/853#issuecomment-44010184
I tested this and this breaks spark-shell in standalone mode with this
error (when the given relative path was `app.jar`)
```
java.io.FileNotFoundException: /root/tdas/file:/root/tdas/app.jar (No such
file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at com.google.common.io.Files$FileByteSource.openStream(Files.java:124)
at com.google.common.io.Files$FileByteSource.openStream(Files.java:114)
at com.google.common.io.ByteSource.copyTo(ByteSource.java:202)
at com.google.common.io.Files.copy(Files.java:436)
at org.apache.spark.HttpFileServer.addFileToDir(HttpFileServer.scala:62)
at org.apache.spark.HttpFileServer.addJar(HttpFileServer.scala:57)
at org.apache.spark.SparkContext.addJar(SparkContext.scala:944)
at
org.apache.spark.SparkContext$$anonfun$11.apply(SparkContext.scala:265)
at
org.apache.spark.SparkContext$$anonfun$11.apply(SparkContext.scala:265)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:265)
at
org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:957)
```
This is because when the Spark Repl adds jars to the spark context, it does
not consider the path in `spark.jars` property as a URI. Rather it considers
the URI to be a file system path, and prepends the current working dir once
again, leading to the above crazy path.
The solution was to make SparkILoop.getAddedJars returns non-URI paths, as
it was designed to. This should be minimally invasive change.
```
--- a/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
+++ b/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
@@ -997,7 +997,8 @@ object SparkILoop {
val propJars = sys.props.get("spark.jars").flatMap { p =>
if (p == "") None else Some(p)
}
- propJars.orElse(envJars).map(_.split(",")).getOrElse(Array.empty)
+ val jars =
propJars.orElse(envJars).map(_.split(",")).getOrElse(Array.empty)
+ jars.map(Utils.resolveURI(_).getPath)
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
