Lucian Timar created SPARK-34225:
------------------------------------
Summary: Jars or file paths which contain spaces are generating
FileNotFoundException exception
Key: SPARK-34225
URL: https://issues.apache.org/jira/browse/SPARK-34225
Project: Spark
Issue Type: Bug
Components: Spark Submit, Windows
Affects Versions: 3.0.1
Environment:
{code:java}
c:>spark-shell --files "c:\Program%20Files\...\myjar.jar"
or
c:>spark-shell --jars"c:\Program%20Files\...\myjar.jar"
or
c:>spark-shell --conf spark.jars="c:\Program%20Files\...\myjar.jar"
any combination produce the same exception
java.io.FileNotFoundException: Jar c:\Program%20Files\........ not found
at
org.apache.spark.SparkContext.addLocalJarFile$1(SparkContext.scala:1833)
at org.apache.spark.SparkContext.addJar(SparkContext.scala:1887)
at org.apache.spark.SparkContext.$anonfun$new$11(SparkContext.scala:490)
at
org.apache.spark.SparkContext.$anonfun$new$11$adapted(SparkContext.scala:490)
at
scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:490)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2574)
at
org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:934)
at scala.Option.getOrElse(Option.scala:189)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:928)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106)
{code}
I have noticed that this is caused by
{code:java}
def addJar(path: String): Unit = {
...
} else {
val uri = new Path(path).toUri
{code}
the path as the string is
"file:///C:/Program%20Files/Nokia/....jar"
this call generates the following uri
"file:///C:/Program*%25*20Files/Nokia/....jar"
which results in an invalid file name.
Maybe using
{code:java}
Utils.resolveURI(path){code}
resolves this issue.
Are there any workarounds to solve this issue ?
Reporter: Lucian Timar
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
