Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21533#discussion_r195143756
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1517,9 +1517,12 @@ class SparkContext(config: SparkConf) extends
Logging {
* only supported for Hadoop-supported filesystems.
*/
def addFile(path: String, recursive: Boolean): Unit = {
- val uri = new Path(path).toUri
+ var uri = new Path(path).toUri
val schemeCorrectedPath = uri.getScheme match {
- case null | "local" => new File(path).getCanonicalFile.toURI.toString
+ case null | "local" =>
+ // SPARK-24195: Local is not a valid scheme for FileSystem, we
should only keep path here.
+ uri = new Path(uri.getPath).toUri
--- End diff --
I mean we `getPath` doesn't include scheme:
```
scala> new Path("local://a/b/c")
res0: org.apache.hadoop.fs.Path = local://a/b/c
scala> new Path("local://a/b/c").toUri
res1: java.net.URI = local://a/b/c
scala> new Path("local://a/b/c").toUri.getScheme
res2: String = local
scala> new Path("local://a/b/c").toUri.getPath
res3: String = /b/c
```
why should we do this again?
```
scala> new Path(new Path("local://a/b/c").toUri.getPath).toUri.getPath
res4: String = /b/c
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
