Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/14960#discussion_r77568657
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1900,7 +1900,20 @@ private[spark] object Utils extends Logging {
*/
def resolveURI(path: String): URI = {
try {
- val uri = new URI(path)
+ val osSafePath = if (Path.isWindowsAbsolutePath(path, false)) {
+ // Make sure C:/ part becomes /C/.
+ val windowsUri = new URI(path)
+ val driveLetter = windowsUri.getScheme
+ s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
+ } else if (Path.isWindowsAbsolutePath(path, true)) {
+ // Make sure /C:/ part becomes /C/.
+ val windowsUri = new URI(path.substring(1))
+ val driveLetter = windowsUri.getScheme
+ s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
--- End diff --
I see. Actually, it seems it has to be
`C:\Users\appveyor\AppData\Local\Temp\1\RtmpkzTq6t\glm78c146111b1.tmp` for path
without the `slash`.
It seems the problem is, in many classes such as `RWrappers`, there are the
usages of `new Path(path, "rMetadata").toString`.
It seems this converts the path, for example,
from
```
C:\Users\appveyor\AppData\Local\Temp\1\RtmpkzTq6t\glm78c146111b1.tmp
```
to
```
C:/Users/appveyor/AppData/Local/Temp/1/RtmpkzTq6t/glm78c146111b1.tmp
```
So, for example, it goes through `textFile` -> `hadoopFile` ->
`Utils.resolveURI`. Here the given path is
`C:/Users/appveyor/AppData/Local/Temp/1/RtmpkzTq6t/glm78c146111b1.tmp` but it
seems `Utils.resolveURI` does not handle this case.
So, this returns this as it is.
```scala
...
val uri = new URI(path)
if (uri.getScheme() != null) { // here it just passes becuase `C:/.../...`
has the scheme `C`.
return uri
}
...
```
and it ends up with an exception as below:
```scala
java.io.IOException: No FileSystem for scheme: C
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
