HyukjinKwon commented on a change in pull request #24446: [SPARK-27552][SQL]The
configuration `hive.exec.stagingdir` is invalid on Windows OS
URL: https://github.com/apache/spark/pull/24446#discussion_r278399474
##########
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/SaveAsHiveFile.scala
##########
@@ -219,7 +219,7 @@ private[hive] trait SaveAsHiveFile extends
DataWritingCommand {
val fs: FileSystem = inputPath.getFileSystem(hadoopConf)
var stagingPathName: String =
if (inputPathName.indexOf(stagingDir) == -1) {
- new Path(inputPathName, stagingDir).toString
+ new Path(inputPathName, stagingDir).toUri.getPath
Review comment:
This drops theme:
```scala
scala> new Path("hdfs://0.0.0:8020/a/b/c").toString
res1: String = hdfs://0.0.0:8020/a/b/c
scala> new Path("hdfs://0.0.0:8020/a/b/c").toUri.getPath
res2: String = /a/b/c
```
So, if the theme is different from the default, it looks it's going to be
ignored.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
