This is an automated email from the ASF dual-hosted git repository.
kabhwan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3cdbc176337 [SPARK-42668][SS][FOLLOW-UP] Do not add file scheme to the
Windows path
3cdbc176337 is described below
commit 3cdbc176337bdb0dbe11a120b0416b4f9014d8d9
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Tue May 9 15:51:16 2023 +0900
[SPARK-42668][SS][FOLLOW-UP] Do not add file scheme to the Windows path
### What changes were proposed in this pull request?
This PR is a followup of https://github.com/apache/spark/pull/40273 that
avoids adding `file://` into Windows path.
### Why are the changes needed?
To fix the broken AppVeyor build
```
== Failed
======================================================================
-- 1. Error ('test_streaming.R:55'): read.stream, write.stream,
awaitTermination
Error: Error in start : org.apache.hadoop.HadoopIllegalArgumentException:
Unsupported name: has scheme but relative path-part
...
```
See also
https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark/builds/46451196
### Does this PR introduce _any_ user-facing change?
No to end users because the change has not been released out yet.
### How was this patch tested?
AppVeyor build on windows.
Closes #41098 from HyukjinKwon/appveyor-testing.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Jungtaek Lim <[email protected]>
---
.../spark/sql/execution/streaming/ResolveWriteToStream.scala | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ResolveWriteToStream.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ResolveWriteToStream.scala
index 9cd1e60b564..35bb7db6a6e 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ResolveWriteToStream.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ResolveWriteToStream.scala
@@ -82,8 +82,15 @@ object ResolveWriteToStream extends Rule[LogicalPlan] with
SQLConfHelper {
s" circumstances, please set
${SQLConf.FORCE_DELETE_TEMP_CHECKPOINT_LOCATION.key} to" +
s" true. Important to know deleting temp checkpoint folder is best
effort.")
// SPARK-42676 - Write temp checkpoints for streaming queries to local
filesystem
- // even if default FS is set differently
- "file://" + tempDir
+ // even if default FS is set differently.
+ // This is a band-aid fix. Ideally we should convert `tempDir` to
URIs, but there
+ // are many legacy behaviors related to this.
+ if (Utils.isWindows) {
+ // For Windows local path, we can't treat that as a URI with file
scheme.
+ tempDir
+ } else {
+ "file://" + tempDir
+ }
} else {
throw QueryCompilationErrors.checkpointLocationNotSpecifiedError()
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
