This is an automated email from the ASF dual-hosted git repository. kabhwan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 3cdbc176337 [SPARK-42668][SS][FOLLOW-UP] Do not add file scheme to the Windows path 3cdbc176337 is described below commit 3cdbc176337bdb0dbe11a120b0416b4f9014d8d9 Author: Hyukjin Kwon <gurwls...@apache.org> AuthorDate: Tue May 9 15:51:16 2023 +0900 [SPARK-42668][SS][FOLLOW-UP] Do not add file scheme to the Windows path ### What changes were proposed in this pull request? This PR is a followup of https://github.com/apache/spark/pull/40273 that avoids adding `file://` into Windows path. ### Why are the changes needed? To fix the broken AppVeyor build ``` == Failed ====================================================================== -- 1. Error ('test_streaming.R:55'): read.stream, write.stream, awaitTermination Error: Error in start : org.apache.hadoop.HadoopIllegalArgumentException: Unsupported name: has scheme but relative path-part ... ``` See also https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark/builds/46451196 ### Does this PR introduce _any_ user-facing change? No to end users because the change has not been released out yet. ### How was this patch tested? AppVeyor build on windows. Closes #41098 from HyukjinKwon/appveyor-testing. Authored-by: Hyukjin Kwon <gurwls...@apache.org> Signed-off-by: Jungtaek Lim <kabhwan.opensou...@gmail.com> --- .../spark/sql/execution/streaming/ResolveWriteToStream.scala | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ResolveWriteToStream.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ResolveWriteToStream.scala index 9cd1e60b564..35bb7db6a6e 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ResolveWriteToStream.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ResolveWriteToStream.scala @@ -82,8 +82,15 @@ object ResolveWriteToStream extends Rule[LogicalPlan] with SQLConfHelper { s" circumstances, please set ${SQLConf.FORCE_DELETE_TEMP_CHECKPOINT_LOCATION.key} to" + s" true. Important to know deleting temp checkpoint folder is best effort.") // SPARK-42676 - Write temp checkpoints for streaming queries to local filesystem - // even if default FS is set differently - "file://" + tempDir + // even if default FS is set differently. + // This is a band-aid fix. Ideally we should convert `tempDir` to URIs, but there + // are many legacy behaviors related to this. + if (Utils.isWindows) { + // For Windows local path, we can't treat that as a URI with file scheme. + tempDir + } else { + "file://" + tempDir + } } else { throw QueryCompilationErrors.checkpointLocationNotSpecifiedError() } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org