Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/19247#discussion_r161352304
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
---
@@ -233,7 +233,7 @@ class FileStreamSource(
}
val files = allFiles.sortBy(_.getModificationTime)(fileSortOrder).map
{ status =>
- (status.getPath.toUri.toString, status.getModificationTime)
+ (status.getPath.toUri.getPath, status.getModificationTime)
--- End diff --
The correct fix is fixing this line:
https://github.com/xysun/spark/blob/4f5979a72ce9cb36a3327e79b8592b9e42bdf5af/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala#L168
It should be `files.map(new Path(new URI(_.path)).toString)`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
