Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19247#discussion_r161387193
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
---
@@ -233,7 +233,7 @@ class FileStreamSource(
}
val files = allFiles.sortBy(_.getModificationTime)(fileSortOrder).map
{ status =>
- (status.getPath.toUri.toString, status.getModificationTime)
+ (status.getPath.toUri.getPath, status.getModificationTime)
--- End diff --
Wait .. @zsxwing do you mean `getPath` (`org.apache.hadoop.fs.Path`) from
`org.apache.hadoop.fs.FileStatus` drops the scheme and credentials?
Here seems `Seq[org.apache.hadoop.fs.FileStatus]` and the code you pointed
out looks
`Array[org.apache.spark.sql.execution.streaming.FileStreamSource.FileEntry]`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
