viirya commented on code in PR #44636:
URL: https://github.com/apache/spark/pull/44636#discussion_r1470767776
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala:
##########
@@ -166,6 +197,25 @@ class FileStreamSource(
// implies "sourceOptions.latestFirst = true" which we want to refresh
the list per batch
(newFiles.take(files.maxFiles()), null)
+ case files: ReadMaxBytes if !sourceOptions.latestFirst =>
+ // we can cache and reuse remaining fetched list of files in further
batches
+ val (FilesSplit(bFiles, _), FilesSplit(usFiles, rSize)) =
+ takeFilesUntilMax(newFiles, files.maxBytes())
+ if (rSize < (files.maxBytes() * DISCARD_UNSEEN_FILES_RATIO).toLong) {
Review Comment:
Hmm, if `BigInt` is used to avoid overflow, does it make sense to cast it to
`Long` to compare here? `rSize` might be larger than long range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
