adriangb commented on code in PR #20592:
URL: https://github.com/apache/datafusion/pull/20592#discussion_r2874931671
##########
datafusion/datasource/src/file_stream.rs:
##########
@@ -179,6 +181,13 @@ impl FileStream {
batch
} else {
let batch = batch.slice(0, *remain);
+ // Count this file, the prefetched
next file
Review Comment:
This is if we hit a `LIMIT`. Maybe `files_processed` is a better term? I
want this to be useful to track the progress of a query, IMO the best metric we
have now is "how many files have we completed work on (for any reason) out of
all of the files we can possibly look at?".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
