berkaysynnada commented on code in PR #15432:
URL: https://github.com/apache/datafusion/pull/15432#discussion_r2016335211
##########
datafusion/core/src/datasource/listing/table.rs:
##########
@@ -1181,6 +1175,92 @@ impl ListingTable {
}
}
+/// Processes a stream of partitioned files and returns a `FileGroup`
containing the files.
+///
+/// This function collects files from the provided stream until either:
+/// 1. The stream is exhausted
+/// 2. The accumulated number of rows exceeds the provided `limit` (if
specified)
+///
+/// # Arguments
+/// * `files` - A stream of `Result<PartitionedFile>` items to process
+/// * `limit` - An optional row count limit. If provided, the function will
stop collecting files
+/// once the accumulated number of rows exceeds this limit
+/// * `collect_stats` - Whether to collect and accumulate statistics from the
files
+///
+/// # Returns
+/// A `Result` containing a `FileGroup` with the collected files
+/// and a boolean indicating whether the statistics are inexact.
+///
+/// # Note
+/// The function will continue processing files if statistics are not
available or if the
+/// limit is not provided. If `collect_stats` is false, statistics won't be
accumulated
+/// but files will still be collected.
+async fn get_files_with_limit(
+ files: impl Stream<Item = Result<PartitionedFile>>,
+ limit: Option<usize>,
+ collect_stats: bool,
+) -> Result<(FileGroup, bool)> {
+ let mut file_group = FileGroup::default();
+ // Fusing the stream allows us to call next safely even once it is
finished.
+ let mut all_files = Box::pin(files.fuse());
+ let mut num_rows = Precision::<usize>::Absent;
+ while let Some(first_file) = all_files.next().await {
Review Comment:
I suggest writing this logic with a state machine, which polls files at one
point only. I believe FSM's would increase the code quality and
understandability
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
