alexanderbianchi opened a new issue, #3681: URL: https://github.com/apache/iceberg-rust/issues/3681
### Describe the bug The Arrow reader opens data files even when their `FileScanTask` has `record_count = Some(0)`. A task with a zero-byte file then fails while loading Parquet metadata (`EOF: file size of 0 is less than footer`); an absent file also fails, although the task cannot contribute any rows. `InclusiveMetricsEvaluator` already excludes zero-record files when `include_empty_files` is false, but unfiltered scans do not necessarily invoke that evaluator. Callers can also supply tasks directly to `ArrowReader::read`, so a reader-side check is needed independently of planning. ### Reproduction On main at `27ce65cd3`: 1. Construct a Parquet `FileScanTask` with `record_count = Some(0)`, `file_size_in_bytes = 0`, and no predicate. 2. Point it at a zero-byte local file, or at a missing file. 3. Pass the task to `ArrowReaderBuilder::new(...).build().read(...)` and collect the stream. The stream fails while opening the file or loading its footer instead of yielding no batches. The same issue affects scans through the DataFusion Iceberg data source. ### Expected behavior A known zero-record data file should yield no batches without opening the data file or its deletes. Positive or unknown record counts must retain normal reading and error handling; zero size alone is not a reason to discard a task. ### Proposed change Return an empty stream at the start of `FileScanTaskReader::process` when `task.record_count() == Some(0)`. This leaves scan planning unchanged and covers directly supplied tasks as well as planned scans. Regression coverage exercises serial/concurrent reads, zero and nonzero declared sizes for known-empty files, mixed empty/nonempty task streams, and positive/unknown record counts. The new skip/mixed-stream tests fail on the baseline and pass with the guard. ### AI disclosure AI assistance was used for investigation, implementation, test preparation, and this report. The behavior was checked with local baseline-versus-patch tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
