alexanderbianchi opened a new issue, #3681:
URL: https://github.com/apache/iceberg-rust/issues/3681

   ### Describe the bug
   
   The Arrow reader opens data files even when their `FileScanTask` has 
`record_count = Some(0)`. A task with a zero-byte file then fails while loading 
Parquet metadata (`EOF: file size of 0 is less than footer`); an absent file 
also fails, although the task cannot contribute any rows.
   
   `InclusiveMetricsEvaluator` already excludes zero-record files when 
`include_empty_files` is false, but unfiltered scans do not necessarily invoke 
that evaluator. Callers can also supply tasks directly to `ArrowReader::read`, 
so a reader-side check is needed independently of planning.
   
   ### Reproduction
   
   On main at `27ce65cd3`:
   
   1. Construct a Parquet `FileScanTask` with `record_count = Some(0)`, 
`file_size_in_bytes = 0`, and no predicate.
   2. Point it at a zero-byte local file, or at a missing file.
   3. Pass the task to `ArrowReaderBuilder::new(...).build().read(...)` and 
collect the stream.
   
   The stream fails while opening the file or loading its footer instead of 
yielding no batches. The same issue affects scans through the DataFusion 
Iceberg data source.
   
   ### Expected behavior
   
   A known zero-record data file should yield no batches without opening the 
data file or its deletes. Positive or unknown record counts must retain normal 
reading and error handling; zero size alone is not a reason to discard a task.
   
   ### Proposed change
   
   Return an empty stream at the start of `FileScanTaskReader::process` when 
`task.record_count() == Some(0)`. This leaves scan planning unchanged and 
covers directly supplied tasks as well as planned scans.
   
   Regression coverage exercises serial/concurrent reads, zero and nonzero 
declared sizes for known-empty files, mixed empty/nonempty task streams, and 
positive/unknown record counts. The new skip/mixed-stream tests fail on the 
baseline and pass with the guard.
   
   ### AI disclosure
   
   AI assistance was used for investigation, implementation, test preparation, 
and this report. The behavior was checked with local baseline-versus-patch 
tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to