unikdahal opened a new pull request, #3345: URL: https://github.com/apache/iceberg-rust/pull/3345
## Which issue does this PR close? - Closes #3344. - Part of #3343. ## What changes are included in this PR? Execution engines that build `FileScanTask`s themselves do not always carry a delete file's size, and a delete file with `file_size_in_bytes == 0` cannot currently be opened. - Tasks planned from manifest entries carry the recorded size and are unaffected. Externally constructed tasks may now use `0` when the size is unavailable (documented on `FileScanTaskDeleteFile::file_size_in_bytes`). - `BasicDeleteFileLoader` resolves a zero size with one metadata request when the Parquet delete file is first loaded. `CachingDeleteFileLoader` already loads each delete file once per reader, so this happens once per delete file even when several data files reference it. - A failed metadata request is reported as `Failed to stat delete file '…'`; `BasicDeleteFileLoader` and `CachingDeleteFileLoader::load_deletes` keep the storage error's kind and retryability. - A size below the 8-byte Parquet footer minimum, whether resolved from storage or supplied by the task, fails the read with `DataInvalid` rather than dropping the file's deletes. - Known sizes take the existing path with no metadata request. Deletion vectors do not use this size and are unchanged. No public type or method signatures change; `FileScanTaskDeleteFile::file_size_in_bytes` additionally accepts `0` for externally constructed tasks where the size is unavailable. ### Not in this PR - `ArrowReader` with a data-file concurrency of 1 wraps task errors as `Unexpected`; that pre-existing wrapping is left to a separate change. - If loading a delete file fails (metadata or open), its delete-cache entry stays `Loading`: later positional-delete waiters for the same file are never notified, and the spawned equality-delete task panics on the dropped channel. This is pre-existing for open failures with known sizes; a failed metadata request is one more way to reach it. Fixing it needs failure propagation to waiters (they must not proceed as if there were no deletes), so it is left to a separate change. ## Are these changes tested? Yes: - A metadata-counting test storage shows known sizes issue no metadata request and unknown sizes issue exactly one per delete file, including when another task references an already loaded delete file; both produce identical deleted positions. - Unknown-size equality deletes loaded through `CachingDeleteFileLoader` for two data files issue one metadata request and produce a bound predicate equal to the known-size one; parsing an unknown-size equality delete file yields an equal predicate. - An encrypted positional delete file is read with both its known size and an unknown size. - A missing delete file with an unknown size reports `Failed to stat delete file` with exactly the storage error's kind and retryability, from the basic loader and through `CachingDeleteFileLoader::load_deletes`; a retryable storage failure stays retryable. - A zero-byte object is rejected for both an unknown and a too-small supplied size. ## AI Disclosure This change was developed with assistance from Claude Code and reviewed by me. Tests and CI results come from this branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
