yihua opened a new issue, #843: URL: https://github.com/apache/hudi-rs/issues/843
## Context parquet 59 deprecates `ParquetObjectReader` (arrow-rs is extracting its `object_store` code so the two can version independently, apache/arrow-rs#10308). The deprecation note asks consumers to implement `AsyncFileReader` directly over `object_store` and to use `SpawnedReader` when I/O should run on a dedicated runtime. #698 keeps the deprecated reader and allows the lint at each use site in `crates/core/src/file_group/base_file/parquet.rs` (the import, the `CountedBuilder` alias, and the constructor call in the builder) rather than for the whole file, with a comment pointing here. Nothing is lost today; this issue tracks the replacement so it is not forgotten when a parquet release removes the type. ## What the replacement needs to carry The reader hudi-rs builds today is `ParquetObjectReader` wrapped in `CountingReader`, which feeds the read-volume counters from #708, with footer metadata run through `normalize_parquet_metadata` for the legacy two-level list case. An in-house `AsyncFileReader` folds those into one type instead of a wrapper around a wrapper: - `get_bytes` / `get_byte_ranges` straight onto `object_store`'s `get_range` / `get_ranges`, counting bytes and requests as they go. Range coalescing becomes hudi-rs's decision rather than parquet's. - `get_metadata` loading the footer with the known file size (every call site has it), applying the metadata normalization once, and keeping the `ArrowReaderMetadata` cache the builder relies on. - Optionally `SpawnedReader` so object-store I/O for the parquet path runs on the storage runtime rather than the caller's, which matters for the Python and DataFusion FFI paths where the caller's runtime is not ours. ## Why it is also an optimization `ParquetObjectReader` issues one `get_range` per requested range with its own coalescing policy and no knowledge of the row-group pruning done one layer up. Owning the reader lets the base-file reader batch the column-chunk ranges it already knows it needs into fewer, larger object-store requests, and lets the counters record requests as well as bytes. ## Not in scope Changing what the counters mean, or the metadata normalization itself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
