hhhizzz opened a new issue, #11147: URL: https://github.com/apache/arrow-rs/issues/11147
## Problem A physical Parquet Page inspector needs the **encoded dictionary entry ID consumed by each non-null value position** of one `RLE_DICTIONARY` Data Page. This is different from the occurrence ordinal and from returning an Arrow `DictionaryArray`: decoded values can repeat while their occurrence ordinals advance, for example ordinals `0, 1, 2` consuming dictionary IDs `2, 2, 0`. `ColumnReader::read_records` returns repetition/definition levels and already materialized values, but does not expose the consumed dictionary IDs. `DictIndexDecoder` would decode the values section, but `parquet::arrow::decoder` is private (including on current `main`). The lower-level `encodings::rle` route is available only via the `experimental` feature, which carries no stability guarantee. This was verified against parquet 58.4.0 with downstream imports failing E0603. We are looking for a supported seam rather than making all decoder internals public. Related #9010 asks to inspect dictionary **contents** in the async reader; this request is for per-occurrence **IDs in a selected Data Page**. ## Possible interface Would maintainers prefer either: 1. an opt-in Page/column decode callback that pairs each level position with the optional dictionary ID and decoded value; or 2. a public, validated value-section dictionary-index decoder with an explicit count contract, documented for composition with levels from `ColumnReader`? The second route would let an inspector split the already decompressed selected Page into V1/V2 level and value sections, then align returned IDs only to positions whose definition level equals the schema maximum. A Page can fall back to `PLAIN`; those positions must have no dictionary ID. The API should permit bounded, opt-in use without adding per-value work to ordinary reads. ## Correctness boundary - V1 and V2 level-section framing differ; a caller must isolate the values section before decoding indices. - The number of IDs must equal the number of non-null physical values, including for nested repetition/definition levels and V2 `num_nulls`. - Reject empty/truncated streams, invalid bit widths, and negative or out-of-range dictionary IDs. Current `main` has the empty-section/bit-width fix from #10725; an exposed seam should retain it. - No dictionary ID can be inferred by searching a decoded value in the dictionary, nor from its occurrence ordinal. - A `PLAIN` fallback Page must report no dictionary ID even when an earlier Page in the Column Chunk used a dictionary. Could you advise which public interface fits parquet-rs ownership and stability expectations? We can contribute a focused implementation after the seam is agreed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
