Rich-T-kid opened a new pull request, #12161:
URL: https://github.com/apache/arrow-rs/pull/12161

   # Which issue does this PR close?
   
   - Closes #12160.
   
   # Rationale for this change
   
   A Parquet `BYTE_ARRAY` column without a UTF8 annotation can be requested as 
`Dictionary(Int32, Utf8)`. The dictionary reader currently decodes its 
dictionary values as `Binary`, so callers must cast whole batches after 
decoding. Pushed predicates can consequently rebuild the dictionary for every 
batch before evaluating a string predicate.
   
   # What changes are included in this PR?
   
   The byte-array dictionary reader decodes into the physical string or binary 
value type, then converts each dictionary page values once to the requested 
compatible type while preserving the encoded keys. The converted values are 
reused across batches backed by the same dictionary page. Plain-encoded 
fallback pages retain the existing dictionary-building path.
   
   # Are these changes tested?
   
   Coverage includes binary dictionary pages requested as `Utf8`, `LargeUtf8`, 
and `Utf8View`, reuse across batches, nulls, invalid UTF-8 rejection, and 
plain-encoded fallback pages.
   
   A draft DataFusion dependency PR will provide the end-to-end 
pushed-predicate benchmark evidence before this PR is marked ready.
   
   # Are there any significant user-facing changes?
   
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to