hcrosse opened a new pull request, #3740:
URL: https://github.com/apache/iceberg-rust/pull/3740

   ## Which issue does this PR close?
   
   - Closes #3614.
   
   ## What changes are included in this PR?
   
   `ArrowReader` currently decodes dictionary encoded string columns into plain 
`Utf8` arrays. I added `dictionary_field_ids` to `ArrowReaderOptions` so 
callers can read selected top level string columns as `Dictionary(Int32, Utf8)`.
   
   - Adds `dictionary_field_ids: BTreeSet<i32>` (builder setter, getter and 
`serde` support) to `ArrowReaderOptions` and updates 
`crates/iceberg/public-api.txt`.
   - Validates requested IDs in `TableScan::to_arrow` and 
`FileScanTaskReader::process` (including zero record tasks), rejecting unknown, 
nested or non string fields and ignoring unprojected valid ones.
   - Rewrites requested projected fields to `Dictionary(Int32, Utf8)` in the 
schema hint, handles plain, fallback, missing, default and constant columns in 
`RecordBatchTransformer`, and supports row filters (`=`, `!=`, `<`, `<=`, `>`, 
`>=`, `IN`, `NOT IN`, `IS NULL`, `IS NOT NULL`, `starts_with`, 
`not_starts_with`) in `PredicateConverter`.
   
   ### Benchmark
   
   30 synthetic Parquet files (30M rows, 295.5 MiB, 4 dictionary string columns 
and 1 plain fallback column, 8,192 row batches, concurrency 1). Medians over 5 
interleaved runs per variant on 4 pinned Linux AMD EPYC vCPUs (`rustc 1.100.0`):
   
   | Workload | `main` (`Utf8`) | Option off | 4 dict cols (vs `main` / vs off) 
|
   | --- | ---: | ---: | ---: |
   | Read and hash strings (30M rows) | 4.26 s | 4.24 s | 3.03 s (-28.8% / 
-28.4%) |
   | Read only, count rows (30M rows) | 2.86 s | 2.71 s | 2.29 s (-19.8% / 
-15.4%) |
   | Filter on dict column + hash (468,750 rows) | 3.50 s | 3.32 s | 2.54 s 
(-27.3% / -23.5%) |
   
   Including the plain fallback column slowed read and hash to 3.86 s. Peak RSS 
stayed 20 to 22 MiB unfiltered and dropped from 37.3 to 32.7 MiB filtered. Row 
counts matched, and hashing runs matched checksums.
   
   ## Are these changes tested?
   
   Yes. I added unit and reader tests for dictionary, plain, fallback, missing, 
default and constant string columns, invalid field IDs, options serialization, 
and row filter predicates. `cargo test -p iceberg --lib`, `cargo clippy -p 
iceberg --all-targets -- -D warnings`, `cargo fmt --all -- --check`, and `cargo 
doc -p iceberg --no-deps --all-features` passed.
   
   ## AI Disclosure
   
   I used AI coding tools to help draft the code, tests and this description. I 
reviewed the changes, ran the checks above, and can explain the design.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to