hcrosse opened a new pull request, #3740: URL: https://github.com/apache/iceberg-rust/pull/3740
## Which issue does this PR close? - Closes #3614. ## What changes are included in this PR? `ArrowReader` currently decodes dictionary encoded string columns into plain `Utf8` arrays. I added `dictionary_field_ids` to `ArrowReaderOptions` so callers can read selected top level string columns as `Dictionary(Int32, Utf8)`. - Adds `dictionary_field_ids: BTreeSet<i32>` (builder setter, getter and `serde` support) to `ArrowReaderOptions` and updates `crates/iceberg/public-api.txt`. - Validates requested IDs in `TableScan::to_arrow` and `FileScanTaskReader::process` (including zero record tasks), rejecting unknown, nested or non string fields and ignoring unprojected valid ones. - Rewrites requested projected fields to `Dictionary(Int32, Utf8)` in the schema hint, handles plain, fallback, missing, default and constant columns in `RecordBatchTransformer`, and supports row filters (`=`, `!=`, `<`, `<=`, `>`, `>=`, `IN`, `NOT IN`, `IS NULL`, `IS NOT NULL`, `starts_with`, `not_starts_with`) in `PredicateConverter`. ### Benchmark 30 synthetic Parquet files (30M rows, 295.5 MiB, 4 dictionary string columns and 1 plain fallback column, 8,192 row batches, concurrency 1). Medians over 5 interleaved runs per variant on 4 pinned Linux AMD EPYC vCPUs (`rustc 1.100.0`): | Workload | `main` (`Utf8`) | Option off | 4 dict cols (vs `main` / vs off) | | --- | ---: | ---: | ---: | | Read and hash strings (30M rows) | 4.26 s | 4.24 s | 3.03 s (-28.8% / -28.4%) | | Read only, count rows (30M rows) | 2.86 s | 2.71 s | 2.29 s (-19.8% / -15.4%) | | Filter on dict column + hash (468,750 rows) | 3.50 s | 3.32 s | 2.54 s (-27.3% / -23.5%) | Including the plain fallback column slowed read and hash to 3.86 s. Peak RSS stayed 20 to 22 MiB unfiltered and dropped from 37.3 to 32.7 MiB filtered. Row counts matched, and hashing runs matched checksums. ## Are these changes tested? Yes. I added unit and reader tests for dictionary, plain, fallback, missing, default and constant string columns, invalid field IDs, options serialization, and row filter predicates. `cargo test -p iceberg --lib`, `cargo clippy -p iceberg --all-targets -- -D warnings`, `cargo fmt --all -- --check`, and `cargo doc -p iceberg --no-deps --all-features` passed. ## AI Disclosure I used AI coding tools to help draft the code, tests and this description. I reviewed the changes, ran the checks above, and can explain the design. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
