alexanderbianchi opened a new pull request, #3739: URL: https://github.com/apache/iceberg-rust/pull/3739
## Which issue does this PR close? Related to #3350. Large literal sets can also come from ordinary scan predicates. ## What changes are included in this PR? The Arrow reader runs an equality kernel over the column for every literal in an `IN` list. At 5,000 distinct keys and 8,192 rows that is about 41 million comparisons per batch. For sets of at least 32 integer or string literals, the reader builds a typed hash set and probes it once per row. Cloned predicates share the index across file tasks. Deserializing a predicate starts a fresh local index. The same lookup handles `NOT IN` and preserves the input validity bitmap. Short lists and incompatible physical types use the scalar comparison path. ## Are these changes tested? Added coverage for nullable and sliced arrays, all three Arrow string encodings, a promoted integer column, and sharing the index across predicate clones while preserving the serialized literal list. All 2,277 library tests passed (4 ignored). Formatting and Clippy passed, and regenerating the public API produced no diff. Local release measurements on an Apple M5 Pro with 5,000 keys and a 50% match rate. | Workload | Base | This PR | | --- | ---: | ---: | | Evaluate one 8,192-row Int64 batch | 4.28 ms | 15.8 µs | | Evaluate one 8,192-row UTF-8 batch | 30.5 ms | 46.1 µs | | Scan 262,144 Int64 rows in one Parquet file | 137 ms | 0.95 ms | | Scan the same rows across 64 Parquet files | 170 ms | 3.73 ms | Batch timings are medians of nine evaluations with a prepared predicate. File timings are medians of five warm local scans, with 8,192-row batches and one file reader at a time. Each scan constructs file tasks from a freshly bound predicate and verifies all 131,072 matching rows. For the 5,000-integer case, the first index construction took about 101 µs. Setting up another file predicate from a clone averaged 0.10 µs over 100 files, down from 252 µs on the base revision. ## AI Disclosure Codex implemented the change and ran the tests and benchmarks. The main review points are the 32-literal cutoff and the memory cost of retaining the typed index alongside the Iceberg literals, especially for long strings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
