alexanderbianchi opened a new pull request, #3739:
URL: https://github.com/apache/iceberg-rust/pull/3739

   ## Which issue does this PR close?
   
   Related to #3350. Large literal sets can also come from ordinary scan 
predicates.
   
   ## What changes are included in this PR?
   
   The Arrow reader runs an equality kernel over the column for every literal 
in an `IN` list. At 5,000 distinct keys and 8,192 rows that is about 41 million 
comparisons per batch.
   
   For sets of at least 32 integer or string literals, the reader builds a 
typed hash set and probes it once per row. Cloned predicates share the index 
across file tasks. Deserializing a predicate starts a fresh local index. The 
same lookup handles `NOT IN` and preserves the input validity bitmap. Short 
lists and incompatible physical types use the scalar comparison path.
   
   ## Are these changes tested?
   
   Added coverage for nullable and sliced arrays, all three Arrow string 
encodings, a promoted integer column, and sharing the index across predicate 
clones while preserving the serialized literal list.
   
   All 2,277 library tests passed (4 ignored). Formatting and Clippy passed, 
and regenerating the public API produced no diff.
   
   Local release measurements on an Apple M5 Pro with 5,000 keys and a 50% 
match rate.
   
   | Workload | Base | This PR |
   | --- | ---: | ---: |
   | Evaluate one 8,192-row Int64 batch | 4.28 ms | 15.8 µs |
   | Evaluate one 8,192-row UTF-8 batch | 30.5 ms | 46.1 µs |
   | Scan 262,144 Int64 rows in one Parquet file | 137 ms | 0.95 ms |
   | Scan the same rows across 64 Parquet files | 170 ms | 3.73 ms |
   
   Batch timings are medians of nine evaluations with a prepared predicate. 
File timings are medians of five warm local scans, with 8,192-row batches and 
one file reader at a time. Each scan constructs file tasks from a freshly bound 
predicate and verifies all 131,072 matching rows. For the 5,000-integer case, 
the first index construction took about 101 µs. Setting up another file 
predicate from a clone averaged 0.10 µs over 100 files, down from 252 µs on the 
base revision.
   
   ## AI Disclosure
   
   Codex implemented the change and ran the tests and benchmarks. The main 
review points are the 32-literal cutoff and the memory cost of retaining the 
typed index alongside the Iceberg literals, especially for long strings.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to