geoffreyclaude opened a new pull request, #24283: URL: https://github.com/apache/datafusion/pull/24283
## Which issue does this PR close? - Part of #19241. - Stacked on [#23014](https://github.com/apache/datafusion/pull/23014). - Next in stack: [#24102](https://github.com/apache/datafusion/pull/24102). ## Rationale for this change #23014 adds direct comparisons for short primitive `IN` lists. The cutoff and larger-list fallback were selected in `strategy.rs`, separately from the primitive filter implementations. That makes the same policy awkward to reuse for other fixed-width representations. This PR moves primitive selection next to the filters it selects. The shared selector counts non-null constants, uses the existing direct-comparison cutoffs, and then chooses the existing bitmap, hash-set, or float filter for that primitive type. Types without one of those filters return to the general path. The refactor also replaces the separate `Int32`, `Int64`, `UInt32`, and `UInt64` hash-set filters with one `PrimitiveHashSetFilter<T>` and reuses the shared SQL result builder. Existing primitive routing and null behavior remain unchanged. `Decimal128` is the one new strategy. Lists with up to four non-null values still use direct comparisons. Larger lists now hash their stored `i128` values instead of using the general Arrow filter. Values are not rescaled, and existing precision/scale compatibility rules are unchanged. ## What changes are included in this PR? - Moves primitive filter selection from `strategy.rs` into `primitive_filter.rs`. - Adds a shared `StaticFilterRef` and generic `PrimitiveHashSetFilter<T>`. - Reuses shared result construction for primitive hash-set filters. - Routes `Decimal128` lists above the four-value cutoff to the primitive hash set. - Adds four `Decimal128` benchmark cases at list sizes 5 and 64. ## Are these changes tested? Tests cover selection around the direct-comparison cutoff, all-null lists, and the larger `Decimal128` path with hits, misses, a null in the list, `IN`, `NOT IN`, dictionary inputs, and compatible precision/scale metadata. ## Are there any user-facing changes? No. SQL results and public APIs are unchanged. ## Benchmark snapshot The benchmark source adds the first hash-set list size (5) and a larger list (64), each with 0% and 50% matches. Filter construction is outside the timed loop. A previous [automated run](https://github.com/apache/datafusion/pull/24102#issuecomment-5256542575) measured these code states before this work was split from #24102. The split does not change the `Decimal128` implementation on either side. Compared baselines: [#23014](https://github.com/apache/datafusion/pull/23014) -> this PR | Benchmark | Before | After | Change | |---|---:|---:|---:| | `primitive/decimal128/large_list/list=5/match=0%` | 18.1 us | 15.2 us | -16.0% (1.19x faster) | | `primitive/decimal128/large_list/list=5/match=50%` | 38.6 us | 19.2 us | -50.3% (2.01x faster) | | `primitive/decimal128/large_list/list=64/match=0%` | 17.6 us | 15.2 us | -13.6% (1.16x faster) | | `primitive/decimal128/large_list/list=64/match=50%` | 37.1 us | 23.5 us | -36.7% (1.58x faster) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
