lucasfang opened a new issue, #262: URL: https://github.com/apache/paimon-cpp/issues/262
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Motivation ## Motivation Evaluating an `IN` / `NOT IN` predicate on a batch currently goes through `MultiLiteralsLeafFunction`, which materializes the whole column into `Literal` objects (one heap allocation per row, and non-owning string literals still require building the `Literal` wrapper) and then linearly scans all literals for every row. The cost is `O(rows × literals)` plus `O(rows)` heap allocations per batch. With large `IN` lists (e.g. thousands of partition keys pushed down from SQL), predicate evaluation becomes a measurable part of scan time even though membership testing only needs set lookups. ## Solution Build an immutable, type-specialized lookup structure once when the predicate is constructed, and probe arrow arrays in `O(rows)` without any per-row allocation: - Introduce `LiteralSet` in `src/paimon/common/predicate/`, created from the predicate literals at construction time and owned by `LeafPredicateImpl`. Unsupported types (FLOAT/DOUBLE with NaN semantics, TIMESTAMP with unit conversion, DECIMAL with cross-scale comparison) or heterogeneous literals make the construction return null, and every path falls back to the existing `In` / `NotIn` implementation, so observable behavior stays identical. - Integer family (TINYINT/SMALLINT/INT/BIGINT/DATE, widened to int64): a dense bitmap when the value span is small and close to the literal count, otherwise a hash set, with a min/max range check rejecting out-of-range values before any lookup. - BOOLEAN: two flags. - STRING/BINARY: a hash set of `string_view`s backed by owned storage, with length-range and first-byte bitmap filters applied before hashing; dictionary-encoded arrays are probed once per dictionary plus `O(rows)` index follows. - NULL semantics match the existing functions exactly: null literals are ignored by `IN`, a null literal makes `NOT IN` false for every row, and null column values never match. - The structure is immutable after construction and shared through `std::shared_ptr<const LiteralSet>` when predicates are rebound to other schemas, so concurrent evaluation stays lock-free and the structure is not rebuilt per reader. ## Anything else? No public API, storage format, or protocol change; everything is internal to `src/paimon/common/predicate/`. The statistics/min-max pruning path keeps using the existing `LeafFunction` implementation. ## Are you willing to submit a PR? - [x] I'm willing to submit a PR! ### Solution _No response_ ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
