JingsongLi opened a new pull request, #11340: URL: https://github.com/apache/paimon/pull/11340
### Purpose Manifest pruning checks individual partition tuples after the global min/max check. With a large partition set and narrow manifest bounds, this can scan many tuples whose values are already outside the manifest's bounds. For sets with at least 64 requested partitions, select the first varying partition column with no nulls. Build a sorted index on demand when a manifest's bounds narrow that column. Binary search the inclusive min/max interval and apply the existing per-tuple check to the candidates in that interval. ### Tests - `PartitionPredicateTest` covers duplicate values at inclusive bounds, multi-column matches, missing statistics, null counts, and serialization. - `PartitionPredicateTest` and `ManifestFileMetaTest` ran on JDK 8 with 109 passing and one skipped. ### Benchmark JMH 1.35 on Apple M4 Pro with Corretto 8. Two forks, three 300 ms warmup iterations, and four 300 ms measurement iterations per case. The input used two `INT` partition keys, with requested tuple `(4*i, 4*(i % 2))` and 256 manifest ranges. Gap manifests cover `[4*i+1, 4*i+3]` on the first key. The wide miss covers the full first-key range and selects 2 on the second key. Average time per operation from `ad4f6e09ef` to this change. | Workload | 1,024 partitions | 16,384 partitions | | --- | ---: | ---: | | Warm gap between requested partitions | 1.285 → 0.124 µs | 31.260 → 0.136 µs | | Warm narrow-range hit | 1.697 → 0.144 µs | 37.694 → 0.153 µs | | Warm wide-range hit | 80.4 → 84.3 ns | 80.2 → 81.7 ns | | Warm wide-range miss on the second key | 4.05 → 4.42 µs | 107.16 → 109.67 µs | | Build predicate and prune 256 gap manifests | 0.359 → 0.076 ms | 9.153 → 2.083 ms | | Build predicate and prune one gap manifest | 0.021 → 0.043 ms | 0.577 → 2.035 ms | The first selective lookup pays for sorting. The 256-manifest batch includes that cost and ran 4.7× and 4.4× faster, respectively. The benchmark measures elapsed time for predicate evaluation with in-memory manifest statistics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
