XiaoHongbo-Hope opened a new pull request, #589: URL: https://github.com/apache/paimon-rust/pull/589
## Summary Search independent sorted global-index shards concurrently instead of reading BTree and bitmap shards one by one. The existing `global-index.thread-num` option now bounds per-scan shard I/O, while a value of `1` preserves sequential execution. ## Changes - Complete file pruning and unsupported-fallback decisions before starting shard I/O, so a later unsupported shard still falls back without racing an earlier read error. - Query independent BTree and bitmap shards with bounded concurrency and fold their row-id bitmaps incrementally into the final result. - Keep predicates within each shard sequential so its reader is opened once, and return successful BTree readers to the existing cache. - Parse `global-index.thread-num` only when a data-evolution scan will actually evaluate global-index predicates. - Include index kind and file name in shard query errors. ## Performance Measured against the same object-store snapshot with identical result signatures: | `global-index.thread-num` | Plan time | |---:|---:| | 1 | 11.809s | | 8 | 2.790s | | 32 | 1.944s | | 64 | 1.810s | ## Testing - `cargo test --locked -p paimon --lib table::global_index_scanner::tests` (39 passed) - `cargo test --locked -p paimon --lib` (1801 passed, 1 ignored) - `cargo fmt --all -- --check` - `cargo clippy --locked -p paimon --all-targets -- -D warnings` The new tests verify both sequential/concurrent limits and that fallback preflight completes before shard I/O. ## Notes This reuses `global-index.thread-num`, introduced for primary-key vector search in #556. It does not depend on #586 or #587. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
