JingsongLi opened a new pull request, #1245: URL: https://github.com/apache/paimon-rust/pull/1245
### Purpose Closes #1243. Global full-text search requests the full candidate count from every archive before applying the caller's Top-K. Retain bounded per-shard candidates when their equivalence to the existing final result can be certified, and provide a reproducible comparison against exhaustive collection. ### Brief change log - Enable the fast path only for FAST-mode global searches with disjoint shard ranges, no raw sources, and no deletion exclusions or enabled deletion vectors. - Ask the native engine to evaluate the complete query and filter with K+1 hits. Keep K only when the cutoff score is strict and selected row IDs are distinct; otherwise retry with the original full candidate limit. - Carry the optional Top-K request through scheduled search tasks, preserving exhaustive behavior for older task frames. - Add `full-text-index.top-k.enabled` (default true) to compare or disable the optimization. - Add a production-path release benchmark with exact ID/score assertions, alternating mode order, distinct-score and tied-score corpora, protocol and raw sample results. ### Tests - `cargo +1.98.0 test --locked -p paimon --all-targets --features vortex -- --test-threads=4`: passed; core library 4,642 passed, 6 ignored, plus all integration targets. - `cargo +1.98.0 test --locked -p paimon-datafusion --test read_tables fulltext_tests -- --test-threads=4`: 4 passed. - Workspace Clippy, all targets with Vortex, denying warnings: passed. - Formatting and diff checks: passed. - New regressions cover strict/tied/duplicate/non-finite cutoffs, inclusive overlapping ranges, filtered local/scheduled result parity, and an assertion that workers actually return K candidates for eligible queries. Existing DV, raw-fallback, snapshot and compound-query tests remain covered. - Independent static review completed without actionable findings. An initial unconstrained core run exceeded the existing shared-cache child test's five-second startup deadline during concurrent builds. The complete four-thread rerun passed. Local validation ran on macOS; other platforms remain for CI. Release sample: Apple M4 Pro, 48 GiB RAM, macOS 27.0.1, Rust 1.98.0; 100,000 matches, four shards, K=10, 20 warm-cache iterations per mode: | Corpus | Exhaustive p50 | Certified p50 | Exhaustive p95 | Certified p95 | | --- | ---: | ---: | ---: | ---: | | Distinct high scores | 65.680 ms | 1.300 ms | 71.194 ms | 1.584 ms | | Boundary ties | 61.498 ms | 59.544 ms | 65.693 ms | 65.988 ms | Every measured result has identical IDs and scores. These synthetic local results are not a general speedup, cold-cache/object-store, RSS, or Lance comparison. ### API and Format Adds a Rust table option/CoreOptions accessor and an optional search-task field with a default. Index archives and score semantics are unchanged. PK and raw temporary-index search paths retain their behavior. ### Documentation Add the benchmark protocol and raw sample timings under benchmarks/full-text, and document the optimization's limits in the performance guide. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
