XiaoHongbo-Hope opened a new pull request, #672: URL: https://github.com/apache/paimon-rust/pull/672
### Purpose Follow-up to #666. A residual `LIKE` predicate on a global BTree index was reported as inexact, so DataFusion did not pass `LIMIT` into Paimon planning. When the fallback index scan exceeded its byte budget, planning fell back to the full table manifests and could exhaust an 8 GiB worker. ### Changes - Report residual `LIKE` as exact when Paimon applies the same row-level matcher, allowing an unordered limit to reach scan planning. - For safe non-PK scans without deletion vectors, evaluate a single global-index `LIKE` sequentially and stop after enough exact row IDs. - Probe a guaranteed subset first (`%` chooses empty and `_` chooses a literal underscore), prune those probes with BTree min/max metadata, and fall back to the full `LIKE` scan if the subset is insufficient. - Cap BTree range/fallback decoding at the remaining limit and add file-level `StartsWith` pruning. - Preserve the existing no-limit, complex-predicate, bitmap-budget, PK, and deletion-vector paths. ### Validation - `cargo test -p paimon btree:: --lib`: 55 passed - `cargo test -p paimon table::global_index_scanner --lib`: 47 passed - `cargo test -p paimon table::read_builder --lib`: 34 passed - `cargo test -p paimon-datafusion --lib`: 329 passed; 7 local failures require the pre-generated Spark warehouse fixtures - `cargo clippy -p paimon -p paimon-datafusion --all-targets -- -D warnings` - Production-scale read-only reproduction (~650 BTree shards / ~73 GB index): `EXPLAIN` completed in 7.69 s at 2.46 GB max RSS, and the actual `LIMIT 1000` query returned and validated 1000 rows in 8.84 s at 2.47 GB max RSS. The previous planning path exceeded 20 GB. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
