XiaoHongbo-Hope opened a new pull request, #682: URL: https://github.com/apache/paimon-rust/pull/682
Recreates the execution-time streaming optimization from #678 directly on the latest main. This PR does not contain or depend on #672 and does not push LIMIT into global-index planning. What changed: - Stream exact single-leaf BTree Eq, StartsWith, and Like row IDs in fixed-size execution batches. - Keep the DataFusion residual filter, so normal Limit can stop after enough valid rows while TopK still consumes all shards. - Assign overlapping shard coverage to disjoint row-id intervals without a query-wide deduplication bitmap. - Reuse prepared snapshot and manifest-list metadata across row-id batches, with a bounded manifest-content cache. - Reject oversized encoded or decompressed BTree blocks before allocation. Out of scope: - Bitmap global indexes. - Compound global-index predicates. - Planner-time LIMIT pushdown or candidate early stop. Validation: - 89 BTree-related paimon tests passed. - 45 global-index scanner tests passed. - Prepared snapshot manifest reuse regression passed. - DataFusion equality LIMIT, prefix LIKE LIMIT, and filtered ORDER BY LIMIT regression passed. - cargo clippy for paimon and paimon-datafusion all targets passed with warnings denied. - cargo fmt and git diff check passed. Related to #665 and supersedes the closed #678 implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
