XiaoHongbo-Hope opened a new pull request, #678:
URL: https://github.com/apache/paimon-rust/pull/678
## Summary
- defer global-index row-id lookup and file planning to execution for `Eq` /
`StartsWith` / `Like` scans under DataFusion Top-K
- stream BTree row IDs in bounded 250k batches instead of materializing a
shard/query-wide `RoaringTreemap`
- preserve one global `SortExec: TopK` across all batches and retain the
exact DataFusion residual
- validate BTree block metadata before allocation and optimize escaped
literal LIKE prefixes to exact `StartsWith`
This is a follow-up to #672 and currently contains its commits because #672
has not merged yet. Once #672 lands, this PR reduces to the streaming commit.
## Validation
- `cargo clippy -p paimon -p paimon-datafusion --all-targets -- -D warnings`
- BTree unit tests: 60 passed
- LIKE predicate tests: 5 passed
- DataFusion unit tests: 331 passed; 7 local fixture-dependent tests could
not find the pre-generated Spark warehouse
- DataFusion streaming Top-K integration test passed
- real DLF/OSS table under an 8 GiB cgroup:
- equality + `ORDER BY ... LIMIT 10`: 10 rows, 76.2 s, 4,236,136 KB peak
RSS
- escaped prefix LIKE + `ORDER BY ... LIMIT 10`: 10 rows, 77.7 s,
4,162,556 KB peak RSS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
