Zouxxyy opened a new pull request, #8353:
URL: https://github.com/apache/paimon/pull/8353
### Purpose
When a format table scan receives an `AndPartitionPredicate` (e.g. from
combining a static
pushed-down partition filter with a runtime/DPP filter), the scan-path
prefix optimization
and per-directory pruning were skipped because only
`DefaultPartitionPredicate` was handled.
This caused a full listing of all partition directories from the table root
— expensive on
object stores like OSS/S3.
This PR adds `extractPartitionPredicate()` which recursively unwraps
`AndPartitionPredicate`
into a single `Predicate` for pruning. Children that cannot be expressed as
a `Predicate`
(e.g. `MultiplePartitionPredicate`) are skipped; correctness is still
guaranteed by the
in-memory `partitionFilter.test()` in `plan()`.
Also replaces `exists()` + `getFileStatus()` with a single `getFileStatus()`
catching
`FileNotFoundException` to save one RPC per directory on object stores.
### Tests
- `FormatTableScanTest#testFindPartitionsWithAndPartitionPredicate` — unit
test verifying
`And(year=2024, month>3 AND month<10)` prunes to 6 partitions with only 1
list call.
- `FormatTableTestBase` — Spark e2e test: join on dt triggers DPP runtime
filter, combined
with static pushdown, verifies the scan prunes to 2 splits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
