hakunamatata-sb opened a new pull request, #784:
URL: https://github.com/apache/hudi-rs/pull/784
## Description
<!--- Describe your changes in detail -->
When the metadata table is unavailable,
`FileLister::list_relevant_partition_paths` fell back to a full recursive
listing of every partition directory (`get_leaf_dirs`), applying
`PartitionPruner::should_include` only after every leaf partition path had
already been discovered. A selective query therefore issued as many storage
`list` calls as an unfiltered one, scaling cost and latency with total
partition count rather than query selectivity.
This change:
- Threads a `should_descend` predicate through `get_leaf_dirs`, checked
before recursing into each child directory.
- Adds `PartitionPruner::should_include_prefix`, a partial-path predicate
used as that descent-time filter. It fails open (never wrongly prunes) whenever
it can't positively prove a resolved segment violates a filter — no filters
present, an opaque single-field partition schema (the case for
timestamp-based-keygen tables, which already collapse to one
`_hoodie_partition_path` field), a malformed hive-style segment, or a
cast/comparison error.
- Keeps the existing full-path `should_include` filter unchanged as the
final correctness backstop — descent pruning is purely an optimization.
<!--- If it fixes an open issue, please link to the issue here. -->
Fixes #779
<!--- Please link any related issues and PRs as well. -->
NA
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
`cargo test -p hudi-core table::partition::tests` — 23/23 pass, including 6
new unit tests for `should_include_prefix` (hive-style, non-hive-style,
url-encoded, malformed segment, no filters, single-field/timestamp-keygen
schema).
`cargo test -p hudi-core table::listing::test` — new regression test
`partition_filter_pruning_reduces_storage_list_calls` asserts exact storage
`list` call counts against a `CountingObjectStore`: 7 calls unfiltered vs. 3
calls with a selective filter (`byteField = "10"`) on
`V6ComplexkeygenHivestyle`, plus correctness of the returned partition paths.
`cargo test -p hudi-core storage::tests::use_storage_to_get_leaf_dirs*`
still pass with the new predicate parameter.
`cargo clippy -p hudi-core --lib -- -D warnings` clean on the touched files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
