hakunamatata-sb commented on code in PR #784:
URL: https://github.com/apache/hudi-rs/pull/784#discussion_r4024063731
##########
crates/core/src/table/partition.rs:
##########
@@ -174,6 +174,71 @@ impl PartitionPruner {
})
}
+ /// Returns `true` if a partition directory prefix should be descended
into.
+ ///
+ /// Unlike `should_include`, `partial_path` may resolve fewer segments than
+ /// the schema has fields (we're mid-descent, not at a leaf yet). Any
segment
+ /// this can't safely reason about — not enough schema fields to compare
+ /// against, a single opaque path field (e.g. under a timestamp-based key
+ /// generator, where a segment carries no independently meaningful value),
+ /// a hive-style segment that doesn't parse as `key=value`, or a
cast/compare
+ /// error — is treated as unconstrained, never as a reason to reject. Only
a
+ /// segment that positively fails an already-resolved filter returns
`false`.
+ pub fn should_include_prefix(&self, partial_path: &str) -> bool {
Review Comment:
fixed the single partition issue now that as well get pruned, added a safety
check for MetaField:PartitionPath in that case when we would have multiple
segments for the single partitions.
Along with hive-style partition and is_url_encoded to be always ensured.
The bug related to encoding and splits have been reordered, have done so in
the the split_segments and reused that parse_segments so they don't diverge.
Thanks for pointing this out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
