hakunamatata-sb opened a new issue, #779:
URL: https://github.com/apache/hudi-rs/issues/779
## Component
`crates/core/src/table/listing.rs`
(`FileLister::list_relevant_partition_paths`), `crates/core/src/storage/mod.rs`
(`get_leaf_dirs`), `crates/core/src/table/partition.rs` (`PartitionPruner`)
## Summary
When `hoodie.metadata.enable` is off (or the metadata table isn't otherwise
available — e.g. `hoodie.table.version <= 6` predates it),
`FileLister::list_relevant_partition_paths` falls back to listing partition
directories directly from storage.
That fallback currently does a **full recursive listing of every partition
directory** under the table's base path, regardless of any partition filters
supplied by the query, and only applies `PartitionPruner::should_include`
*after* every leaf partition path has already been discovered
(`crates/core/src/table/listing.rs`, current `list_relevant_partition_paths`):
```rust
async fn list_relevant_partition_paths(&self) -> Result<Vec<String>> {
if !is_table_partitioned(&self.hudi_configs)? {
return Ok(vec![EMPTY_PARTITION_PATH.to_string()]);
}
let top_level_dirs: Vec<String> = self
.storage
.list_dirs(None)
.await?
.into_iter()
.filter(|dir| !LAKE_FORMAT_METADATA_DIRS.contains(&dir.as_str()))
.collect();
let mut partition_paths = Vec::new();
for dir in top_level_dirs {
partition_paths.extend(get_leaf_dirs(&self.storage,
Some(&dir)).await?);
}
// should_include is only applied here, after the full tree is already
listed —
// no filter narrows top_level_dirs or the get_leaf_dirs recursion above
it.
if partition_paths.is_empty() || self.partition_pruner.is_empty() {
return Ok(partition_paths);
}
Ok(partition_paths
.into_iter()
.filter(|path_str| self.partition_pruner.should_include(path_str))
.collect())
}
```
`get_leaf_dirs` (`crates/core/src/storage/mod.rs`) recurses into every
directory unconditionally — it takes no predicate parameter at all, so there is
nothing to check before calling `storage.list_dirs(subdir)` on a candidate
directory. Every directory in the partition tree incurs a storage `list` call
(a network round trip on cloud object stores or HDFS), even when its name
already violates a supplied filter and could never contain a matching row.
## Impact
For a table with N partitions and a highly selective query (e.g. filtering
down to a single partition), this results in O(N) storage list calls per query
instead of O(1) or O(depth). On cloud storage (S3/GCS) or HDFS, each `list`
call is a network round trip, so this scales query latency and API cost
directly with total partition count rather than with the query's actual
selectivity — independent of whether the query is otherwise cheap. This is
worse on tables with many partitions and only gets worse as the table grows,
even though the query itself doesn't.
## Suggested fix
Reject a directory *before* listing its children whenever a supplied
partition filter already rules it out, so `list_dirs` is only ever called on
the subtree that could possibly contain a match.
## Cross-reference
The Java implementation already does something like this at the listing
layer. In `apache/hudi`,
`hudi-common/src/main/java/org/apache/hudi/metadata/FileSystemBackedTableMetadata.java`,
`getPartitionPathWithPathPrefixUsingFilterExpression` prunes directories
level-by-level during its recursive walk, using a partially-bound expression
before descending and a fully-bound one on completed partition paths:
```java
// evaluated against each directory before recursing into it
pathsToList.addAll(result.stream()
.filter(entry -> entry.getValue().isPresent())
.map(entry -> entry.getValue().get())
.filter(path -> partialBoundExpr instanceof Predicates.TrueExpression
|| (Boolean) partialBoundExpr.eval(extractPartitionValues(...)))
.collect(Collectors.toList()));
```
Same two-tier shape as the fix proposed above (partial predicate during
descent, full predicate at the leaf)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
