github-actions[bot] commented on code in PR #66481:
URL: https://github.com/apache/doris/pull/66481#discussion_r3757475316
##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -1160,24 +1205,45 @@ private List<String> prunePartitionNames(List<String>
allPartNames,
}
/**
- * Prunes Hudi RELATIVE partition paths (positional {@code "2024/01"} or
hive-style {@code
- * "year=2024/month=01"}) using {@link
HudiScanPlanProvider#parsePartitionValues} (handles both layouts and
- * unescapes) + {@link #matchesPredicates}. Used by the non-hive-sync
{@link #applyFilter} branch, whose
- * candidate source is the Hudi metadata listing — the same relative-path
shape the scan feeds fsView. Static +
- * package-private for offline unit testing.
+ * Prunes Hudi RELATIVE partition paths using the table-config layout
carried alongside the Hudi metadata
+ * listing. Used by the non-hive-sync {@link #applyFilter} branch, whose
candidate source is the same
+ * relative-path shape the scan feeds fsView. Static + package-private for
offline unit testing.
*/
static List<String> prunePartitionPaths(List<String> allPartPaths,
- List<String> partKeyNames, Map<String, List<String>> predicates) {
+ List<String> partKeyNames, Map<String, List<String>> predicates,
boolean hiveStylePartitioning) {
List<String> matched = new ArrayList<>();
for (String partPath : allPartPaths) {
- Map<String, String> partValues =
HudiScanPlanProvider.parsePartitionValues(partPath, partKeyNames);
+ Map<String, String> partValues =
HudiScanPlanProvider.parsePartitionValues(
+ partPath, partKeyNames, hiveStylePartitioning);
if (matchesPredicates(partValues, predicates)) {
matched.add(partPath);
}
}
return matched;
}
+ /**
+ * Translates Hive Sync partition names to the relative physical paths
accepted by Hudi's FileSystemView.
+ * HMS names are always hive-style, while the physical paths follow {@code
hiveStylePartitioning}; matching
+ * their canonical unescaped value maps preserves the HMS-pruned set
without passing HMS-only names into the
+ * scan. Static + package-private for the offline Hive Sync +
positional-layout regression test.
+ */
+ static List<String> matchPhysicalPartitionPaths(List<String>
matchedHmsPartitionNames,
+ List<String> physicalPartitionPaths, List<String> partKeyNames,
boolean hiveStylePartitioning) {
+ Set<Map<String, String>> matchedValues =
matchedHmsPartitionNames.stream()
+ .map(partitionName -> parsePartitionName(partitionName,
partKeyNames))
+ .collect(Collectors.toSet());
+ List<String> matchedPaths = new ArrayList<>();
+ for (String physicalPath : physicalPartitionPaths) {
+ Map<String, String> physicalValues =
HudiScanPlanProvider.parsePartitionValues(
+ physicalPath, partKeyNames, hiveStylePartitioning);
+ if (matchedValues.contains(physicalValues)) {
Review Comment:
[P1] Preserve Hive Sync extractor provenance when mapping physical paths
This comparison assumes Hive Sync's partition extractor is an identity
decode of the physical path. [Hudi 1.0.2
documents](https://hudi.apache.org/docs/1.0.2/syncing_metastore/) that
`SinglePartPartitionValueExtractor` can register a one-column physical path
such as `2024/03/15` in HMS as `event_date=2024-03-15`. After that HMS name is
predicate-pruned, `matchedValues` contains the hyphenated value, while
`parsePartitionValues("2024/03/15", ..., false)` returns the slash value, so
this equality rejects the only physical path; the handle carries an empty
pruned set and both COW and MOR scans return zero splits. Resolve matched HMS
partitions to their storage locations while carrying their logical values
separately, or apply the configured extractor to physical candidates. Please
add a sync-enabled extractor regression that verifies both the exact
FileSystemView path and emitted `columns_from_path` value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
