cshuo commented on code in PR #20328:
URL: https://github.com/apache/hudi/pull/20328#discussion_r4488665943
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2253,27 +2215,34 @@ private static
List<HoodieColumnRangeMetadata<Comparable>> getFileStatsRangeMeta
return readColumnRangeMetadataFrom(partitionPath, fileName,
datasetMetaClient, columnsToIndex, maxBufferSize, indexVersion);
}
- public static HoodieData<HoodieRecord>
convertMetadataToPartitionStatsRecords(HoodiePairData<String,
List<HoodieColumnRangeMetadata<Comparable>>> columnRangeMetadataPartitionPair,
-
HoodieTableMetaClient dataMetaClient,
-
Map<String, HoodieSchema> colsToIndexSchemaMap,
-
HoodieIndexVersion partitionStatsIndexVersion
- ) {
- try {
- return columnRangeMetadataPartitionPair
- .flatMapValues(List::iterator)
- .groupByKey()
- .map(pair -> {
- final String partitionName = pair.getLeft();
- return collectAndProcessColumnMetadata(pair.getRight(),
partitionName, isShouldScanColStatsForTightBound(dataMetaClient),
Option.empty(), colsToIndexSchemaMap, partitionStatsIndexVersion);
- })
- .flatMap(recordStream -> recordStream.iterator());
- } catch (Exception e) {
- throw new HoodieException("Failed to generate column stats records for
metadata table", e);
- }
- }
-
- public static boolean
isShouldScanColStatsForTightBound(HoodieTableMetaClient dataMetaClient) {
- return
MetadataPartitionType.COLUMN_STATS.isMetadataPartitionAvailable(dataMetaClient);
+ /**
+ * Generates partition statistics only for columns with statistics covering
every expected base and log file.
+ * Incomplete columns produce partition stats delete records to invalidate
any previously stored bounds for those columns.
+ * The expected file names must come from the file inventory, independently
of statistics extraction.
+ */
+ public static Stream<HoodieRecord>
convertMetadataToPartitionStatsRecords(String partitionPath,
+
List<HoodieColumnRangeMetadata<Comparable>> fileColumnMetadata,
+
Set<String> expectedFileNames,
+
Map<String, HoodieSchema> colsToIndexSchemaMap,
+
HoodieIndexVersion partitionStatsIndexVersion) {
+ Map<String, List<HoodieColumnRangeMetadata<Comparable>>> columnMetadataMap
= fileColumnMetadata.stream()
+ // Negative counts belong to missing/deleted statistics, whereas
all-null and empty files have valid counts.
+ .filter(range -> range.getValueCount() >= 0 && range.getNullCount() >=
0)
+
.collect(Collectors.groupingBy(HoodieColumnRangeMetadata::getColumnName));
+ return colsToIndexSchemaMap.keySet().stream().flatMap(column -> {
+ List<HoodieColumnRangeMetadata<Comparable>> ranges =
columnMetadataMap.getOrDefault(column, Collections.emptyList());
+ Set<String> coveredFiles = ranges.stream()
+ .map(range -> new StoragePath(range.getFilePath()).getName())
+ .collect(Collectors.toSet());
+ if (expectedFileNames.isEmpty() ||
!coveredFiles.containsAll(expectedFileNames)) {
Review Comment:
Thanks for raising this. This PR invalidates partition stats for the new
column when older files lack coverage. Queries remain correct, though pruning
may be reduced.
I added an `ADD COLUMNS` test for COW and MOR. We’ll file a follow-up issue
to optimize partition stats for schema evolution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]