voonhous commented on code in PR #20328:
URL: https://github.com/apache/hudi/pull/20328#discussion_r4488277082


##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2253,27 +2215,34 @@ private static 
List<HoodieColumnRangeMetadata<Comparable>> getFileStatsRangeMeta
     return readColumnRangeMetadataFrom(partitionPath, fileName, 
datasetMetaClient, columnsToIndex, maxBufferSize, indexVersion);
   }
 
-  public static HoodieData<HoodieRecord> 
convertMetadataToPartitionStatsRecords(HoodiePairData<String, 
List<HoodieColumnRangeMetadata<Comparable>>> columnRangeMetadataPartitionPair,
-                                                                               
  HoodieTableMetaClient dataMetaClient,
-                                                                               
  Map<String, HoodieSchema> colsToIndexSchemaMap,
-                                                                               
  HoodieIndexVersion partitionStatsIndexVersion
-  ) {
-    try {
-      return columnRangeMetadataPartitionPair
-          .flatMapValues(List::iterator)
-          .groupByKey()
-          .map(pair -> {
-            final String partitionName = pair.getLeft();
-            return collectAndProcessColumnMetadata(pair.getRight(), 
partitionName, isShouldScanColStatsForTightBound(dataMetaClient), 
Option.empty(), colsToIndexSchemaMap, partitionStatsIndexVersion);
-          })
-          .flatMap(recordStream -> recordStream.iterator());
-    } catch (Exception e) {
-      throw new HoodieException("Failed to generate column stats records for 
metadata table", e);
-    }
-  }
-
-  public static boolean 
isShouldScanColStatsForTightBound(HoodieTableMetaClient dataMetaClient) {
-    return 
MetadataPartitionType.COLUMN_STATS.isMetadataPartitionAvailable(dataMetaClient);
+  /**
+   * Generates partition statistics only for columns with statistics covering 
every expected base and log file.
+   * Incomplete columns produce partition stats delete records to invalidate 
any previously stored bounds for those columns.
+   * The expected file names must come from the file inventory, independently 
of statistics extraction.
+   */
+  public static Stream<HoodieRecord> 
convertMetadataToPartitionStatsRecords(String partitionPath,
+                                                                           
List<HoodieColumnRangeMetadata<Comparable>> fileColumnMetadata,
+                                                                           
Set<String> expectedFileNames,
+                                                                           
Map<String, HoodieSchema> colsToIndexSchemaMap,
+                                                                           
HoodieIndexVersion partitionStatsIndexVersion) {
+    Map<String, List<HoodieColumnRangeMetadata<Comparable>>> columnMetadataMap 
= fileColumnMetadata.stream()
+        // Negative counts belong to missing/deleted statistics, whereas 
all-null and empty files have valid counts.
+        .filter(range -> range.getValueCount() >= 0 && range.getNullCount() >= 
0)
+        
.collect(Collectors.groupingBy(HoodieColumnRangeMetadata::getColumnName));
+    return colsToIndexSchemaMap.keySet().stream().flatMap(column -> {
+      List<HoodieColumnRangeMetadata<Comparable>> ranges = 
columnMetadataMap.getOrDefault(column, Collections.emptyList());
+      Set<String> coveredFiles = ranges.stream()
+          .map(range -> new StoragePath(range.getFilePath()).getName())
+          .collect(Collectors.toSet());
+      if (expectedFileNames.isEmpty() || 
!coveredFiles.containsAll(expectedFileNames)) {

Review Comment:
   When a column is added via schema evolution and default list at play, the 
new column joins the index automatically. However, 
`ParquetUtils#readColumnStatsFrommetadata` only emit ranges from columns that 
exist in the file footer, hence, files that were written before the schema 
evolution never "cover" it IIUC. 
   
   Could we treat a file that lacks the column as an all-null range, or at 
least add an ADD COLUMNS test that pins the intended behaviour?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to