nsivabalan commented on code in PR #12511:
URL: https://github.com/apache/hudi/pull/12511#discussion_r1892357594
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2564,18 +2598,19 @@ public static HoodieData<HoodieRecord>
convertMetadataToPartitionStatsRecords(Ho
.filter(Objects::nonNull)
.collect(Collectors.toSet());
// Fetch metadata table COLUMN_STATS partition records for above
files
- List<HoodieColumnRangeMetadata<Comparable>> partitionColumnMetadata =
-
tableMetadata.getRecordsByKeyPrefixes(generateKeyPrefixes(validColumnsToIndex,
partitionName), MetadataPartitionType.COLUMN_STATS.getPartitionPath(), false)
- // schema and properties are ignored in getInsertValue, so
simply pass as null
- .map(record -> record.getData().getInsertValue(null, null))
- .filter(Option::isPresent)
- .map(data -> ((HoodieMetadataRecord)
data.get()).getColumnStatsMetadata())
- .filter(stats -> fileNames.contains(stats.getFileName()))
- .map(HoodieColumnRangeMetadata::fromColumnStats)
- .collectAsList();
- // incase of shouldScanColStatsForTightBound = true, we compute
stats for the partition of interest for all files from getLatestFileSlice()
excluding current commit here
- // already fileColumnMetadata contains stats for files from the
current infliht commit. so, we are adding both together and sending it to
collectAndProcessColumnMetadata
- fileColumnMetadata.add(partitionColumnMetadata);
+ List<HoodieColumnRangeMetadata<Comparable>> partitionColumnMetadata
= tableMetadata
+
.getRecordsByKeyPrefixes(generateKeyPrefixes(validColumnsToIndex,
partitionName), MetadataPartitionType.COLUMN_STATS.getPartitionPath(), false)
+ // schema and properties are ignored in getInsertValue, so
simply pass as null
+ .map(record ->
((HoodieMetadataPayload)record.getData()).getColumnStatMetadata())
+ .filter(Option::isPresent)
+ .map(colStatsOpt -> colStatsOpt.get())
+ .filter(stats -> fileNames.contains(stats.getFileName()))
+ .map(HoodieColumnRangeMetadata::fromColumnStats).collectAsList();
+ if (!partitionColumnMetadata.isEmpty()) {
Review Comment:
no changes as such. just added this `if` block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
