nsivabalan commented on code in PR #12121:
URL: https://github.com/apache/hudi/pull/12121#discussion_r1805227999
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2201,7 +2201,7 @@ public static HoodieData<HoodieRecord>
convertMetadataToPartitionStatsRecords(Ho
int parallelism = Math.max(Math.min(partitionedWriteStats.size(),
metadataConfig.getPartitionStatsIndexParallelism()), 1);
boolean shouldScanColStatsForTightBound =
MetadataPartitionType.COLUMN_STATS.isMetadataPartitionAvailable(dataMetaClient)
- && metadataConfig.isPartitionStatsIndexTightBoundEnabled() &&
WriteOperationType.isPartitionStatsTightBoundRequired(commitMetadata.getOperationType());
+ &&
(metadataConfig.isPartitionStatsIndexTightBoundEnabledOnEveryCommit() ||
WriteOperationType.isPartitionStatsTightBoundRequired(commitMetadata.getOperationType()));
Review Comment:
I am not sure if this is doing what we wanted to achieve.
what we discussed was:
the config of interest (subject to name change), when enabled:
We will do tighter bound computation after every commit.
if not:
we only do tighter bound on certain cases.
a. if operation type if compaction or clusteirng.
OR b. if a new file slice(i.e. base file) is added to an existing file
group.
And 2nd condition in L 2203 should be doing that. but I only see it checks
for operation types (compaction, clustering).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
