yihua commented on code in PR #19802:
URL: https://github.com/apache/hudi/pull/19802#discussion_r4163300788
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -523,16 +523,14 @@ String generateUniqueInstantTime(String
initializationTime) {
protected abstract EngineType getEngineType();
- private Lazy<List<FileSliceAndPartition>> getLazyMergedFileSlices() {
+ private Lazy<List<FileSliceAndPartition>> getLazyMergedFileSlices(String
maxInstantTime) {
Review Comment:
The `maxInstantTime` here after the change is
(1) from MDT writer:
`dataMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant().map(HoodieInstant::requestedTime).orElse(SOLO_COMMIT_TIMESTAMP);`,
i.e., last completed instant T. Before the change, the slice bound was the
last completed-or-pending-compaction instant, which is T or a pending
compaction C later than T. The merged file slice view returns the same base
file, base instant, and committed logs either way, so this path has no
observable change. The file listing is also cut at T, so slices and listing
remain the same.
(2) from async indexer: the indexing action's instant time. This is
different from the last completed-or-pending-compaction instant, whenever a
data commit with a requested time later than the requested indexing action
completes before the index action runs. In that case, master's bootstrap
indexes records from file slices for such completed data commits after the
requested indexing action that the FILES listing had excluded, and the indexing
catch-up that starts at indexUptoInstant then re-applies those same commits.
There is no correctness issue; just that the same data commit is processed
twice which is wasteful.
We should file this as an independent fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
