yihua commented on code in PR #19802:
URL: https://github.com/apache/hudi/pull/19802#discussion_r4163300788
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -523,16 +523,14 @@ String generateUniqueInstantTime(String
initializationTime) {
protected abstract EngineType getEngineType();
- private Lazy<List<FileSliceAndPartition>> getLazyMergedFileSlices() {
+ private Lazy<List<FileSliceAndPartition>> getLazyMergedFileSlices(String
maxInstantTime) {
Review Comment:
The `maxInstantTime` here after the change is
(1) from MDT writer:
`dataMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant().map(HoodieInstant::requestedTime).orElse(SOLO_COMMIT_TIMESTAMP);`,
i.e., last completed instant T. Before the change, the slice bound was the
last completed-or-pending-compaction instant, which is T or a pending
compaction C later than T. The merged file slice view returns the same base
file, base instant, and committed logs either way, so this path has no
observable change. The file listing is also cut at T, so slices and listing
remain the same.
(2) from async indexer: the indexing action's instant time. This is
different from the last completed-or-pending-compaction instant, whenever a
data commit with a requested time later than the requested indexing action
completes before the index action runs. In that case, master's bootstrap
indexes records from file slices for such completed data commits after the
requested indexing action that the FILES listing had excluded, and the indexing
catch-up that starts at indexUptoInstant does not re-apply them, since it skips
every instant that already has a commit in the metadata table
(`AbstractIndexingCatchupTask#awaitInstantCaughtUp`). So master's run-time
slice bound is the only thing covering those commits for RLI/SI, and bounding
the slices at the indexing instant drops them from the index (reproduced with a
test: 0 of 100 keys of such a commit end up in the record index). The
listing-based indexes (column stats, bloom filters) do miss those commits on
master, since the FILES l
isting is cut at the indexing instant; that is fixed independently in #20216.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
