nsivabalan commented on a change in pull request #4821:
URL: https://github.com/apache/hudi/pull/4821#discussion_r814892221
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java
##########
@@ -429,25 +428,21 @@ public void mergeArchiveFiles(List<FileStatus>
compactCandidate) throws IOExcept
.collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
HoodieInstant.getComparableAction(i.getAction()))));
- // If metadata table is enabled, do not archive instants which are more
recent than the last compaction on the
- // metadata table.
- if (config.isMetadataTableEnabled()) {
- try (HoodieTableMetadata tableMetadata =
HoodieTableMetadata.create(table.getContext(), config.getMetadataConfig(),
- config.getBasePath(),
FileSystemViewStorageConfig.SPILLABLE_DIR.defaultValue())) {
- Option<String> latestCompactionTime =
tableMetadata.getLatestCompactionTime();
Review comment:
sorry. is this addressed? if not, can you help me understand how we will
overcome the above scenario if we remove this code block ?
bcoz, our filtering of additional commits in metadata table happens with
LogRecordReader. which is effective only w/ latest file slice. Once compaction
kicks in, we may never know what all commits have gone into the compacted base
file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
