yihua commented on code in PR #8378:
URL: https://github.com/apache/hudi/pull/8378#discussion_r1187784550
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -320,11 +320,9 @@ public void refreshTimeline() throws IOException {
.build();
switch (meta.getTableType()) {
case COPY_ON_WRITE:
- this.commitTimelineOpt =
Option.of(meta.getActiveTimeline().getCommitTimeline().filterCompletedInstants());
- this.allCommitsTimelineOpt =
Option.of(meta.getActiveTimeline().getAllCommitsTimeline());
- break;
case MERGE_ON_READ:
- this.commitTimelineOpt =
Option.of(meta.getActiveTimeline().getDeltaCommitTimeline().filterCompletedInstants());
+ // we can use getCommitsTimeline for both COW and MOR here,
because for COW there is no deltacommit
+ this.commitsTimelineOpt =
Option.of(meta.getActiveTimeline().getCommitsTimeline().filterCompletedInstants());
Review Comment:
In that case, for MOR table, is the data from compaction commit also pulled
in during incremental pulls, which can cause duplication?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
