nsivabalan commented on code in PR #12369:
URL: https://github.com/apache/hudi/pull/12369#discussion_r1862760556
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/index/AbstractIndexingCatchupTask.java:
##########
@@ -155,6 +164,15 @@ HoodieInstant awaitInstantCaughtUp(HoodieInstant instant) {
return null;
}
if (!instant.isCompleted()) {
+ // check if instant heartbeat expired, if so then ignore this instant
+ try {
+ if (table.getConfig().getFailedWritesCleanPolicy().isLazy() &&
heartbeatClient.isHeartbeatExpired(instant.requestedTime())) {
+ LOG.info("Ignoring instant " + instant + " as heartbeat expired");
+ return null;
Review Comment:
let me summarize my understanding.
I also went over the design in the RFC
https://github.com/apache/hudi/blob/master/rfc/rfc-45/rfc-45.md as well.
```
6 Start indexing catchup in a separate thread (that can be interrupted upon
timeout)
6.a For each instant to catchup
6.a.i if instant is completed and has corresponding deltacommit in
metadata timeline then continue
6.a.ii if instant is inflight, then reload active timeline periodically
until completed or timed out
6.a.iii update metadata table, if needed, within a lock
```
so, in this case, just that we missed to account for a failed write
scenario.
a. for a single writer, we should ignore the instant to catch up and
continue in the loop to next instant.
b. for multi-writer scenario, if there is active heart beat, we have to
wait. so that it either goes to completion. or it elapses time out on which
case we can again ignore from the list of instant to catchup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
