danny0405 commented on code in PR #20445:
URL: https://github.com/apache/hudi/pull/20445#discussion_r4533336040


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -108,11 +114,65 @@ protected boolean shouldStopCompactor() {
     return false;
   }
 
+  /**
+   * Whether the compaction instant is still pending, i.e. a retry of it has 
work to do.
+   */
+  @Override
+  protected boolean isPendingInstant(BaseHoodieWriteClient client, String 
instantTime) {
+    return isPendingCompaction(instantTime);
+  }
+
+  protected boolean isPendingCompaction(String instantTime) {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(context.getStorageConf().newInstance())
+        .setBasePath(compactor.compactionClient.getConfig().getBasePath())
+        .build();
+    return 
metaClient.getActiveTimeline().filterPendingCompactionTimeline().containsInstant(instantTime);
+  }
+
+  /**
+   * With multi-writer concurrency, a compaction run holds a heartbeat on its 
instant, and a new run refuses to start
+   * while that heartbeat is fresh (it may belong to a concurrent writer). A 
run that failed because its client was
+   * closed leaves its heartbeat behind: closing a client stops its heartbeats 
without deleting their files, and the
+   * failed run can no longer delete it either. So wait for the heartbeat to 
expire before retrying. Without knowing
+   * whose heartbeat it is, this is safe either way: an orphaned heartbeat 
expires within the heartbeat timeout, and a
+   * heartbeat a concurrent writer keeps refreshing is waited out, after which 
the instant is no longer pending.
+   */
+  @Override
+  protected void prepareRetry(BaseHoodieWriteClient retryClient, String 
instantTime) throws IOException, InterruptedException {
+    HoodieWriteConfig config = retryClient.getConfig();
+    if (config.getWriteConcurrencyMode().supportsMultiWriter()) {
+      awaitHeartbeatExpired(config, instantTime);
+    }
+  }
+
+  private void awaitHeartbeatExpired(HoodieWriteConfig config, String 
instantTime) throws IOException, InterruptedException {
+    // same expiry as HoodieHeartbeatClient#isHeartbeatExpired, which 
compact() checks
+    long heartbeatIntervalMs = config.getHoodieClientHeartbeatIntervalInMs();
+    long maxHeartbeatIntervalMs = heartbeatIntervalMs * 
config.getHoodieClientHeartbeatTolerableMisses();
+    HoodieStorage storage = getStorage(config);
+    while (!isShutdownRequested()) {
+      long lastHeartbeatTime = 
HoodieHeartbeatUtils.getLastHeartbeatTime(storage, config.getBasePath(), 
instantTime);
+      long remainingMs = lastHeartbeatTime + maxHeartbeatIntervalMs - 
System.currentTimeMillis();
+      if (remainingMs < 0) {
+        return;
+      }
+      log.info("Waiting up to {} ms for the heartbeat of compaction instant {} 
to expire before retrying it", remainingMs, instantTime);
+      // re-read at least once per heartbeat interval, in case a concurrent 
writer is refreshing it
+      Thread.sleep(Math.min(remainingMs + 1, heartbeatIntervalMs));

Review Comment:
   [P2] Make the heartbeat wait responsive to graceful shutdown.
   
   This sleep can last the full heartbeat interval (60 seconds by default). 
HoodieAsyncService.shutdown(false) sets shutdownRequested and calls 
executor.shutdown()/awaitTermination(), but does not interrupt the sleeping 
worker. Consequently, HoodieStreamingSink.reset(false) can hold the sink 
monitor for almost a minute even though the compaction has already failed and 
no operation is running. The shutdown flag is only checked after the sleep 
finishes.
   
   A focused reproduction with a 4-second heartbeat interval made graceful 
shutdown wait 3.9 seconds. Could we use a wait that shutdown can signal, or cap 
each sleep between shutdown checks to a short interval? A test requesting 
graceful shutdown after entering heartbeat polling would cover this.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -108,11 +114,65 @@ protected boolean shouldStopCompactor() {
     return false;
   }
 
+  /**
+   * Whether the compaction instant is still pending, i.e. a retry of it has 
work to do.
+   */
+  @Override
+  protected boolean isPendingInstant(BaseHoodieWriteClient client, String 
instantTime) {
+    return isPendingCompaction(instantTime);

Review Comment:
   Optional simplification: isPendingInstant(client, instantTime) ignores the 
supplied client and delegates to isPendingCompaction(), which rereads 
compactor.compactionClient. Could we implement the pending check directly here 
using client.getConfig().getBasePath(), and do the same for clustering? That 
removes the forwarding methods and keeps the new hook consistent with the 
explicit client snapshot passed by runWithClientHandoff. The tests can override 
isPendingInstant directly.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -108,11 +114,65 @@ protected boolean shouldStopCompactor() {
     return false;
   }
 
+  /**
+   * Whether the compaction instant is still pending, i.e. a retry of it has 
work to do.
+   */
+  @Override
+  protected boolean isPendingInstant(BaseHoodieWriteClient client, String 
instantTime) {
+    return isPendingCompaction(instantTime);
+  }
+
+  protected boolean isPendingCompaction(String instantTime) {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(context.getStorageConf().newInstance())
+        .setBasePath(compactor.compactionClient.getConfig().getBasePath())
+        .build();
+    return 
metaClient.getActiveTimeline().filterPendingCompactionTimeline().containsInstant(instantTime);
+  }
+
+  /**
+   * With multi-writer concurrency, a compaction run holds a heartbeat on its 
instant, and a new run refuses to start
+   * while that heartbeat is fresh (it may belong to a concurrent writer). A 
run that failed because its client was
+   * closed leaves its heartbeat behind: closing a client stops its heartbeats 
without deleting their files, and the
+   * failed run can no longer delete it either. So wait for the heartbeat to 
expire before retrying. Without knowing
+   * whose heartbeat it is, this is safe either way: an orphaned heartbeat 
expires within the heartbeat timeout, and a
+   * heartbeat a concurrent writer keeps refreshing is waited out, after which 
the instant is no longer pending.
+   */
+  @Override
+  protected void prepareRetry(BaseHoodieWriteClient retryClient, String 
instantTime) throws IOException, InterruptedException {
+    HoodieWriteConfig config = retryClient.getConfig();
+    if (config.getWriteConcurrencyMode().supportsMultiWriter()) {
+      awaitHeartbeatExpired(config, instantTime);
+    }
+  }
+
+  private void awaitHeartbeatExpired(HoodieWriteConfig config, String 
instantTime) throws IOException, InterruptedException {
+    // same expiry as HoodieHeartbeatClient#isHeartbeatExpired, which 
compact() checks
+    long heartbeatIntervalMs = config.getHoodieClientHeartbeatIntervalInMs();
+    long maxHeartbeatIntervalMs = heartbeatIntervalMs * 
config.getHoodieClientHeartbeatTolerableMisses();
+    HoodieStorage storage = getStorage(config);
+    while (!isShutdownRequested()) {
+      long lastHeartbeatTime = 
HoodieHeartbeatUtils.getLastHeartbeatTime(storage, config.getBasePath(), 
instantTime);

Review Comment:
   [P2] Treat a heartbeat disappearing during the poll as expired.
   
   HoodieHeartbeatUtils.getLastHeartbeatTime() calls storage.exists() and then 
storage.getPathInfo(). A concurrent writer can complete this instant and delete 
its heartbeat between those calls. FileNotFoundException then escapes 
prepareRetry(), so the outer service loop terminates and the streaming query 
fails before isPendingInstant() can discover that the compaction already 
completed. This is particularly relevant here because the new wait explicitly 
supports a concurrent writer refreshing/completing the instant.
   
   I reproduced the service stopping with HoodieIOException by making exists() 
return true and getPathInfo() throw FileNotFoundException. Could we treat this 
specific missing-file race as an expired heartbeat, then recheck the pending 
instant, while preserving other storage errors? Please add a regression test 
for deletion between the two storage calls.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to