nsivabalan commented on code in PR #20445:
URL: https://github.com/apache/hudi/pull/20445#discussion_r4594003622


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -108,11 +114,65 @@ protected boolean shouldStopCompactor() {
     return false;
   }
 
+  /**
+   * Whether the compaction instant is still pending, i.e. a retry of it has 
work to do.
+   */
+  @Override
+  protected boolean isPendingInstant(BaseHoodieWriteClient client, String 
instantTime) {
+    return isPendingCompaction(instantTime);
+  }
+
+  protected boolean isPendingCompaction(String instantTime) {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(context.getStorageConf().newInstance())
+        .setBasePath(compactor.compactionClient.getConfig().getBasePath())
+        .build();
+    return 
metaClient.getActiveTimeline().filterPendingCompactionTimeline().containsInstant(instantTime);
+  }
+
+  /**
+   * With multi-writer concurrency, a compaction run holds a heartbeat on its 
instant, and a new run refuses to start
+   * while that heartbeat is fresh (it may belong to a concurrent writer). A 
run that failed because its client was
+   * closed leaves its heartbeat behind: closing a client stops its heartbeats 
without deleting their files, and the
+   * failed run can no longer delete it either. So wait for the heartbeat to 
expire before retrying. Without knowing
+   * whose heartbeat it is, this is safe either way: an orphaned heartbeat 
expires within the heartbeat timeout, and a
+   * heartbeat a concurrent writer keeps refreshing is waited out, after which 
the instant is no longer pending.
+   */
+  @Override
+  protected void prepareRetry(BaseHoodieWriteClient retryClient, String 
instantTime) throws IOException, InterruptedException {
+    HoodieWriteConfig config = retryClient.getConfig();
+    if (config.getWriteConcurrencyMode().supportsMultiWriter()) {
+      awaitHeartbeatExpired(config, instantTime);
+    }
+  }
+
+  private void awaitHeartbeatExpired(HoodieWriteConfig config, String 
instantTime) throws IOException, InterruptedException {
+    // same expiry as HoodieHeartbeatClient#isHeartbeatExpired, which 
compact() checks
+    long heartbeatIntervalMs = config.getHoodieClientHeartbeatIntervalInMs();
+    long maxHeartbeatIntervalMs = heartbeatIntervalMs * 
config.getHoodieClientHeartbeatTolerableMisses();
+    HoodieStorage storage = getStorage(config);
+    while (!isShutdownRequested()) {
+      long lastHeartbeatTime = 
HoodieHeartbeatUtils.getLastHeartbeatTime(storage, config.getBasePath(), 
instantTime);

Review Comment:
   Fixed in 183b787f. `awaitHeartbeatExpired` now catches 
`FileNotFoundException` from `getLastHeartbeatTime` and treats the heartbeat as 
expired. Other storage errors still propagate. `runWithClientHandoff` then 
re-checks `isPendingInstant`, so if the concurrent writer completed the 
instant, the service logs that and moves on instead of stopping.
   
   Regression test: 
`TestAsyncTableServiceRetry#testHeartbeatDeletedWhileReadingCountsAsExpired`. 
`exists()` returns true, `getPathInfo()` throws `FileNotFoundException`, and 
the service stays up and runs the retry.
   



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -108,11 +114,65 @@ protected boolean shouldStopCompactor() {
     return false;
   }
 
+  /**
+   * Whether the compaction instant is still pending, i.e. a retry of it has 
work to do.
+   */
+  @Override
+  protected boolean isPendingInstant(BaseHoodieWriteClient client, String 
instantTime) {
+    return isPendingCompaction(instantTime);
+  }
+
+  protected boolean isPendingCompaction(String instantTime) {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(context.getStorageConf().newInstance())
+        .setBasePath(compactor.compactionClient.getConfig().getBasePath())
+        .build();
+    return 
metaClient.getActiveTimeline().filterPendingCompactionTimeline().containsInstant(instantTime);
+  }
+
+  /**
+   * With multi-writer concurrency, a compaction run holds a heartbeat on its 
instant, and a new run refuses to start
+   * while that heartbeat is fresh (it may belong to a concurrent writer). A 
run that failed because its client was
+   * closed leaves its heartbeat behind: closing a client stops its heartbeats 
without deleting their files, and the
+   * failed run can no longer delete it either. So wait for the heartbeat to 
expire before retrying. Without knowing
+   * whose heartbeat it is, this is safe either way: an orphaned heartbeat 
expires within the heartbeat timeout, and a
+   * heartbeat a concurrent writer keeps refreshing is waited out, after which 
the instant is no longer pending.
+   */
+  @Override
+  protected void prepareRetry(BaseHoodieWriteClient retryClient, String 
instantTime) throws IOException, InterruptedException {
+    HoodieWriteConfig config = retryClient.getConfig();
+    if (config.getWriteConcurrencyMode().supportsMultiWriter()) {
+      awaitHeartbeatExpired(config, instantTime);
+    }
+  }
+
+  private void awaitHeartbeatExpired(HoodieWriteConfig config, String 
instantTime) throws IOException, InterruptedException {
+    // same expiry as HoodieHeartbeatClient#isHeartbeatExpired, which 
compact() checks
+    long heartbeatIntervalMs = config.getHoodieClientHeartbeatIntervalInMs();
+    long maxHeartbeatIntervalMs = heartbeatIntervalMs * 
config.getHoodieClientHeartbeatTolerableMisses();
+    HoodieStorage storage = getStorage(config);
+    while (!isShutdownRequested()) {
+      long lastHeartbeatTime = 
HoodieHeartbeatUtils.getLastHeartbeatTime(storage, config.getBasePath(), 
instantTime);
+      long remainingMs = lastHeartbeatTime + maxHeartbeatIntervalMs - 
System.currentTimeMillis();
+      if (remainingMs < 0) {
+        return;
+      }
+      log.info("Waiting up to {} ms for the heartbeat of compaction instant {} 
to expire before retrying it", remainingMs, instantTime);
+      // re-read at least once per heartbeat interval, in case a concurrent 
writer is refreshing it
+      Thread.sleep(Math.min(remainingMs + 1, heartbeatIntervalMs));

Review Comment:
   Fixed in 183b787f. The heartbeat wait no longer uses `Thread.sleep`. It 
waits on the service monitor through a new 
`HoodieAsyncTableService.waitUnlessShutdown(millis)`. I added an 
`onShutdownRequested()` hook to `HoodieAsyncService`, called right after the 
shutdown flags are set, and the table services override it to `notifyAll()`. So 
`shutdown(false)` wakes the waiting worker immediately, and the retry returns 
without re-enqueueing. The wait also re-checks the flag before waiting, so a 
shutdown that lands just before the wait isn't missed.
   
   Regression test: 
`TestAsyncTableServiceRetry#testGracefulShutdownEndsHeartbeatWait`. With a 30s 
heartbeat interval, it asserts `shutdown(false)` returns in well under a third 
of the interval and that no retry is enqueued.
   



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -108,11 +114,65 @@ protected boolean shouldStopCompactor() {
     return false;
   }
 
+  /**
+   * Whether the compaction instant is still pending, i.e. a retry of it has 
work to do.
+   */
+  @Override
+  protected boolean isPendingInstant(BaseHoodieWriteClient client, String 
instantTime) {
+    return isPendingCompaction(instantTime);

Review Comment:
   Done in 183b787f. `AsyncCompactService.isPendingInstant` and 
`AsyncClusteringService.isPendingInstant` now build the meta client from 
`client.getConfig().getBasePath()`, and the `isPendingCompaction` / 
`isPendingClustering` forwarders are removed. The test harness overrides 
`isPendingInstant` directly.
   
   Related, from your summary: the retry now checks `isPendingInstant` before 
the heartbeat wait as well as after. An instant that completed in the meantime 
is dropped without waiting. Covered by 
`testNoHeartbeatWaitForAnInstantNoLongerPending`, which verifies the heartbeat 
file is never read.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to