danny0405 commented on code in PR #20445:
URL: https://github.com/apache/hudi/pull/20445#discussion_r4533336040
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -108,11 +114,65 @@ protected boolean shouldStopCompactor() {
return false;
}
+ /**
+ * Whether the compaction instant is still pending, i.e. a retry of it has
work to do.
+ */
+ @Override
+ protected boolean isPendingInstant(BaseHoodieWriteClient client, String
instantTime) {
+ return isPendingCompaction(instantTime);
+ }
+
+ protected boolean isPendingCompaction(String instantTime) {
+ HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+ .setConf(context.getStorageConf().newInstance())
+ .setBasePath(compactor.compactionClient.getConfig().getBasePath())
+ .build();
+ return
metaClient.getActiveTimeline().filterPendingCompactionTimeline().containsInstant(instantTime);
+ }
+
+ /**
+ * With multi-writer concurrency, a compaction run holds a heartbeat on its
instant, and a new run refuses to start
+ * while that heartbeat is fresh (it may belong to a concurrent writer). A
run that failed because its client was
+ * closed leaves its heartbeat behind: closing a client stops its heartbeats
without deleting their files, and the
+ * failed run can no longer delete it either. So wait for the heartbeat to
expire before retrying. Without knowing
+ * whose heartbeat it is, this is safe either way: an orphaned heartbeat
expires within the heartbeat timeout, and a
+ * heartbeat a concurrent writer keeps refreshing is waited out, after which
the instant is no longer pending.
+ */
+ @Override
+ protected void prepareRetry(BaseHoodieWriteClient retryClient, String
instantTime) throws IOException, InterruptedException {
+ HoodieWriteConfig config = retryClient.getConfig();
+ if (config.getWriteConcurrencyMode().supportsMultiWriter()) {
+ awaitHeartbeatExpired(config, instantTime);
+ }
+ }
+
+ private void awaitHeartbeatExpired(HoodieWriteConfig config, String
instantTime) throws IOException, InterruptedException {
+ // same expiry as HoodieHeartbeatClient#isHeartbeatExpired, which
compact() checks
+ long heartbeatIntervalMs = config.getHoodieClientHeartbeatIntervalInMs();
+ long maxHeartbeatIntervalMs = heartbeatIntervalMs *
config.getHoodieClientHeartbeatTolerableMisses();
+ HoodieStorage storage = getStorage(config);
+ while (!isShutdownRequested()) {
+ long lastHeartbeatTime =
HoodieHeartbeatUtils.getLastHeartbeatTime(storage, config.getBasePath(),
instantTime);
+ long remainingMs = lastHeartbeatTime + maxHeartbeatIntervalMs -
System.currentTimeMillis();
+ if (remainingMs < 0) {
+ return;
+ }
+ log.info("Waiting up to {} ms for the heartbeat of compaction instant {}
to expire before retrying it", remainingMs, instantTime);
+ // re-read at least once per heartbeat interval, in case a concurrent
writer is refreshing it
+ Thread.sleep(Math.min(remainingMs + 1, heartbeatIntervalMs));
Review Comment:
[P2] Make the heartbeat wait responsive to graceful shutdown.
This sleep can last the full heartbeat interval (60 seconds by default).
HoodieAsyncService.shutdown(false) sets shutdownRequested and calls
executor.shutdown()/awaitTermination(), but does not interrupt the sleeping
worker. Consequently, HoodieStreamingSink.reset(false) can hold the sink
monitor for almost a minute even though the compaction has already failed and
no operation is running. The shutdown flag is only checked after the sleep
finishes.
A focused reproduction with a 4-second heartbeat interval made graceful
shutdown wait 3.9 seconds. Could we use a wait that shutdown can signal, or cap
each sleep between shutdown checks to a short interval? A test requesting
graceful shutdown after entering heartbeat polling would cover this.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -108,11 +114,65 @@ protected boolean shouldStopCompactor() {
return false;
}
+ /**
+ * Whether the compaction instant is still pending, i.e. a retry of it has
work to do.
+ */
+ @Override
+ protected boolean isPendingInstant(BaseHoodieWriteClient client, String
instantTime) {
+ return isPendingCompaction(instantTime);
Review Comment:
Optional simplification: isPendingInstant(client, instantTime) ignores the
supplied client and delegates to isPendingCompaction(), which rereads
compactor.compactionClient. Could we implement the pending check directly here
using client.getConfig().getBasePath(), and do the same for clustering? That
removes the forwarding methods and keeps the new hook consistent with the
explicit client snapshot passed by runWithClientHandoff. The tests can override
isPendingInstant directly.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -108,11 +114,65 @@ protected boolean shouldStopCompactor() {
return false;
}
+ /**
+ * Whether the compaction instant is still pending, i.e. a retry of it has
work to do.
+ */
+ @Override
+ protected boolean isPendingInstant(BaseHoodieWriteClient client, String
instantTime) {
+ return isPendingCompaction(instantTime);
+ }
+
+ protected boolean isPendingCompaction(String instantTime) {
+ HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+ .setConf(context.getStorageConf().newInstance())
+ .setBasePath(compactor.compactionClient.getConfig().getBasePath())
+ .build();
+ return
metaClient.getActiveTimeline().filterPendingCompactionTimeline().containsInstant(instantTime);
+ }
+
+ /**
+ * With multi-writer concurrency, a compaction run holds a heartbeat on its
instant, and a new run refuses to start
+ * while that heartbeat is fresh (it may belong to a concurrent writer). A
run that failed because its client was
+ * closed leaves its heartbeat behind: closing a client stops its heartbeats
without deleting their files, and the
+ * failed run can no longer delete it either. So wait for the heartbeat to
expire before retrying. Without knowing
+ * whose heartbeat it is, this is safe either way: an orphaned heartbeat
expires within the heartbeat timeout, and a
+ * heartbeat a concurrent writer keeps refreshing is waited out, after which
the instant is no longer pending.
+ */
+ @Override
+ protected void prepareRetry(BaseHoodieWriteClient retryClient, String
instantTime) throws IOException, InterruptedException {
+ HoodieWriteConfig config = retryClient.getConfig();
+ if (config.getWriteConcurrencyMode().supportsMultiWriter()) {
+ awaitHeartbeatExpired(config, instantTime);
+ }
+ }
+
+ private void awaitHeartbeatExpired(HoodieWriteConfig config, String
instantTime) throws IOException, InterruptedException {
+ // same expiry as HoodieHeartbeatClient#isHeartbeatExpired, which
compact() checks
+ long heartbeatIntervalMs = config.getHoodieClientHeartbeatIntervalInMs();
+ long maxHeartbeatIntervalMs = heartbeatIntervalMs *
config.getHoodieClientHeartbeatTolerableMisses();
+ HoodieStorage storage = getStorage(config);
+ while (!isShutdownRequested()) {
+ long lastHeartbeatTime =
HoodieHeartbeatUtils.getLastHeartbeatTime(storage, config.getBasePath(),
instantTime);
Review Comment:
[P2] Treat a heartbeat disappearing during the poll as expired.
HoodieHeartbeatUtils.getLastHeartbeatTime() calls storage.exists() and then
storage.getPathInfo(). A concurrent writer can complete this instant and delete
its heartbeat between those calls. FileNotFoundException then escapes
prepareRetry(), so the outer service loop terminates and the streaming query
fails before isPendingInstant() can discover that the compaction already
completed. This is particularly relevant here because the new wait explicitly
supports a concurrent writer refreshing/completing the instant.
I reproduced the service stopping with HoodieIOException by making exists()
return true and getPathInfo() throw FileNotFoundException. Could we treat this
specific missing-file race as an expired heartbeat, then recheck the pending
instant, while preserving other storage errors? Please add a regression test
for deletion between the two storage calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]