danny0405 commented on code in PR #20445:
URL: https://github.com/apache/hudi/pull/20445#discussion_r4594726237
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncTableService.java:
##########
@@ -46,4 +53,103 @@ public void start(Function<Boolean, Boolean>
onShutdownCallback) {
}
super.start(onShutdownCallback);
}
+
+ /**
+ * Runs a table service instant with one write client.
+ */
+ @FunctionalInterface
+ protected interface WriteClientRun {
+ void run(BaseHoodieWriteClient client, String instantTime) throws
IOException;
+ }
+
+ /**
+ * Runs the table service for the instant with the write client the service
has when the run starts. That same
+ * client runs the instant and is passed to {@link #isRetryableFailure}, so
a failure is never attributed to a
+ * different client than the one it happened on. On a retryable failure,
waits until the writer has switched the
+ * service to another client, lets the service prepare the retry, and
re-enqueues the instant if it is still pending.
+ *
+ * @param action name of the table service, for logging
+ * @param currentClient the write client the service currently runs with
+ * @param run runs the instant with the given client
+ * @return whether the run completed; false if it failed and was handed back
for a retry or dropped
+ */
+ protected boolean runWithClientHandoff(String action, String instantTime,
Supplier<BaseHoodieWriteClient> currentClient,
+ WriteClientRun run) throws
IOException, InterruptedException {
+ BaseHoodieWriteClient runClient = currentClient.get();
+ try {
+ run.run(runClient, instantTime);
+ return true;
+ } catch (Exception e) {
+ if (!isRetryableFailure(runClient, e)) {
+ throw e;
+ }
+ log.warn("{} for instant {} failed with a write client that was
replaced, retrying it with the new write client",
+ action, instantTime, e);
+ awaitWriteClientReplaced(runClient, currentClient);
+ if (isShutdownRequested()) {
+ return false;
+ }
+ BaseHoodieWriteClient retryClient = currentClient.get();
+ // checked before preparing the retry too: preparing it may wait, which
an instant completed meanwhile doesn't need
+ if (isPendingInstant(retryClient, instantTime)) {
+ prepareRetry(retryClient, instantTime);
+ if (isShutdownRequested()) {
+ return false;
+ }
+ if (isPendingInstant(retryClient, instantTime)) {
+ enqueuePendingAsyncServiceInstant(instantTime);
Review Comment:
[P2] Preserve retry eligibility checks through dequeue.
The heartbeat/pending checks happen before this instant is appended to the
queue, but the next execution goes directly to run.run() without repeating
them. For example, failed instant A is requeued behind B; while B runs, a
concurrent writer acquires A or completes it. When A reaches the front,
compact() rejects the fresh heartbeat or the no-longer-pending instant. That
attempt uses the replacement, non-retired client, so isRetryableFailure()
returns false and the service/query stops despite this being a recoverable
ownership/completion change.
Could we preserve retry state in the queued work and revalidate when
executing it, with handling for concurrent acquisition/completion during the
locked compaction validation as well? A regression test can hold B, requeue A
behind it, mark A completed, then release B and assert that A is skipped and
the service stays alive.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncTableService.java:
##########
@@ -46,4 +53,103 @@ public void start(Function<Boolean, Boolean>
onShutdownCallback) {
}
super.start(onShutdownCallback);
}
+
+ /**
+ * Runs a table service instant with one write client.
+ */
+ @FunctionalInterface
+ protected interface WriteClientRun {
+ void run(BaseHoodieWriteClient client, String instantTime) throws
IOException;
+ }
+
+ /**
+ * Runs the table service for the instant with the write client the service
has when the run starts. That same
+ * client runs the instant and is passed to {@link #isRetryableFailure}, so
a failure is never attributed to a
+ * different client than the one it happened on. On a retryable failure,
waits until the writer has switched the
+ * service to another client, lets the service prepare the retry, and
re-enqueues the instant if it is still pending.
+ *
+ * @param action name of the table service, for logging
+ * @param currentClient the write client the service currently runs with
+ * @param run runs the instant with the given client
+ * @return whether the run completed; false if it failed and was handed back
for a retry or dropped
+ */
+ protected boolean runWithClientHandoff(String action, String instantTime,
Supplier<BaseHoodieWriteClient> currentClient,
+ WriteClientRun run) throws
IOException, InterruptedException {
+ BaseHoodieWriteClient runClient = currentClient.get();
+ try {
+ run.run(runClient, instantTime);
+ return true;
+ } catch (Exception e) {
+ if (!isRetryableFailure(runClient, e)) {
+ throw e;
+ }
+ log.warn("{} for instant {} failed with a write client that was
replaced, retrying it with the new write client",
+ action, instantTime, e);
+ awaitWriteClientReplaced(runClient, currentClient);
+ if (isShutdownRequested()) {
+ return false;
+ }
+ BaseHoodieWriteClient retryClient = currentClient.get();
+ // checked before preparing the retry too: preparing it may wait, which
an instant completed meanwhile doesn't need
+ if (isPendingInstant(retryClient, instantTime)) {
+ prepareRetry(retryClient, instantTime);
+ if (isShutdownRequested()) {
+ return false;
+ }
+ if (isPendingInstant(retryClient, instantTime)) {
+ enqueuePendingAsyncServiceInstant(instantTime);
+ return false;
+ }
+ }
+ log.info("{} for instant {} is no longer pending, not retrying it",
action, instantTime);
+ return false;
+ }
+ }
+
+ /**
+ * Whether a run that failed with the given exception is retried. The run
used {@code runClient}, the client of the
+ * service when the run started. By default no failure is retried and the
service stops with it.
+ */
+ protected boolean isRetryableFailure(BaseHoodieWriteClient runClient,
Exception failure) {
+ return false;
+ }
+
+ /**
+ * Whether the instant is still pending, i.e. a retry of it has work to do.
Only consulted for retryable failures.
+ */
+ protected boolean isPendingInstant(BaseHoodieWriteClient client, String
instantTime) {
+ return true;
+ }
+
+ /**
+ * Prepares the retry of a failed run of the instant with {@code
retryClient}, before it is re-enqueued.
+ */
+ protected void prepareRetry(BaseHoodieWriteClient retryClient, String
instantTime) throws IOException, InterruptedException {
+ }
+
+ /**
+ * Waits up to the given time, returning early when shutdown is requested
(see {@link #onShutdownRequested}) or the
+ * writer switches this service to another client. Callers re-check what
they wait for after it returns.
+ */
+ protected synchronized void waitUnlessShutdown(long millis) throws
InterruptedException {
+ if (!isShutdownRequested()) {
+ wait(millis);
+ }
+ }
+
+ @Override
+ protected synchronized void onShutdownRequested() {
+ notifyAll();
+ }
+
+ /**
+ * Waits until the writer has switched this service away from the given
client. Services notify on this monitor
+ * when they switch clients.
+ */
+ private synchronized void awaitWriteClientReplaced(BaseHoodieWriteClient
client, Supplier<BaseHoodieWriteClient> currentClient)
+ throws InterruptedException {
+ while (!isShutdownRequested() && currentClient.get() == client) {
+ wait(WRITE_CLIENT_POLL_MS);
Review Comment:
Optional simplification: now that updateWriteClient() and
onShutdownRequested() both notifyAll() on this monitor, is the one-second
polling still needed? The condition is checked under the same monitor and the
while loop already handles spurious wakeups, so wait() should suffice for the
built-in compaction/clustering services. This would remove WRITE_CLIENT_POLL_MS
and periodic wakeups while awaiting a handoff.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStructuredStreaming.scala:
##########
@@ -805,6 +810,111 @@ class TestStructuredStreaming extends
HoodieSparkClientTestBase {
}
}
+ private def compactionControlledOpts: Map[String, String] =
+ getCompactionOpts(HoodieTableType.MERGE_ON_READ, isAsyncCompaction = true)
++ Map(
+ DataSourceWriteOptions.OPERATION.key -> UPSERT_OPERATION_OPT_VAL,
+ DataSourceWriteOptions.RECORD_MERGE_MODE.key -> "CUSTOM",
+ DataSourceWriteOptions.RECORD_MERGE_STRATEGY_ID.key ->
CompactionControlledSparkRecordMerger.STRATEGY_ID,
+ DataSourceWriteOptions.RECORD_MERGE_IMPL_CLASSES.key ->
classOf[CompactionControlledSparkRecordMerger].getName,
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet")
+
+ private def writeUpdatesToSource(sourcePath: String, instantTime: String,
numRecords: Int): Unit = {
+ spark.read.json(spark.sparkContext.parallelize(
+ recordsToStrings(dataGen.generateUniqueUpdates(instantTime,
numRecords)).asScala.toList, 1))
+ .write.mode(SaveMode.Append).json(sourcePath)
+ }
+
+ /**
+ * An async compaction running on the sink's write client when a schema
change makes the sink re-create its client
+ * may fail once that client is closed. Its service retries it with the new
client, and the query keeps running.
+ */
+ @Test
+ def testStructuredStreamingRetriesAsyncCompactionFailedByWriteClientSwap():
Unit = {
+ val (sourcePath, destPath) = initStreamingSourceAndDestPath("source",
"dest")
+ val opts = compactionControlledOpts +
(DataSourceWriteOptions.SET_NULL_FOR_MISSING_COLUMNS.key -> "true")
+ val inputDF =
spark.read.json(spark.sparkContext.parallelize(recordsToStrings(dataGen.generateInsertsForPartition(
+ "000", 20,
HoodieTestDataGenerator.DEFAULT_FIRST_PARTITION_PATH)).asScala.toList, 1))
+ inputDF.write.mode(SaveMode.Append).json(sourcePath)
+ val query = initWritingStreamingQuery(inputDF.schema, sourcePath,
destPath, opts)
+ val compactionStarted = new CountDownLatch(1)
+ val releaseCompaction = new CountDownLatch(1)
+ try {
+ query.processAllAvailable()
+
CompactionControlledSparkRecordMerger.holdNextCompactionThenFail(compactionStarted,
releaseCompaction)
+ writeUpdatesToSource(sourcePath, "001", 10)
+ query.processAllAvailable()
+ assertTrue(compactionStarted.await(2, TimeUnit.MINUTES), "the async
compaction must start")
+
+ // evolve the table schema outside of the query, so that the next
micro-batch re-creates the sink's write client
+ spark.read.json(spark.sparkContext.parallelize(
+ recordsToStrings(dataGen.generateUniqueUpdates("002",
5)).asScala.toList, 1))
+ .withColumn("new_col", lit("evolved"))
+
.write.format("org.apache.hudi").options(opts).mode(SaveMode.Append).save(destPath)
+ writeUpdatesToSource(sourcePath, "003", 10)
+ query.processAllAvailable()
+ // the micro-batch that replaced the write client completed without
waiting for the running compaction
+ assertTrue(CompactionControlledSparkRecordMerger.holdingCompaction,
+ "the micro-batch must not wait for the running async compaction")
+
+ // the compaction held on the previous write client now fails, and is
retried with the new one
+ releaseCompaction.countDown()
+ waitTillAtleastNCompletedCompactions(destPath, 1)
Review Comment:
Test improvement: could we capture the instant held by
CompactionControlledSparkRecordMerger and wait for that specific instant to
complete? This helper only checks that the table has at least one completed
compaction, so a different completed instant can satisfy it without proving the
failed instant was retried. Asserting a second attempt for the captured instant
on the replacement client would make the end-to-end client-swap guarantee much
stronger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]