YinChunGuang opened a new issue, #19301:
URL: https://github.com/apache/hudi/issues/19301
h2. Summary
When a Flink streaming writer encounters a transient HDFS error during
metadata table (MDT) log file write, the entire deltacommit fails. Because the
MDT write happens before `saveAsComplete`, the data table's inflight instant is
never transitioned to complete. On restart, the coordinator retries from the
oldest pending instant, but the inflight file may no longer exist on HDFS,
causing a permanent restart failure loop that exhausts all retry attempts and
kills the job.
h2. Environment
|| Component || Version ||
| Apache Flink | 1.18.1 |
| Apache Hudi | 1.1.1 |
| Hadoop | 3.1.1 |
| JDK | 1.8 |
| Table Type | MOR (Merge On Read) |
| Index Type | BUCKET |
| Metadata Table | Enabled (default) |
h2. Table Configuration
{code:sql}
CREATE TABLE `t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock`(
...
)
PARTITIONED BY (`dt`)
WITH (
'connector' = 'hudi',
'table.type' = 'MERGE_ON_READ',
'index.type' = 'BUCKET',
'hoodie.metadata.enable' = 'true', -- MDT enabled
'hoodie.cleaner.policy' = 'KEEP_LATEST_COMMITS',
'hoodie.cleaner.commits.retained' = '10',
'hoodie.clean.manual.batch.enable' = 'true', -- Auto clean OFF
'hoodie.archive.manual.batch.enable' = 'true', -- Auto archive OFF
'compaction.tasks' = '1',
'compaction.async.enabled' = 'true',
'compaction.delta_commits' = '20',
'compaction.trigger.strategy' = 'num_commits',
'write.tasks' = '8',
'changelog.enabled' = 'true',
'write.precombine' = 'true',
'write.ignore.failed' = 'true'
);
{code}
h2. Failure Timeline
h3. Phase 1: MDT Write Failure → Global Job Failure
# A high-priority compaction completes at *02:53:51*
# The next deltacommit `20260710185353327` starts at *02:53:51*
# The MDT log file is created successfully:
{code}
02:53:51,332 INFO DFSClient [] - DFSClient_createFile_..._FOR_CREATE
[.metadata/.hoodie_mt_log_...]
{code}
# 4.4 seconds later, while writing to the MDT log file, `hsync` fails:
{code}
02:53:55,728 WARN DFSClient [] - Caught exception in hsync: Holder
DFSClient_-563620450 does not have any open files
02:53:55,758 WARN DFSClient [] - DataStreamer Exception:
InterruptedException
02:53:55,760 WARN DFSClient [] - Abandoning all blocks in the pipeline
{code}
# The {{HoodieAppendException}} propagates up through
{{BaseHoodieWriteClient.commit()}}:
{code}
02:53:55,761 ERROR BaseFlinkCommitActionExecutor [] - Failed to commit
deltacommit at 20260710185353327
{code}
# The uncaught exception triggers Flink's global failure mechanism:
{code}
02:53:55,787 INFO NonThrownExecutor [] - Executing global failure.
02:53:55,794 INFO ExecutionGraph [] - Switching TaskExecutionJobVertex from
RUNNING to FAILED
{code}
h3. Phase 2: Permanent Restart Failure Loop
After restart, the coordinator finds the incomplete instant and retries, but
the inflight file was cleaned up or lost during the failed attempt:
{code}
02:59:15 INFO StreamWriteOperatorCoordinator [] - Coordinator starts.
02:59:15 INFO StreamWriteOperatorCoordinator [] - Found the earliest
pending deltacommit instant: 20260710185353327
02:59:15 INFO StreamWriteOperatorCoordinator [] - New checkpoint triggers
the pending instant commit: 20260710185353327
02:59:15 ERROR StreamWriteOperatorCoordinator [] - Executor execute action
[commitInstants] failed.
org.apache.hudi.exception.HoodieIOException: Failed to commit deltacommit
instant 20260710185353327
Caused by: org.apache.hudi.exception.HoodieIOException: Cannot find
inflight file for ...20260710185353327.deltacommit.inflight
{code}
This repeats for *10 restart attempts over 30 minutes*. All fail with the
same error. At *03:26:41* the job reaches terminal state FAILED.
h2. Log Evidence
h3. 1. MDT log file created (successful)
{code}
02:53:51,332 INFO DFSClient [] -
DFSClient_createFile_DFSClient_-563620450_1246882867_SYNC_CREATE_FOR_CREATE_ONLY
[.metadata/.hoodie_mt_log_20260710185353327_0-1586-02-000003055]
{code}
h3. 2. hsync fails after 4.4 seconds
{code}
02:53:55,728 WARN DFSClient [] - Caught exception in hsync: Holder
DFSClient_-563620450 does not have any open files
java.io.IOException: The file /tmp/.../ .metadata/.hoodie_mt_log_... has
been closed and is unavailable for write
{code}
h3. 3. DataNode pipeline broken
{code}
02:53:55,758 WARN DFSClient [] - DataStreamer Exception:
InterruptedException
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:853)
at
org.apache.hudi.org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:16861)
02:53:55,760 WARN DFSClient [] - Abandoning all blocks in the pipeline
{code}
h3. 4. Commit fails
{code}
02:53:55,761 ERROR BaseFlinkCommitActionExecutor [] - Failed to commit
deltacommit at 20260710185353327
org.apache.hudi.exception.HoodieAppendException: Failed to close writer for
metadata log file
Caused by: java.io.IOException: The file has been closed and is
unavailable for write
{code}
h3. 5. Flink global failure
{code}
02:53:55,787 INFO NonThrownExecutor [] - Executing global failure.
02:53:55,794 INFO ExecutionGraph [] - Switching from RUNNING to FAILED.
Failure Error: org.apache.hudi.exception.HoodieAppendException
{code}
h3. 6. Restart fails — inflight file missing
{code}
02:59:15 ERROR StreamWriteOperatorCoordinator [] - Executor execute action
[commitInstants] failed.
Caused by: org.apache.hudi.exception.HoodieIOException: Cannot find inflight
file for ...20260710185353327.deltacommit.inflight
{code}
h3. 7. Terminal state FAILED
{code}
03:26:41,727 INFO JobMaster [] - Job job_f6d4153a4696d9b1f29c924dc8704f9b
reached terminal state FAILED.
{code}
h2. Root Cause Analysis
h3. Why the first deltacommit failed
The {{BaseHoodieWriteClient.commit()}} method writes to the metadata table
*before* calling {{saveAsComplete}} on the data table timeline. When the MDT
write fails (in this case due to an HDFS DataNode pipeline break),
{{saveAsComplete}} is never reached, leaving the data table's deltacommit stuck
in INFLIGHT state.
Additionally, in {{commitStats()}}, the {{catch}} block only catches
{{IOException}}, but {{HoodieAppendException}} extends {{RuntimeException}}, so
it bypasses the error handling and propagates directly to Flink's
{{NonThrownExecutor}}, triggering a global job failure.
{code:java}
// BaseHoodieWriteClient.commit() — line ~320
// ① Write MDT (BEFORE saveAsComplete)
writeToMetadataTable(skipStreamingWritesToMetadataTable, table, instantTime,
tableWriteStats.getMetadataTableWriteStats(), metadata);
// ② Never reached if ① fails
activeTimeline.saveAsComplete(false,
table.getMetaClient().createNewInstant(
HoodieInstant.State.INFLIGHT, commitActionType, instantTime),
Option.of(metadata), ...);
{code}
{code:java}
// BaseHoodieWriteClient.commitStats() — line ~279
try {
commit(table, commitActionType, instantTime, metadata,
tableWriteStats, skipStreamingWritesToMetadataTable);
} catch (IOException e) {
// HoodieAppendException extends RuntimeException — NOT caught here!
throw new HoodieCommitException(...);
}
// HoodieAppendException escapes to Flink's NonThrownExecutor → global
failure
{code}
h3. Why every restart attempt fails
The {{StreamWriteOperatorCoordinator}} retries from the oldest pending
instant on each restart. However, the inflight file for that instant no longer
exists — it was either cleaned up during the failed write or lost due to the
pipeline failure. The {{ActiveTimelineV2.transitionStateToComplete()}} method
requires the inflight file to exist, so it throws:
{code:java}
// ActiveTimelineV2.transitionStateToComplete()
Preconditions.checkArgument(isInflight(inflightInstant),
"Unable to find instant (%s)", inflightInstant);
{code}
h3. Why Flink is more affected than Spark
{{SparkRDDWriteClient}} overrides {{writeToMetadataTable()}} with
streaming-aware handling. {{HoodieFlinkWriteClient}} does *not* override this
method — it uses the base class default which always performs synchronous MDT
writes. Furthermore, Flink hardcodes {{skipStreamingWritesToMetadataTable =
false}}, meaning MDT writes are never skipped.
h2. Proposed Fixes
h3. Fix 1: Tolerate MDT write failure (BaseHoodieWriteClient.commit)
Wrap the MDT write in a try-catch block so its failure does not block
{{saveAsComplete}} on the data table:
{code:java}
protected void commit(HoodieTable table, String commitActionType, String
instantTime,
HoodieCommitMetadata metadata, TableWriteStats
tableWriteStats,
boolean skipStreamingWritesToMetadataTable) throws
IOException {
finalizeWrite(table, instantTime,
tableWriteStats.getDataTableWriteStats());
saveInternalSchema(table, instantTime, metadata);
try {
writeToMetadataTable(skipStreamingWritesToMetadataTable, table,
instantTime,
tableWriteStats.getMetadataTableWriteStats(), metadata);
} catch (Exception e) {
LOG.error("Failed to write to metadata table for instant {}, "
+ "proceeding with data table commit. MDT can be recovered via
bootstrap.",
instantTime, e);
}
if (isLockRequiredForActiveTimeline(table.getMetaClient())) {
this.txnManager.beginStateChange(
Option.of(table.getMetaClient().createNewInstant(
HoodieInstant.State.INFLIGHT, commitActionType,
instantTime)));
}
activeTimeline.saveAsComplete(false,
table.getMetaClient().createNewInstant(
HoodieInstant.State.INFLIGHT, commitActionType, instantTime),
Option.of(metadata), tableWriteStats.getRuntimeStats());
}
{code}
h3. Fix 2: Skip missing inflight files on restart
(StreamWriteOperatorCoordinator)
When the inflight file is missing, skip the instant instead of failing:
{code:java}
// StreamWriteOperatorCoordinator.commitInstants()
if (allInstants.isEmpty()) {
// no pending commits, skip
return;
}
HoodieInstant instant = allInstants.get(0);
// Skip if inflight file no longer exists (e.g., cleaned up after a previous
failed attempt)
if (!writeClient.getTable().getMetaClient().getActiveTimeline()
.filter(inflights -> inflights.contains(instant)).isPresent()) {
LOG.warn("Skipping instant {} — inflight file not found in active
timeline", instant);
return;
}
{code}
h3. Fix 3: Catch HoodieAppendException in commitStats
Extend the catch block to include {{RuntimeException}} or specifically
{{HoodieAppendException}}:
{code:java}
try {
commit(table, commitActionType, instantTime, metadata,
tableWriteStats, skipStreamingWritesToMetadataTable);
} catch (IOException | HoodieAppendException e) {
throw new HoodieCommitException(
"Failed to commit " + instantTime + " time=" + commitTime, e);
}
{code}
h2. Current Workaround
{code:sql}
ALTER TABLE t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock
SET TBLPROPERTIES (
'hoodie.metadata.enable' = 'false'
);
{code}
This disables MDT writes entirely, preventing the coupling issue. It is safe
for single-writer scenarios using BUCKET index with compaction enabled.
**additionLog**
`2026-07-16 03:02:37,282 INFO
org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] -
Completed [==>20260716025337008__deltacommit__INFLIGHT]
2026-07-16 03:02:37,287 INFO org.apache.hudi.common.fs.FSUtils
[] - Removed directory at
hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/.temp/20260716025337008
2026-07-16 03:02:37,288 INFO org.apache.hudi.client.BaseHoodieWriteClient
[] - Committed 20260716025337008
2026-07-16 03:02:37,293 INFO
org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] -
Marking instant complete [==>20260716025337008__deltacommit__INFLIGHT]
2026-07-16 03:02:37,294 INFO
org.apache.hudi.client.transaction.TransactionManager [] - State change
ending for action instant
Option{val=[==>20260716025337008__deltacommit__INFLIGHT]}
2026-07-16 03:02:37,295 INFO
org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock,
Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1efbc60b[Write
locks = 1, Read locks = 0], Thread meta-event-handle, In-process lock state
RELEASING
2026-07-16 03:02:37,295 INFO
org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock,
Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1efbc60b[Write
locks = 0, Read locks = 0], Thread meta-event-handle, In-process lock state
RELEASED
2026-07-16 03:02:37,296 INFO
org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock,
Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1efbc60b[Write
locks = 0, Read locks = 0], Thread meta-event-handle, In-process lock state
ALREADY_RELEASED
2026-07-16 03:02:37,297 INFO
org.apache.hudi.client.transaction.lock.LockManager [] - Released
connection created for acquiring lock
2026-07-16 03:02:37,297 INFO
org.apache.hudi.client.transaction.TransactionManager [] - State change
ended for action instant
Option{val=[==>20260716025337008__deltacommit__INFLIGHT]}
2026-07-16 03:02:37,297 INFO
org.apache.hudi.client.heartbeat.HoodieHeartbeatClient [] - Stopping
heartbeat for instant 20260716025337008
2026-07-16 03:02:37,298 INFO
org.apache.hudi.client.heartbeat.HoodieHeartbeatClient [] - Stopped
heartbeat for instant 20260716025337008
2026-07-16 03:02:37,300 INFO
org.apache.hudi.client.heartbeat.HeartbeatUtils [] - Deleted the
heartbeat for instant 20260716025337008
2026-07-16 03:02:37,300 INFO
org.apache.hudi.client.heartbeat.HoodieHeartbeatClient [] - Deleted
heartbeat file for instant 20260716025337008
2026-07-16 03:02:37,301 ERROR
org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor
executes action [commits the instant 20260716030218327] error
java.lang.IllegalArgumentException: File
hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/timeline/20260716025337008.deltacommit.inflight
does not exist!
at
org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.transitionStateToComplete(ActiveTimelineV2.java:563)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:171)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:162)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:178)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:321)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:107)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:617)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:602)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$commitInstants$10(StreamWriteOperatorCoordinator.java:557)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
~[?:1.8.0_281]
at
java.util.concurrent.ConcurrentSkipListMap$EntrySpliterator.forEachRemaining(ConcurrentSkipListMap.java:3535)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_281]
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[?:1.8.0_281]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstants(StreamWriteOperatorCoordinator.java:557)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$3(StreamWriteOperatorCoordinator.java:299)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:131)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_281]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_281]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
2026-07-16 03:02:37,304 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Trying to recover from a global failure.
org.apache.flink.util.FlinkException: Global failure triggered by
OperatorCoordinator for 'bucket_write:
yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock'
(operator b754eda8ef527d9494a0b43e9bf7abf3).
at
org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:624)
~[flink-dist-1.18.1.jar:1.18.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$start$0(StreamWriteOperatorCoordinator.java:230)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.utils.NonThrownExecutor.handleException(NonThrownExecutor.java:143)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:134)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_281]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_281]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_281]
Caused by: org.apache.hudi.exception.HoodieException: Executor executes
action [commits the instant 20260716030218327] error
... 6 more
Caused by: java.lang.IllegalArgumentException: File
hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/timeline/20260716025337008.deltacommit.inflight
does not exist!
at
org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.transitionStateToComplete(ActiveTimelineV2.java:563)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:171)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:162)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:178)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:321)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:107)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:617)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:602)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$commitInstants$10(StreamWriteOperatorCoordinator.java:557)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
~[?:1.8.0_281]
at
java.util.concurrent.ConcurrentSkipListMap$EntrySpliterator.forEachRemaining(ConcurrentSkipListMap.java:3535)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_281]
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[?:1.8.0_281]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstants(StreamWriteOperatorCoordinator.java:557)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$3(StreamWriteOperatorCoordinator.java:299)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:131)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
... 3 more
2026-07-16 03:02:37,305 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - 4 tasks will be restarted to recover from a global failure.
2026-07-16 03:02:37,306 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
insert-into_hoodie_catalog.yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock
(a9775e03fcf62008b13b248ee8693bb9) switched from state RUNNING to RESTARTING.
2026-07-16 03:02:37,307 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
compact_plan_generate (1/1)
(f49a8f6332f3f91794ef2cae46a713ef_2c190601d2e331d8fbe808700bcf6c4e_0_2)
switched from RUNNING to CANCELING.
2026-07-16 03:02:37,308 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_task
-> Sink: compact_commit (1/1)
(f49a8f6332f3f91794ef2cae46a713ef_edd9ec76dbb7cd545d93de35f574c84e_0_2)
switched from RUNNING to CANCELING.
2026-07-16 03:02:37,308 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] ->
row_data_to_hoodie_record (1/1)
(f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_2)
switched from RUNNING to CANCELING.
2026-07-16 03:02:37,308 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing
registered reader after failure for subtask 0 (#2) of source Source:
source_kafka_01[1].
2026-07-16 03:02:37,309 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - bucket_write:
yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock
(1/1) (f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_2)
switched from RUNNING to CANCELING.
2026-07-16 03:02:37,325 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
compact_plan_generate (1/1)
(f49a8f6332f3f91794ef2cae46a713ef_2c190601d2e331d8fbe808700bcf6c4e_0_2)
switched from CANCELING to CANCELED.
2026-07-16 03:02:37,326 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_task
-> Sink: compact_commit (1/1)
(f49a8f6332f3f91794ef2cae46a713ef_edd9ec76dbb7cd545d93de35f574c84e_0_2)
switched from CANCELING to CANCELED.
2026-07-16 03:02:37,330 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - bucket_write:
yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock
(1/1) (f49a8f6332f3f91794ef2cae46a713ef_b754eda8ef527d9494a0b43e9bf7abf3_0_2)
switched from CANCELING to CANCELED.
2026-07-16 03:02:37,335 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
source_kafka_01[1] -> Calc[2] -> ConstraintEnforcer[3] ->
row_data_to_hoodie_record (1/1)
(f49a8f6332f3f91794ef2cae46a713ef_cbc357ccb763df2852fee8c4fc7d55f2_0_2)
switched from CANCELING to CANCELED.
2026-07-16 03:02:37,336 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Clearing resource requirements of job a9775e03fcf62008b13b248ee8693bb9
2026-07-16 03:02:37,337 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedTaskManagerTracker
[] - Clear all pending allocations for job a9775e03fcf62008b13b248ee8693bb9.
2026-07-16 03:03:37,311 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
insert-into_hoodie_catalog.yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock
(a9775e03fcf62008b13b248ee8693bb9) switched from state RESTARTING to RUNNING.
2026-07-16 03:03:37,312 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring job
a9775e03fcf62008b13b248ee8693bb9 from Checkpoint 74 @ 1784142138303 for
a9775e03fcf62008b13b248ee8693bb9 located at
hdfs://rtcluster/user/flink/checkpoints/a9775e03fcf62008b13b248ee8693bb9/chk-74.
2026-07-16 03:03:37,313 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No master
state to restore
2026-07-16 03:03:37,313 INFO
org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator
[] - Resetting coordinator to checkpoint.
2026-07-16 03:03:37,314 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Closing
SourceCoordinator for source Source: source_kafka_01[1].
2026-07-16 03:03:37,317 INFO
org.apache.flink.kafka.shaded.org.apache.kafka.common.utils.AppInfoParser [] -
App info kafka.admin.client for
hudi-11-test-1-ym3_11_mem94-enumerator-admin-client unregistered
2026-07-16 03:03:37,318 INFO
org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.Metrics [] -
Metrics scheduler closed
2026-07-16 03:03:37,319 INFO
org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.Metrics [] -
Closing reporter
org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.JmxReporter
2026-07-16 03:03:37,319 INFO
org.apache.flink.kafka.shaded.org.apache.kafka.common.metrics.Metrics [] -
Metrics reporters closed
2026-07-16 03:03:37,320 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source
coordinator for source Source: source_kafka_01[1] closed.
2026-07-16 03:03:37,321 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Restoring
SplitEnumerator of source Source: source_kafka_01[1] from checkpoint.
2026-07-16 03:03:37,322 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Starting
split enumerator for source Source: source_kafka_01[1].
2026-07-16 03:03:37,332 INFO
org.apache.flink.kafka.shaded.org.apache.kafka.clients.admin.AdminClientConfig
[] - AdminClientConfig values:
auto.include.jmx.reporter = true
bootstrap.servers = [bd-dw-kafka-n1.bestpay.mid:9092,
bd-dw-kafka-n2.bestpay.mid:9092, bd-dw-kafka-n3.bestpay.mid:9092,
bd-dw-kafka-n4.bestpay.mid:9092, bd-dw-kafka-n5.bestpay.mid:9092]
client.dns.lookup = use_all_dns_ips
client.id = hudi-11-test-1-ym3_11_mem94-enumerator-admin-client
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.connect.timeout.ms = null
sasl.login.read.timeout.ms = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.login.retry.backoff.max.ms = 10000
sasl.login.retry.backoff.ms = 100
sasl.mechanism = GSSAPI
sasl.oauthbearer.clock.skew.seconds = 30
sasl.oauthbearer.expected.audience = null
sasl.oauthbearer.expected.issuer = null
sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
sasl.oauthbearer.jwks.endpoint.url = null
sasl.oauthbearer.scope.claim.name = scope
sasl.oauthbearer.sub.claim.name = sub
sasl.oauthbearer.token.endpoint.url = null
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.2
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
