YinChunGuang opened a new issue, #19332:
URL: https://github.com/apache/hudi/issues/19332
### Bug Description
**What happened:**
Flink streaming write jobs fail with ArrayIndexOutOfBoundsException: -3 when
the Metadata Table (MDT) contains corrupted HFile entries left behind by HDFS
lease expiration. Once corrupted, the MDT file persists on disk and causes
every subsequent job restart to fail — creating a permanent recovery loop.
**Phase 1: Write-path failure (07-18 01:05:11)**
MDT log file write succeeds (935ms) but close() → completeFile() fails
because NameNode reports the DFSClient no longer holds any open files:
Caused by: org.apache.hadoop.ipc.RemoteException: File does not exist:
.../.files-0000-0_20260718010427669.log.1_0-1-0 (inode 5373900553)
Holder DFSClient_NONMAPREDUCE_217638024_99 does not have any open files.
at FSNamesystem.checkLease(FSNamesystem.java:2857)
The same DFSClient had just completed an MDT compaction base HFile write 4
seconds earlier (01:05:11)
No GC evidence — log timestamps are continuous with ≤2s gaps
Creates an empty/truncated MDT log file on HDFS
**Phase 2: Read-path failure (07-19 02:08:33)**
25 hours later, after 20 successful checkpoints, compaction plan generation
reads the MDT:
HoodieTableMetadataUtil.loadingLatestMergedFileSlices → HoodieFileGroupReader
→ HFileReaderImpl.readTrailer()
→ readMajorVersion(bytes, -3)
→ ArrayIndexOutOfBoundsException: -3
This propagates through:
BaseTableMetadata.getAllFilesInPartition() → uncaught
Timeline Service HTTP endpoint → HTTP 500
PriorityBasedFileSystemView falls back to InMemory view (compaction path
only)
BucketStreamWriteFunction queries MDT directly — no fallback → job crashes
### Environment
**Hudi version:** 1.1.1
**Query engine:** flink1.18.1
**Relevant configs:**
### Logs and Stack Trace
2026-07-18 01:05:14,529 INFO
org.apache.hudi.common.table.view.FileSystemViewManager [] - Creating View
Manager with storage type MEMORY.
2026-07-18 01:05:14,529 INFO
org.apache.hudi.keygen.constant.KeyGeneratorType [] - No key
generator type is set properly
2026-07-18 01:05:14,531 INFO
org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder [] - Building
HoodieLogFormat Writer
2026-07-18 01:05:14,531 INFO
org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder [] - Computing
the next log version for files-0000-0 in
hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/files
2026-07-18 01:05:14,532 INFO
org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder [] - Computed
the next log version for files-0000-0 in
hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/files
as 1 with write-token 0-1-0
2026-07-18 01:05:14,532 INFO
org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder [] -
HoodieLogFile on path
hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/files/.files-0000-0_20260718010427669.log.1_0-1-0
2026-07-18 01:05:14,587 INFO
org.apache.hudi.table.marker.DirectWriteMarkers [] - [direct]
Created marker file
hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/.hoodie/.temp/20260718010427669/files/.files-0000-0_20260718010427669.log.1_0-1-0.marker.CREATE
in 4 ms
2026-07-18 01:05:14,589 INFO
org.apache.hudi.common.table.log.HoodieLogFormatWriter [] - Created a new
log file:
HoodieLogFile{pathStr='hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/files/.files-0000-0_20260718010427669.log.1_0-1-0',
fileLen=0}
2026-07-18 01:05:15,465 INFO org.apache.hudi.io.HoodieAppendHandle
[] - AppendHandle for partitionPath files filePath
files/.files-0000-0_20260718010427669.log.1_0-1-0, took 935 ms.
2026-07-18 01:05:16,295 ERROR
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor [] - Error
upserting bucketType UPDATE for partition :files
org.apache.hudi.exception.HoodieUpsertException: Failed to close UpdateHandle
at
org.apache.hudi.io.HoodieAppendHandle.close(HoodieAppendHandle.java:581)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.io.FlinkAppendHandle.close(FlinkAppendHandle.java:106)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.delta.BaseFlinkDeltaCommitActionExecutor.handleUpdate(BaseFlinkDeltaCommitActionExecutor.java:57)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.handleUpsertPartition(BaseFlinkCommitActionExecutor.java:178)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:121)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:105)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:100)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.delta.FlinkUpsertPreppedDeltaCommitActionExecutor.execute(FlinkUpsertPreppedDeltaCommitActionExecutor.java:54)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.HoodieFlinkMergeOnReadTable.upsertPrepped(HoodieFlinkMergeOnReadTable.java:87)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.lambda$upsertPreppedRecords$2(HoodieFlinkWriteClient.java:171)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_281]
at
java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1628)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_281]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
~[?:1.8.0_281]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
~[?:1.8.0_281]
at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
~[?:1.8.0_281]
at
java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
~[?:1.8.0_281]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
~[?:1.8.0_281]
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
~[?:1.8.0_281]
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
~[?:1.8.0_281]
at
java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[?:1.8.0_281]
at
org.apache.hudi.client.HoodieFlinkWriteClient.upsertPreppedRecords(HoodieFlinkWriteClient.java:174)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.upsertPreppedRecords(HoodieFlinkWriteClient.java:74)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.upsertAndCommit(FlinkHoodieBackedTableMetadataWriter.java:161)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.upsertAndCommit(FlinkHoodieBackedTableMetadataWriter.java:54)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.commitInternal(HoodieBackedTableMetadataWriter.java:1819)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.commitInternal(FlinkHoodieBackedTableMetadataWriter.java:128)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.commit(FlinkHoodieBackedTableMetadataWriter.java:104)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:1167)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:1399)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieClient.writeTableMetadata(BaseHoodieClient.java:285)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.writeToMetadataTable(BaseHoodieWriteClient.java:339)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:320)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:107)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:617)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:602)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$commitInstants$10(StreamWriteOperatorCoordinator.java:557)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
~[?:1.8.0_281]
at
java.util.concurrent.ConcurrentSkipListMap$EntrySpliterator.forEachRemaining(ConcurrentSkipListMap.java:3535)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_281]
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[?:1.8.0_281]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstants(StreamWriteOperatorCoordinator.java:557)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$3(StreamWriteOperatorCoordinator.java:299)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:131)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_281]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_281]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
Caused by: org.apache.hadoop.ipc.RemoteException: File does not exist:
/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/files/.files-0000-0_20260718010427669.log.1_0-1-0
(inode 5373900553) Holder DFSClient_NONMAPREDUCE_217638024_99 does not have
any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2857)
at
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFileInternal(FSDirWriteFileOp.java:693)
at
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFile(FSDirWriteFileOp.java:679)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2900)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:959)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:614)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at org.apache.hadoop.ipc.Client.call(Client.java:1508)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at org.apache.hadoop.ipc.Client.call(Client.java:1405)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at com.sun.proxy.$Proxy41.complete(Unknown Source) ~[?:?]
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:563)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_281]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_281]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at com.sun.proxy.$Proxy42.complete(Unknown Source) ~[?:?]
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:975)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:932)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:896)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:844)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hudi.hadoop.fs.SizeAwareFSDataOutputStream.close(SizeAwareFSDataOutputStream.java:79)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hudi.common.table.log.HoodieLogFormatWriter.closeStream(HoodieLogFormatWriter.java:248)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.common.table.log.HoodieLogFormatWriter.close(HoodieLogFormatWriter.java:238)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.io.HoodieAppendHandle.close(HoodieAppendHandle.java:555)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
... 57 more
2026-07-18 01:05:16,333 INFO
org.apache.hudi.client.transaction.TransactionManager [] - State change
ending for action instant
Option{val=[==>20260718010427669__deltacommit__INFLIGHT]}
2026-07-18 01:05:16,334 INFO
org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock,
Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1dd9cf6d[Write
locks = 1, Read locks = 0], Thread meta-event-handle, In-process lock state
RELEASING
2026-07-18 01:05:16,334 INFO
org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock,
Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1dd9cf6d[Write
locks = 0, Read locks = 0], Thread meta-event-handle, In-process lock state
RELEASED
2026-07-18 01:05:16,335 INFO
org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
hdfs://bipcluster/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock,
Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@1dd9cf6d[Write
locks = 0, Read locks = 0], Thread meta-event-handle, In-process lock state
ALREADY_RELEASED
2026-07-18 01:05:16,335 INFO
org.apache.hudi.client.transaction.lock.LockManager [] - Released
connection created for acquiring lock
2026-07-18 01:05:16,336 INFO
org.apache.hudi.client.transaction.TransactionManager [] - State change
ended for action instant
Option{val=[==>20260718010427669__deltacommit__INFLIGHT]}
2026-07-18 01:05:16,336 INFO
org.apache.hudi.client.heartbeat.HoodieHeartbeatClient [] - Stopping
heartbeat for instant 20260718010427669
2026-07-18 01:05:16,337 INFO
org.apache.hudi.client.heartbeat.HoodieHeartbeatClient [] - Stopped
heartbeat for instant 20260718010427669
2026-07-18 01:05:16,339 INFO
org.apache.hudi.client.heartbeat.HeartbeatUtils [] - Deleted the
heartbeat for instant 20260718010427669
2026-07-18 01:05:16,339 INFO
org.apache.hudi.client.heartbeat.HoodieHeartbeatClient [] - Deleted
heartbeat file for instant 20260718010427669
2026-07-18 01:05:16,340 ERROR
org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor
executes action [commits the instant 20260718010427669] error
org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType
UPDATE for partition :files
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.handleUpsertPartition(BaseFlinkCommitActionExecutor.java:186)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:121)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:105)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:100)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.delta.FlinkUpsertPreppedDeltaCommitActionExecutor.execute(FlinkUpsertPreppedDeltaCommitActionExecutor.java:54)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.HoodieFlinkMergeOnReadTable.upsertPrepped(HoodieFlinkMergeOnReadTable.java:87)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.lambda$upsertPreppedRecords$2(HoodieFlinkWriteClient.java:171)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_281]
at
java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1628)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_281]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
~[?:1.8.0_281]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
~[?:1.8.0_281]
at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
~[?:1.8.0_281]
at
java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
~[?:1.8.0_281]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
~[?:1.8.0_281]
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
~[?:1.8.0_281]
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
~[?:1.8.0_281]
at
java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[?:1.8.0_281]
at
org.apache.hudi.client.HoodieFlinkWriteClient.upsertPreppedRecords(HoodieFlinkWriteClient.java:174)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.upsertPreppedRecords(HoodieFlinkWriteClient.java:74)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.upsertAndCommit(FlinkHoodieBackedTableMetadataWriter.java:161)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.upsertAndCommit(FlinkHoodieBackedTableMetadataWriter.java:54)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.commitInternal(HoodieBackedTableMetadataWriter.java:1819)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.commitInternal(FlinkHoodieBackedTableMetadataWriter.java:128)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.commit(FlinkHoodieBackedTableMetadataWriter.java:104)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:1167)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:1399)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieClient.writeTableMetadata(BaseHoodieClient.java:285)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.writeToMetadataTable(BaseHoodieWriteClient.java:339)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:320)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:107)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:617)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:602)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$commitInstants$10(StreamWriteOperatorCoordinator.java:557)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
~[?:1.8.0_281]
at
java.util.concurrent.ConcurrentSkipListMap$EntrySpliterator.forEachRemaining(ConcurrentSkipListMap.java:3535)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_281]
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[?:1.8.0_281]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstants(StreamWriteOperatorCoordinator.java:557)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$3(StreamWriteOperatorCoordinator.java:299)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:131)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_281]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_281]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to close
UpdateHandle
at
org.apache.hudi.io.HoodieAppendHandle.close(HoodieAppendHandle.java:581)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.io.FlinkAppendHandle.close(FlinkAppendHandle.java:106)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.delta.BaseFlinkDeltaCommitActionExecutor.handleUpdate(BaseFlinkDeltaCommitActionExecutor.java:57)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.handleUpsertPartition(BaseFlinkCommitActionExecutor.java:178)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
... 54 more
Caused by: org.apache.hadoop.ipc.RemoteException: File does not exist:
/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/files/.files-0000-0_20260718010427669.log.1_0-1-0
(inode 5373900553) Holder DFSClient_NONMAPREDUCE_217638024_99 does not have
any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2857)
at
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFileInternal(FSDirWriteFileOp.java:693)
at
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFile(FSDirWriteFileOp.java:679)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2900)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:959)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:614)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at org.apache.hadoop.ipc.Client.call(Client.java:1508)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at org.apache.hadoop.ipc.Client.call(Client.java:1405)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at com.sun.proxy.$Proxy41.complete(Unknown Source) ~[?:?]
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:563)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_281]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_281]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at com.sun.proxy.$Proxy42.complete(Unknown Source) ~[?:?]
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:975)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:932)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:896)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:844)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hudi.hadoop.fs.SizeAwareFSDataOutputStream.close(SizeAwareFSDataOutputStream.java:79)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hudi.common.table.log.HoodieLogFormatWriter.closeStream(HoodieLogFormatWriter.java:248)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.common.table.log.HoodieLogFormatWriter.close(HoodieLogFormatWriter.java:238)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.io.HoodieAppendHandle.close(HoodieAppendHandle.java:555)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.io.FlinkAppendHandle.close(FlinkAppendHandle.java:106)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.delta.BaseFlinkDeltaCommitActionExecutor.handleUpdate(BaseFlinkDeltaCommitActionExecutor.java:57)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.handleUpsertPartition(BaseFlinkCommitActionExecutor.java:178)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
... 54 more
2026-07-18 01:05:16,367 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Trying to recover from a global failure.
org.apache.flink.util.FlinkException: Global failure triggered by
OperatorCoordinator for 'bucket_write:
yuqi_oldetl_test.t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock'
(operator b754eda8ef527d9494a0b43e9bf7abf3).
at
org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:624)
~[flink-dist-1.18.1.jar:1.18.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$start$0(StreamWriteOperatorCoordinator.java:230)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.utils.NonThrownExecutor.handleException(NonThrownExecutor.java:143)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:134)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_281]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_281]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_281]
Caused by: org.apache.hudi.exception.HoodieException: Executor executes
action [commits the instant 20260718010427669] error
... 6 more
Caused by: org.apache.hudi.exception.HoodieUpsertException: Error upserting
bucketType UPDATE for partition :files
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.handleUpsertPartition(BaseFlinkCommitActionExecutor.java:186)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:121)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:105)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:100)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.delta.FlinkUpsertPreppedDeltaCommitActionExecutor.execute(FlinkUpsertPreppedDeltaCommitActionExecutor.java:54)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.HoodieFlinkMergeOnReadTable.upsertPrepped(HoodieFlinkMergeOnReadTable.java:87)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.lambda$upsertPreppedRecords$2(HoodieFlinkWriteClient.java:171)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_281]
at
java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1628)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_281]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
~[?:1.8.0_281]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
~[?:1.8.0_281]
at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
~[?:1.8.0_281]
at
java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
~[?:1.8.0_281]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
~[?:1.8.0_281]
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
~[?:1.8.0_281]
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
~[?:1.8.0_281]
at
java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[?:1.8.0_281]
at
org.apache.hudi.client.HoodieFlinkWriteClient.upsertPreppedRecords(HoodieFlinkWriteClient.java:174)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.upsertPreppedRecords(HoodieFlinkWriteClient.java:74)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.upsertAndCommit(FlinkHoodieBackedTableMetadataWriter.java:161)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.upsertAndCommit(FlinkHoodieBackedTableMetadataWriter.java:54)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.commitInternal(HoodieBackedTableMetadataWriter.java:1819)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.commitInternal(FlinkHoodieBackedTableMetadataWriter.java:128)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.commit(FlinkHoodieBackedTableMetadataWriter.java:104)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:1167)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:1399)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieClient.writeTableMetadata(BaseHoodieClient.java:285)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.writeToMetadataTable(BaseHoodieWriteClient.java:339)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:320)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:107)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:617)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:602)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$commitInstants$10(StreamWriteOperatorCoordinator.java:557)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
~[?:1.8.0_281]
at
java.util.concurrent.ConcurrentSkipListMap$EntrySpliterator.forEachRemaining(ConcurrentSkipListMap.java:3535)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_281]
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[?:1.8.0_281]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstants(StreamWriteOperatorCoordinator.java:557)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$3(StreamWriteOperatorCoordinator.java:299)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:131)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
... 3 more
Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to close
UpdateHandle
at
org.apache.hudi.io.HoodieAppendHandle.close(HoodieAppendHandle.java:581)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.io.FlinkAppendHandle.close(FlinkAppendHandle.java:106)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.delta.BaseFlinkDeltaCommitActionExecutor.handleUpdate(BaseFlinkDeltaCommitActionExecutor.java:57)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.handleUpsertPartition(BaseFlinkCommitActionExecutor.java:178)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:121)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:105)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:100)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.delta.FlinkUpsertPreppedDeltaCommitActionExecutor.execute(FlinkUpsertPreppedDeltaCommitActionExecutor.java:54)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.HoodieFlinkMergeOnReadTable.upsertPrepped(HoodieFlinkMergeOnReadTable.java:87)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.lambda$upsertPreppedRecords$2(HoodieFlinkWriteClient.java:171)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_281]
at
java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1628)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_281]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
~[?:1.8.0_281]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
~[?:1.8.0_281]
at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
~[?:1.8.0_281]
at
java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
~[?:1.8.0_281]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
~[?:1.8.0_281]
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
~[?:1.8.0_281]
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
~[?:1.8.0_281]
at
java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[?:1.8.0_281]
at
org.apache.hudi.client.HoodieFlinkWriteClient.upsertPreppedRecords(HoodieFlinkWriteClient.java:174)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.upsertPreppedRecords(HoodieFlinkWriteClient.java:74)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.upsertAndCommit(FlinkHoodieBackedTableMetadataWriter.java:161)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.upsertAndCommit(FlinkHoodieBackedTableMetadataWriter.java:54)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.commitInternal(HoodieBackedTableMetadataWriter.java:1819)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.commitInternal(FlinkHoodieBackedTableMetadataWriter.java:128)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.commit(FlinkHoodieBackedTableMetadataWriter.java:104)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:1167)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:1399)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieClient.writeTableMetadata(BaseHoodieClient.java:285)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.writeToMetadataTable(BaseHoodieWriteClient.java:339)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:320)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:107)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:617)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:602)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$commitInstants$10(StreamWriteOperatorCoordinator.java:557)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
~[?:1.8.0_281]
at
java.util.concurrent.ConcurrentSkipListMap$EntrySpliterator.forEachRemaining(ConcurrentSkipListMap.java:3535)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_281]
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[?:1.8.0_281]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstants(StreamWriteOperatorCoordinator.java:557)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$3(StreamWriteOperatorCoordinator.java:299)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:131)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
... 3 more
Caused by: org.apache.hadoop.ipc.RemoteException: File does not exist:
/user/hive/warehouse/yuqi_oldetl_test.db/t_info_instl_reservation_order_rtm11_chunguang_11_9_memlock/.hoodie/metadata/files/.files-0000-0_20260718010427669.log.1_0-1-0
(inode 5373900553) Holder DFSClient_NONMAPREDUCE_217638024_99 does not have
any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2857)
at
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFileInternal(FSDirWriteFileOp.java:693)
at
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFile(FSDirWriteFileOp.java:679)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2900)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:959)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:614)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at org.apache.hadoop.ipc.Client.call(Client.java:1508)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at org.apache.hadoop.ipc.Client.call(Client.java:1405)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at com.sun.proxy.$Proxy41.complete(Unknown Source) ~[?:?]
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:563)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_281]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_281]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at com.sun.proxy.$Proxy42.complete(Unknown Source) ~[?:?]
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:975)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:932)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:896)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:844)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hudi.hadoop.fs.SizeAwareFSDataOutputStream.close(SizeAwareFSDataOutputStream.java:79)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
~[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0]
at
org.apache.hudi.common.table.log.HoodieLogFormatWriter.closeStream(HoodieLogFormatWriter.java:248)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.common.table.log.HoodieLogFormatWriter.close(HoodieLogFormatWriter.java:238)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.io.HoodieAppendHandle.close(HoodieAppendHandle.java:555)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.io.FlinkAppendHandle.close(FlinkAppendHandle.java:106)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.delta.BaseFlinkDeltaCommitActionExecutor.handleUpdate(BaseFlinkDeltaCommitActionExecutor.java:57)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.handleUpsertPartition(BaseFlinkCommitActionExecutor.java:178)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:121)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:105)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:100)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.action.commit.delta.FlinkUpsertPreppedDeltaCommitActionExecutor.execute(FlinkUpsertPreppedDeltaCommitActionExecutor.java:54)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.table.HoodieFlinkMergeOnReadTable.upsertPrepped(HoodieFlinkMergeOnReadTable.java:87)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.lambda$upsertPreppedRecords$2(HoodieFlinkWriteClient.java:171)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_281]
at
java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1628)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_281]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
~[?:1.8.0_281]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
~[?:1.8.0_281]
at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
~[?:1.8.0_281]
at
java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
~[?:1.8.0_281]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
~[?:1.8.0_281]
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
~[?:1.8.0_281]
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
~[?:1.8.0_281]
at
java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[?:1.8.0_281]
at
org.apache.hudi.client.HoodieFlinkWriteClient.upsertPreppedRecords(HoodieFlinkWriteClient.java:174)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.upsertPreppedRecords(HoodieFlinkWriteClient.java:74)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.upsertAndCommit(FlinkHoodieBackedTableMetadataWriter.java:161)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.upsertAndCommit(FlinkHoodieBackedTableMetadataWriter.java:54)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.commitInternal(HoodieBackedTableMetadataWriter.java:1819)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.commitInternal(FlinkHoodieBackedTableMetadataWriter.java:128)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.commit(FlinkHoodieBackedTableMetadataWriter.java:104)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:1167)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:1399)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieClient.writeTableMetadata(BaseHoodieClient.java:285)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.writeToMetadataTable(BaseHoodieWriteClient.java:339)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:320)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:107)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:617)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:602)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$commitInstants$10(StreamWriteOperatorCoordinator.java:557)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
~[?:1.8.0_281]
at
java.util.concurrent.ConcurrentSkipListMap$EntrySpliterator.forEachRemaining(ConcurrentSkipListMap.java:3535)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_281]
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
~[?:1.8.0_281]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
~[?:1.8.0_281]
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[?:1.8.0_281]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstants(StreamWriteOperatorCoordinator.java:557)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$3(StreamWriteOperatorCoordinator.java:299)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:131)
~[hudi-flink1.18-bundle-1.1.1.jar:1.1.1]
... 3 more
2026-07-18 01:05:16,377 I
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
