Lee-ship316 opened a new issue, #12554:
URL: https://github.com/apache/hudi/issues/12554
操作步骤:
1.Flink 批处理将Hive历史数据迁移到Hudi表中；
2.Flink 接入Kafka数据写入HDFS；
3.Hudi 采用的是离线合并；
4.Spark每小时读取Hudi全量数据。
**出现的问题**
Caused by: org.apache.hudi.exception.HoodieIOException: Exception create
input stream from file:
HoodieLogFile{pathStr='hdfs://nameservice1/xxx/.00000056-15ec-459f-bb67-5f8c2b319203_20241228152030542.log.1_1-5-1',
fileLen=-1}
**运行环境**
* Hudi version : 0.14.1
* Spark version : 3.3.2
* Hive version : 11.0-cdh5.13.2
* Hadoop version : 2.6.0-cdh5.13.2
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) : no
**具体的报错**
2024-12-28 16:30:28,806 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Failed to
trigger checkpoint for job dbb88271f16bc32329acd07b5da75321 because Some tasks
of the job have already finished and checkpointing with finished tasks is not
enabled. Failure reason: Not all required tasks are currently running.
2024-12-28 16:30:37,983 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_task
(38/60) (8d6d4c0f31ae7ce0af767c5dd919b407) switched from RUNNING to FAILED on
container_1724338424009_17149855_01_011191 @ bdc-dn3077.bigdata.jxtelecom.cn
(dataPort=38642).
org.apache.hudi.exception.HoodieException: Exception when reading log file
at
org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternalV1(AbstractHoodieLogRecordReader.java:414)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:220)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.performScan(HoodieMergedLogRecordScanner.java:201)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.<init>(HoodieMergedLogRecordScanner.java:117)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.<init>(HoodieMergedLogRecordScanner.java:76)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner$Builder.build(HoodieMergedLogRecordScanner.java:466)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:210)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:147)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.sink.compact.CompactOperator.doCompaction(CompactOperator.java:142)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.sink.compact.CompactOperator.processElement(CompactOperator.java:130)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:233)
~[flink-dist_2.11-1.14.2.jar:1.14.2]
at
org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
~[flink-dist_2.11-1.14.2.jar:1.14.2]
at
org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
~[flink-dist_2.11-1.14.2.jar:1.14.2]
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
~[flink-dist_2.11-1.14.2.jar:1.14.2]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:496)
~[flink-dist_2.11-1.14.2.jar:1.14.2]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
~[flink-dist_2.11-1.14.2.jar:1.14.2]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
~[flink-dist_2.11-1.14.2.jar:1.14.2]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
~[flink-dist_2.11-1.14.2.jar:1.14.2]
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
~[flink-dist_2.11-1.14.2.jar:1.14.2]
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
~[flink-dist_2.11-1.14.2.jar:1.14.2]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_111]
Caused by: org.apache.hudi.exception.HoodieIOException: Exception create
input stream from file:
HoodieLogFile{pathStr='hdfs://bdccluster-1/telecom_data/CORE_MODEL/IP/BSS/DAY/INF_SS_OFFER_INST_D_HUDI_ALL_FLINK2/.00000056-15ec-459f-bb67-5f8c2b319203_20241228152030542.log.1_1-5-1',
fileLen=-1}
at
org.apache.hudi.common.table.log.HoodieLogFileReader.getFSDataInputStream(HoodieLogFileReader.java:486)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.common.table.log.HoodieLogFileReader.<init>(HoodieLogFileReader.java:121)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.common.table.log.HoodieLogFormatReader.<init>(HoodieLogFormatReader.java:64)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternalV1(AbstractHoodieLogRecordReader.java:245)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
... 22 more
Caused by: org.apache.hadoop.ipc.RemoteException:
java.lang.ArrayIndexOutOfBoundsException
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.ipc.Client.call(Client.java:1413)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at com.sun.proxy.$Proxy35.getBlockLocations(Unknown Source) ~[?:?]
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_111]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_111]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at com.sun.proxy.$Proxy36.getBlockLocations(Unknown Source) ~[?:?]
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1235)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1222)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:306)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:272)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at
org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:264)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1535)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312)
~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at
org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:203)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.common.table.log.HoodieLogFileReader.getFSDataInputStream(HoodieLogFileReader.java:484)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.common.table.log.HoodieLogFileReader.<init>(HoodieLogFileReader.java:121)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.common.table.log.HoodieLogFormatReader.<init>(HoodieLogFormatReader.java:64)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at
org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternalV1(AbstractHoodieLogRecordReader.java:245)
~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
... 22 more
2024-12-28 16:30:37,984 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager []
- Received resource requirements from job dbb88271f16bc32329acd07b5da75321:
[ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
numberOfRequiredSlots=59}]
2024-12-28 16:30:37,984 INFO
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
[] - Calculating tasks to restart to recover the failed task
0d75f8675b58c9bef24c60fb5f5d0a16_37.
2024-12-28 16:30:37,984 INFO
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
[] - 62 tasks should be restarted to recover the failed task
0d75f8675b58c9bef24c60fb5f5d0a16_37.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
