waywtdcc created HUDI-3005:
------------------------------
Summary: flink write hudi throw Not an Avro data file exception
Key: HUDI-3005
URL: https://issues.apache.org/jira/browse/HUDI-3005
Project: Apache Hudi
Issue Type: Bug
Components: Flink Integration
Reporter: waywtdcc
Fix For: 0.9.0
An error is reported when the Flink is written.
*error:*
{code:java}
2021-12-14 10:18:53,554 INFO
org.apache.hudi.common.table.log.HoodieLogFormatWriter [] -
HoodieLogFile{pathStr='hdfs:/user/hive/warehouse/hudi.db/datagen_hudi_test2/.hoodie/.commits_.archive.1_1-0-1',
fileLen=0} exists. Appending to existing file
2021-12-14 10:18:53,880 ERROR org.apache.hudi.table.HoodieTimelineArchiveLog
[] - Failed to archive commits, .commit file: 20211210103323.rollback
org.apache.hudi.org.apache.avro.InvalidAvroMagicException: Not an Avro data file
at
org.apache.hudi.org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:56)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.common.table.timeline.TimelineMetadataUtils.deserializeAvroMetadata(TimelineMetadataUtils.java:178)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.client.utils.MetadataConversionUtils.createMetaWrapper(MetadataConversionUtils.java:103)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.table.HoodieTimelineArchiveLog.convertToAvroRecord(HoodieTimelineArchiveLog.java:341)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.table.HoodieTimelineArchiveLog.archive(HoodieTimelineArchiveLog.java:305)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.table.HoodieTimelineArchiveLog.archiveIfRequired(HoodieTimelineArchiveLog.java:128)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.client.HoodieFlinkWriteClient.postCommit(HoodieFlinkWriteClient.java:323)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:191)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:120)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:462)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:438)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$1(StreamWriteOperatorCoordinator.java:207)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:67)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_221]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_221]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]
2021-12-14 10:18:53,895 ERROR
org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor
executes action [commits the instant 20211214101738] error
org.apache.hudi.exception.HoodieCommitException: Failed to archive commits
at
org.apache.hudi.table.HoodieTimelineArchiveLog.archive(HoodieTimelineArchiveLog.java:318)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.table.HoodieTimelineArchiveLog.archiveIfRequired(HoodieTimelineArchiveLog.java:128)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.client.HoodieFlinkWriteClient.postCommit(HoodieFlinkWriteClient.java:323)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:191)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:120)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:462)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:438)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$1(StreamWriteOperatorCoordinator.java:207)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:67)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_221]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_221]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]
Caused by: org.apache.hudi.org.apache.avro.InvalidAvroMagicException: Not an
Avro data file
at
org.apache.hudi.org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:56)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.common.table.timeline.TimelineMetadataUtils.deserializeAvroMetadata(TimelineMetadataUtils.java:178)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.client.utils.MetadataConversionUtils.createMetaWrapper(MetadataConversionUtils.java:103)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.table.HoodieTimelineArchiveLog.convertToAvroRecord(HoodieTimelineArchiveLog.java:341)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.table.HoodieTimelineArchiveLog.archive(HoodieTimelineArchiveLog.java:305)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
... 11 more
2021-12-14 10:18:53,896 INFO
org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor
executes action [taking checkpoint 2] success!
2021-12-14 10:18:53,912 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Trying to recover from a global failure.
org.apache.flink.util.FlinkException: Global failure triggered by
OperatorCoordinator for 'hoodie_stream_write' (operator
37faf88697f0a69c783562897fa7eaeb).
at
org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:492)
~[flink-dist_2.11-1.12.2.jar:1.12.2]
at
org.apache.hudi.sink.utils.CoordinatorExecutor.exceptionHook(CoordinatorExecutor.java:44)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:76)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_221]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_221]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_221]
Caused by: org.apache.hudi.exception.HoodieException: Executor executes action
[commits the instant 20211214101738] error
... 5 more
Caused by: org.apache.hudi.exception.HoodieCommitException: Failed to archive
commits
at
org.apache.hudi.table.HoodieTimelineArchiveLog.archive(HoodieTimelineArchiveLog.java:318)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.table.HoodieTimelineArchiveLog.archiveIfRequired(HoodieTimelineArchiveLog.java:128)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.client.HoodieFlinkWriteClient.postCommit(HoodieFlinkWriteClient.java:323)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:191)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:120)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:462)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:438)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$1(StreamWriteOperatorCoordinator.java:207)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:67)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
... 3 more
Caused by: org.apache.hudi.org.apache.avro.InvalidAvroMagicException: Not an
Avro data file
at
org.apache.hudi.org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:56)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.common.table.timeline.TimelineMetadataUtils.deserializeAvroMetadata(TimelineMetadataUtils.java:178)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.client.utils.MetadataConversionUtils.createMetaWrapper(MetadataConversionUtils.java:103)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.table.HoodieTimelineArchiveLog.convertToAvroRecord(HoodieTimelineArchiveLog.java:341)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.table.HoodieTimelineArchiveLog.archive(HoodieTimelineArchiveLog.java:305)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.table.HoodieTimelineArchiveLog.archiveIfRequired(HoodieTimelineArchiveLog.java:128)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.client.HoodieFlinkWriteClient.postCommit(HoodieFlinkWriteClient.java:323)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:191)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:120)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:462)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:438)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$1(StreamWriteOperatorCoordinator.java:207)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:67)
~[hudi-flink-bundle_2.11-0.9.0.jar:0.9.0]
... 3 more{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
