wjcwin opened a new issue #4249:
URL: https://github.com/apache/hudi/issues/4249
FLINK CDC WRITE HUDI, restart job get
exception:org.apache.hudi.org.apache.avro.InvalidAvroMagicException: Not an
Avro data file
logs:
```
org.apache.flink.util.FlinkException: Global failure triggered by
OperatorCoordinator for 'hoodie_stream_write' (operator
f1d7c56f4bf5fc204e4401416e5b3884).
at
org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:557)
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$start$0(StreamWriteOperatorCoordinator.java:170)
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:103)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: Executor executes
action [initialize instant ] error
... 5 more
Caused by: org.apache.hudi.exception.HoodieIOException: Fetching rollback
plan failed for [==>20211208144119644__rollback__REQUESTED]
at
org.apache.hudi.client.AbstractHoodieWriteClient.lambda$getPendingRollbackInfos$8(AbstractHoodieWriteClient.java:883)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at
org.apache.hudi.client.AbstractHoodieWriteClient.getPendingRollbackInfos(AbstractHoodieWriteClient.java:886)
at
org.apache.hudi.client.AbstractHoodieWriteClient.rollbackFailedWrites(AbstractHoodieWriteClient.java:909)
at
org.apache.hudi.client.AbstractHoodieWriteClient.rollbackFailedWrites(AbstractHoodieWriteClient.java:899)
at
org.apache.hudi.client.AbstractHoodieWriteClient.lambda$startCommitWithTime$97cdbdca$1(AbstractHoodieWriteClient.java:785)
at
org.apache.hudi.common.util.CleanerUtils.rollbackFailedWrites(CleanerUtils.java:143)
at
org.apache.hudi.client.AbstractHoodieWriteClient.startCommitWithTime(AbstractHoodieWriteClient.java:784)
at
org.apache.hudi.client.AbstractHoodieWriteClient.startCommitWithTime(AbstractHoodieWriteClient.java:777)
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.startInstant(StreamWriteOperatorCoordinator.java:334)
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$initInstant$5(StreamWriteOperatorCoordinator.java:361)
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:93)
... 3 more
Caused by: org.apache.hudi.org.apache.avro.InvalidAvroMagicException: Not an
Avro data file
at
org.apache.hudi.org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:56)
at
org.apache.hudi.common.table.timeline.TimelineMetadataUtils.deserializeAvroMetadata(TimelineMetadataUtils.java:183)
at
org.apache.hudi.table.action.rollback.RollbackUtils.getRollbackPlan(RollbackUtils.java:68)
at
org.apache.hudi.client.AbstractHoodieWriteClient.lambda$getPendingRollbackInfos$8(AbstractHoodieWriteClient.java:879)
... 20 more
```
it seems like some 0 byte files caused , i try delete this files, become
normal , then i cancel , submit job , this problem appear again
can you help me . thks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
