[
https://issues.apache.org/jira/browse/HUDI-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
konwu updated HUDI-5025:
------------------------
Description:
Currently when rollOver happened in rollback was not create new log file ,thus
caused below exception
2022-10-13 16:58:54,613 INFO
org.apache.hudi.common.table.log.HoodieLogFormatWriter [] -
HoodieLogFile\{pathStr='viewfs://dcfs/ns-common/car/dws/dws_order_info_by_flinkbatch_history/2022-10-12/.00000002-1251-4f1c-8f75-71ff51071ee3_20221013052439696.log.1_2-4-0',
fileLen=0} exists. Appending to existing file
2022-10-13 16:58:54,974 INFO
org.apache.hudi.table.action.rollback.BaseRollbackHelper [] - after
testrollback writer.LogFile:
viewfs://dcfs/ns-common/car/dws/dws_order_info_by_flinkbatch_history/2022-10-12/.00000002-1251-4f1c-8f75-71ff51071ee3_20221013052439696.log.2_1-0-1
was:
Currently when rollOver happened in rollback was not create new log file ,thus
caused below exception
{code:java}
org.apache.hudi.exception.HoodieRollbackException: Failed to rollback
viewfs://dcfs/ns-common/car/dws/dws_order_info_by_flinkbatch_history commits
20221013092938828 at
org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:823)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:1282)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:1265)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:1253)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.client.BaseHoodieWriteClient.lambda$startCommit$afea71c0$1(BaseHoodieWriteClient.java:976)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.common.util.CleanerUtils.rollbackFailedWrites(CleanerUtils.java:153)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.client.BaseHoodieWriteClient.startCommit(BaseHoodieWriteClient.java:975)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.startInstant(StreamWriteOperatorCoordinator.java:374)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$initInstant$6(StreamWriteOperatorCoordinator.java:402)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_191] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_191] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]Caused
by: org.apache.hudi.exception.HoodieException:
org.apache.hudi.exception.HoodieException: Error occurs when executing flatMap
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
~[?:1.8.0_191] at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
~[?:1.8.0_191] at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
~[?:1.8.0_191] at
java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
at
java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:593)
~[?:1.8.0_191] at
java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
~[?:1.8.0_191] at
java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735) ~[?:1.8.0_191]
at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
~[?:1.8.0_191] at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
~[?:1.8.0_191] at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[?:1.8.0_191] at
org.apache.hudi.client.common.HoodieFlinkEngineContext.flatMap(HoodieFlinkEngineContext.java:135)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.table.action.rollback.BaseRollbackHelper.maybeDeleteAndCollectStats(BaseRollbackHelper.java:114)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.table.action.rollback.BaseRollbackHelper.performRollback(BaseRollbackHelper.java:81)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.executeRollback(BaseRollbackActionExecutor.java:232)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.table.action.rollback.MergeOnReadRollbackActionExecutor.executeRollback(MergeOnReadRollbackActionExecutor.java:89)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.doRollbackAndGetStats(BaseRollbackActionExecutor.java:214)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.runRollback(BaseRollbackActionExecutor.java:108)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.execute(BaseRollbackActionExecutor.java:135)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.table.HoodieFlinkMergeOnReadTable.rollback(HoodieFlinkMergeOnReadTable.java:135)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:806)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] ... 12
moreCaused by: org.apache.hudi.exception.HoodieException: Error occurs when
executing flatMap at
org.apache.hudi.common.function.FunctionWrapper.lambda$throwingFlatMapWrapper$1(FunctionWrapper.java:50)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
~[?:1.8.0_191] at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
~[?:1.8.0_191] at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
~[?:1.8.0_191] at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
~[?:1.8.0_191] at
java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747) ~[?:1.8.0_191]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
~[?:1.8.0_191] at
java.util.stream.AbstractTask.compute(AbstractTask.java:316) ~[?:1.8.0_191]
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
~[?:1.8.0_191] at
java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_191]
at
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
~[?:1.8.0_191] at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
~[?:1.8.0_191] at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
~[?:1.8.0_191]Caused by: java.io.FileNotFoundException: File does not exist:
/ns-common/car/dws/dws_order_info_by_flinkbatch_history/2022-10-12/.00000002-1251-4f1c-8f75-71ff51071ee3_20221013052439696.log.2_1-0-1
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
~[hadoop-hdfs-2.7.7.jar:?] at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
~[hadoop-hdfs-2.7.7.jar:?] at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
~[hadoop-common-2.7.7.jar:?] at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
~[hadoop-hdfs-2.7.7.jar:?] at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:428)
~[hadoop-common-2.7.7.jar:?] at
org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getFileStatus(ChRootedFileSystem.java:226)
~[hadoop-common-2.7.7.jar:?] at
org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:383)
~[hadoop-common-2.7.7.jar:?] at
org.apache.hudi.common.fs.HoodieWrapperFileSystem.lambda$getFileStatus$17(HoodieWrapperFileSystem.java:402)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.common.fs.HoodieWrapperFileSystem.executeFuncWithTimeMetrics(HoodieWrapperFileSystem.java:106)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.common.fs.HoodieWrapperFileSystem.getFileStatus(HoodieWrapperFileSystem.java:396)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.table.action.rollback.BaseRollbackHelper.lambda$maybeDeleteAndCollectStats$309309f3$1(BaseRollbackHelper.java:157)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
org.apache.hudi.common.function.FunctionWrapper.lambda$throwingFlatMapWrapper$1(FunctionWrapper.java:48)
~[hudi-flink1.14-bundle-0.13.0-SNAPSHOT.jar:0.13.0-SNAPSHOT] at
java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
~[?:1.8.0_191] at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
~[?:1.8.0_191] at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
~[?:1.8.0_191] at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
~[?:1.8.0_191] at
java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747) ~[?:1.8.0_191]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
~[?:1.8.0_191] at
java.util.stream.AbstractTask.compute(AbstractTask.java:316) ~[?:1.8.0_191]
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
~[?:1.8.0_191] at
java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_191]
at
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
~[?:1.8.0_191] at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
~[?:1.8.0_191] at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
~[?:1.8.0_191] {code}
> Rollback failed with log file not found when rollOver in rollback process
> -------------------------------------------------------------------------
>
> Key: HUDI-5025
> URL: https://issues.apache.org/jira/browse/HUDI-5025
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: konwu
> Priority: Major
> Attachments: image-2022-10-13-18-15-57-023.png
>
>
> Currently when rollOver happened in rollback was not create new log file
> ,thus caused below exception
> 2022-10-13 16:58:54,613 INFO
> org.apache.hudi.common.table.log.HoodieLogFormatWriter [] -
> HoodieLogFile\{pathStr='viewfs://dcfs/ns-common/car/dws/dws_order_info_by_flinkbatch_history/2022-10-12/.00000002-1251-4f1c-8f75-71ff51071ee3_20221013052439696.log.1_2-4-0',
> fileLen=0} exists. Appending to existing file
> 2022-10-13 16:58:54,974 INFO
> org.apache.hudi.table.action.rollback.BaseRollbackHelper [] - after
> testrollback writer.LogFile:
> viewfs://dcfs/ns-common/car/dws/dws_order_info_by_flinkbatch_history/2022-10-12/.00000002-1251-4f1c-8f75-71ff51071ee3_20221013052439696.log.2_1-0-1
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
