abhijeetkushe opened a new issue #2533: URL: https://github.com/apache/hudi/issues/2533
I am no longer able to write to a hudi table because of the error mentioned in the subject.This happened probably because 2 simultaneous EMRs were running at the sametime.I did terminate one EMR but I keep getting the above error.I also tried to running hudi-cli clean and rollback operations based on https://github.com/apache/hudi/issues/1785 but those are failing as well,I have added the stacktrace and commit details.Let me know if anything more is needed **Environment Description** * Hudi version : 0.6.0 * Spark version : 2.4.4 * Hive version : 2.3.7 * Hadoop version : 2.10.0 * Storage (HDFS/S3/GCS..) : S3 * Running on Docker? (yes/no) : no Stacktrace: 2021-02-01T19:13:38.849+0000: 84.629: Application time: 0.0001674 seconds 2021-02-01T19:13:38.849+0000: 84.630: Total time for which application threads were stopped: 0.0001441 seconds, Stopping threads took: 0.0000495 seconds 2021-02-01 19:13:38 ERROR AbstractAsyncService:149 - Monitor noticed one or more threads failed. Requesting graceful shutdown of other threads java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException: Failed to rollback s3://akctct-transient-dev-us-east-1-eigi-datalake/hudi_deltastreamer_large_100/engmt_open_cow_1 commits 20210129225129 at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) at org.apache.hudi.async.AbstractAsyncService.lambda$monitorThreads$1(AbstractAsyncService.java:147) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hudi.exception.HoodieException: Failed to rollback s3://akctct-transient-dev-us-east-1-eigi-datalake/hudi_deltastreamer_large_100/engmt_open_cow_1 commits 20210129225129 at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:595) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ... 3 more Caused by: org.apache.hudi.exception.HoodieRollbackException: Failed to rollback s3://akctct-transient-dev-us-east-1-eigi-datalake/hudi_deltastreamer_large_100/engmt_open_cow_1 commits 20210129225129 at org.apache.hudi.client.HoodieWriteClient.rollback(HoodieWriteClient.java:505) at org.apache.hudi.client.HoodieWriteClient.rollbackPendingCommits(HoodieWriteClient.java:708) at org.apache.hudi.client.HoodieWriteClient.startCommit(HoodieWriteClient.java:576) at org.apache.hudi.utilities.deltastreamer.DeltaSync.startCommit(DeltaSync.java:465) at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:381) at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:245) at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:579) ... 4 more Caused by: org.apache.hudi.exception.HoodieRollbackException: Found commits after time :20210129225129, please rollback greater commits first at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.validateRollbackCommitSequence(BaseRollbackActionExecutor.java:145) at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.doRollbackAndGetStats(BaseRollbackActionExecutor.java:171) at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.execute(BaseRollbackActionExecutor.java:108) at org.apache.hudi.table.HoodieCopyOnWriteTable.rollback(HoodieCopyOnWriteTable.java:193) at org.apache.hudi.client.HoodieWriteClient.rollback(HoodieWriteClient.java:494) ... 10 more Commit Loaded instants [[20210129211107__commit__COMPLETED], [20210129211434__clean__COMPLETED], [20210129211434__commit__COMPLETED], [20210129211810__clean__COMPLETED], [20210129211810__commit__COMPLETED], [20210129212234__clean__COMPLETED], [20210129212234__commit__COMPLETED], [20210129212622__clean__COMPLETED], [20210129212622__commit__COMPLETED], [20210129212913__clean__COMPLETED], [20210129212913__commit__COMPLETED], [20210129213326__clean__COMPLETED], [20210129213326__commit__COMPLETED], [20210129213638__clean__COMPLETED], [20210129213638__commit__COMPLETED], [20210129214016__clean__COMPLETED], [20210129214016__commit__COMPLETED], [20210129214432__clean__COMPLETED], [20210129214432__commit__COMPLETED], [20210129214813__clean__COMPLETED], [20210129214813__commit__COMPLETED], [20210129215121__clean__COMPLETED], [20210129215121__commit__COMPLETED], [20210129215538__clean__COMPLETED], [20210129215538__commit__COMPLETED], [20210129215816__clean__COMPLETED], [20210129215816__commit__COMP LETED], [20210129220159__clean__COMPLETED], [20210129220159__commit__COMPLETED], [20210129220532__clean__COMPLETED], [20210129220532__commit__COMPLETED], [20210129220839__clean__COMPLETED], [20210129220839__commit__COMPLETED], [20210129221301__clean__COMPLETED], [20210129221301__commit__COMPLETED], [20210129221623__clean__COMPLETED], [20210129221623__commit__COMPLETED], [20210129221930__clean__COMPLETED], [20210129221930__commit__COMPLETED], [20210129222350__clean__COMPLETED], [20210129222350__commit__COMPLETED], [20210129222736__clean__COMPLETED], [20210129222736__commit__COMPLETED], [20210129223034__clean__COMPLETED], [20210129223034__commit__COMPLETED], [20210129223422__clean__COMPLETED], [20210129223422__commit__COMPLETED], [20210129223818__clean__COMPLETED], [20210129223818__commit__COMPLETED], [20210129224110__clean__COMPLETED], [20210129224110__commit__COMPLETED], [20210129224446__clean__COMPLETED], [20210129224446__commit__COMPLETED], [20210129224849__clean__COMPLETED], [202 10129224849__commit__COMPLETED], [20210129225129__clean__COMPLETED], [==>20210129225129__commit__INFLIGHT], [20210129225133__commit__COMPLETED], [20210201183500__clean__COMPLETED]] ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
