BalaMahesh opened a new issue, #5591: URL: https://github.com/apache/hudi/issues/5591
**_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at [email protected]. - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly. **Describe the problem you faced** A clear and concise description of the problem. We have recently upgraded hudi to 0.11.0 and application has been running good for last few days. Then all of sudden due to some reason, marker files in .hoodie/.temp directory are not created with .marker extension and the application while trying to roll back this commit, validation of marker files is failing and application is getting stopped. I have attached the screenshot of marker files created inside the folder. Steps to reproduce the behavior: 1. Start the application and keep it running. 2. We don't know exactly what has stopped the application, but it started trying to rollback the failed commit and it is failing to do that. **Expected behavior** Run and rollback the failed commits if application has failed. **Environment Description** * Hudi version : 0.11.0 * Spark version : 3.2.1 * Hive version : 2.1.3 * Hadoop version : 3.x.x * Storage (HDFS/S3/GCS..) : GCS * Running on Docker? (yes/no) : yes(k8's). **Additional context** **Stacktrace** ``` Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRollbackException: Failed to rollback gs://xxx/hudi/xxxx/ commits 20220514180424825 at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source) at org.apache.hudi.async.HoodieAsyncService.waitForShutdown(HoodieAsyncService.java:103) at org.apache.hudi.async.AsyncCleanerService.waitForCompletion(AsyncCleanerService.java:75) ... 11 more Caused by: org.apache.hudi.exception.HoodieRollbackException: Failed to rollback gs://xxx/hudi/xxx/ commits 20220514180424825 at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:783) at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:1193) at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:1176) at org.apache.hudi.client.BaseHoodieWriteClient.lambda$clean$33796fd2$1(BaseHoodieWriteClient.java:856) at org.apache.hudi.common.util.CleanerUtils.rollbackFailedWrites(CleanerUtils.java:142) at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:855) at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:825) at org.apache.hudi.async.AsyncCleanerService.lambda$startService$0(AsyncCleanerService.java:55) ... 4 more Caused by: org.apache.hudi.exception.HoodieRollbackException: Error rolling back using marker files written for [==>20220514180424825__commit__INFLIGHT] at org.apache.hudi.table.action.rollback.MarkerBasedRollbackStrategy.getRollbackRequests(MarkerBasedRollbackStrategy.java:103) at org.apache.hudi.table.action.rollback.BaseRollbackPlanActionExecutor.requestRollback(BaseRollbackPlanActionExecutor.java:109) at org.apache.hudi.table.action.rollback.BaseRollbackPlanActionExecutor.execute(BaseRollbackPlanActionExecutor.java:132) at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.scheduleRollback(HoodieSparkCopyOnWriteTable.java:212) at org.apache.hudi.client.BaseHoodieWriteClient.lambda$rollback$6(BaseHoodieWriteClient.java:757) at org.apache.hudi.common.util.Option.orElseGet(Option.java:142) at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:757) ... 11 more Caused by: java.lang.IllegalArgumentException at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:31) at org.apache.hudi.common.util.MarkerUtils.stripMarkerFolderPrefix(MarkerUtils.java:67) at org.apache.hudi.table.marker.DirectWriteMarkers.lambda$allMarkerFilePaths$0(DirectWriteMarkers.java:136) at org.apache.hudi.common.fs.FSUtils.processFiles(FSUtils.java:277) at org.apache.hudi.table.marker.DirectWriteMarkers.allMarkerFilePaths(DirectWriteMarkers.java:135) at org.apache.hudi.table.marker.MarkerBasedRollbackUtils.getAllMarkerPaths(MarkerBasedRollbackUtils.java:62) at org.apache.hudi.table.action.rollback.MarkerBasedRollbackStrategy.getRollbackRequests(MarkerBasedRollbackStrategy.java:76) ... 17 more ``` <img width="1360" alt="Screenshot 2022-05-16 at 11 01 13 AM" src="https://user-images.githubusercontent.com/25053668/168527450-2acf3a5e-a34b-49ca-a6be-d5fa78d69d09.png";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/25053668/168527450-2acf3a5e-a34b-49ca-a6be-d5fa78d69d09.png"](https://user-images.githubusercontent.com/25053668/168527450-2acf3a5e-a34b-49ca-a6be-d5fa78d69d09.png"){kind=link}