prashanthvg89 opened a new issue #2153: URL: https://github.com/apache/hudi/issues/2153
**_Tips before filing an issue_** - Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)? - Join the mailing list to engage in conversations and get faster support at [email protected]. - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly. **Describe the problem you faced** Random "Failed to delete key" error during UPSERT operation in a Spark Structured Streaming job even with "hoodie.consistency.check.enabled" set to true **To Reproduce** Steps to reproduce the behavior: UNKOWN - Occurs randomly after running streaming job - sometimes for 15 hours and other times for about couple days **Expected behavior** UPSERT should be a simple operation and if there is an application bug then we should have faced this as soon as I launched the application but it appears intermittently. The only resolution so far is to restart the job **Environment Description** * Hudi version : 0.5.2-incubating * Spark version : 2.4.4 * Hive version : 2.3.6 * Hadoop version : EMR 5.29.0 * Storage (HDFS/S3/GCS..) : S3 * Running on Docker? (yes/no) : no **Additional context** This is a simple streaming application which listens to Kinesis stream with a batch interval of 15 minutes and updates the Hudi table using MERGE_ON_READ **Stacktrace** ```Caused by: org.apache.hudi.exception.HoodieIOException: Failed to delete key: <tableName>/.hoodie/.temp/20201006182950 at org.apache.hudi.table.HoodieTable.deleteMarkerDir(HoodieTable.java:333) at org.apache.hudi.table.HoodieTable.cleanFailedWrites(HoodieTable.java:409) at org.apache.hudi.table.HoodieTable.finalizeWrite(HoodieTable.java:315) at org.apache.hudi.table.HoodieMergeOnReadTable.finalizeWrite(HoodieMergeOnReadTable.java:317) at org.apache.hudi.client.AbstractHoodieWriteClient.finalizeWrite(AbstractHoodieWriteClient.java:195) ... 66 more Caused by: java.io.IOException: Failed to delete key: <tableName>/.hoodie/.temp/20201006182950 at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.delete(S3NativeFileSystem.java:767) at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.delete(EmrFileSystem.java:337) at org.apache.hudi.common.io.storage.HoodieWrapperFileSystem.delete(HoodieWrapperFileSystem.java:261) at org.apache.hudi.table.HoodieTable.deleteMarkerDir(HoodieTable.java:330) ... 70 more Caused by: java.io.IOException: 1 exceptions thrown from 5 batch deletes at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.deleteAll(Jets3tNativeFileSystemStore.java:390) at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.doSingleThreadedBatchDelete(S3NativeFileSystem.java:1494) at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.delete(S3NativeFileSystem.java:765) ... 73 more Caused by: java.io.IOException: MultiObjectDeleteException thrown with 2 keys in error: <tableName>/.hoodie/.temp/20201006182950/195/2ecca5ce-ba13-4d5a-a2e3-79713261dc49-0_2061-53-37527_20201006182950.marker, <tableName>/.hoodie/.temp/20201006182950/290_$folder$ at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.deleteAll(Jets3tNativeFileSystemStore.java:375) ... 75 more Caused by: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 457C53995454141D; S3 Extended Request ID: NKQEApW06BHPRG5oQBP4RffTd6OZQXOCNl6jurU690Ee+iE3cgbRbbtNPugjqa3qyADj6x5zqBk=), S3 Extended Request ID: NKQEApW06BHPRG5oQBP4RffTd6OZQXOCNl6jurU690Ee+iE3cgbRbbtNPugjqa3qyADj6x5zqBk= at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2267) at com.amazon.ws.emr.hadoop.fs.s3.lite.call.DeleteObjectsCall.perform(DeleteObjectsCall.java:24) at com.amazon.ws.emr.hadoop.fs.s3.lite.call.DeleteObjectsCall.perform(DeleteObjectsCall.java:10) at com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.execute(GlobalS3Executor.java:110) at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:189) at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:184) at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.deleteObjects(AmazonS3LiteClient.java:128) at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.deleteAll(Jets3tNativeFileSystemStore.java:370)``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
