[
https://issues.apache.org/jira/browse/HUDI-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu updated HUDI-2648:
-----------------------------
Issue Type: Improvement (was: Task)
> Retry FileSystem action when caught runtime exception
> ------------------------------------------------------
>
> Key: HUDI-2648
> URL: https://issues.apache.org/jira/browse/HUDI-2648
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Yue Zhang
> Assignee: Yue Zhang
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.11.0
>
>
> Hoodie will do lots of list/get/put/delete etc actions on filesystem.
> Sometimes will meet the fileSystem performance issue or short service
> suspension, for example
> Using hoodie on S3, and will throw lots of exception like this
> {code:java}
> com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 503, AWS
> Service: Amazon S3, AWS Request ID: J58R9W4E324DWJ1Y, AWS Error Code:
> SlowDown, AWS Error Message: Please reduce your request rate., S3 Extended
> Request ID:
> Pkd6VCJKTuq3NJ6vUQJB1TUznKGvAfjIZkmJOxt+j6oAD5roz+ojKf9rtcBOgUBw4sWeDwxdTv4=com.amazonaws.services.s3.model.AmazonS3Exception:
> Status Code: 503, AWS Service: Amazon S3, AWS Request ID: J58R9W4E324DWJ1Y,
> AWS Error Code: SlowDown, AWS Error Message: Please reduce your request
> rate., S3 Extended Request ID:
> Pkd6VCJKTuq3NJ6vUQJB1TUznKGvAfjIZkmJOxt+j6oAD5roz+ojKf9rtcBOgUBw4sWeDwxdTv4=
> at
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:1191)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:1168)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:871) at
> org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1881) at
> org.apache.hudi.common.fs.HoodieWrapperFileSystem.lambda$mkdirs$25(HoodieWrapperFileSystem.java:632)
> at
> org.apache.hudi.common.fs.HoodieWrapperFileSystem.executeFuncWithTimeMetrics(HoodieWrapperFileSystem.java:100)
> at
> org.apache.hudi.common.fs.HoodieWrapperFileSystem.mkdirs(HoodieWrapperFileSystem.java:631)
> at org.apache.hudi.table.MarkerFiles.create(MarkerFiles.java:195) at
> org.apache.hudi.io.HoodieWriteHandle.createMarkerFile(HoodieWriteHandle.java:129)
> at
> org.apache.hudi.io.HoodieCreateHandle.lambda$new$1(HoodieCreateHandle.java:97)
> at org.apache.hudi.common.util.RetryHelper.start(RetryHelper.java:61) at
> org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:100) at
> org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:67) at
> org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:34) at
> org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:83)
> at
> org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
> at
> org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
> at
> org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> It's because of meet the throttle of S3 list/get/put/delete.
> This exception will cause app crashed, which can be solved by retry wrapper.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
