Yue Zhang created HUDI-2648:
-------------------------------
Summary: Retry FileSystem action when caught runtime exception
Key: HUDI-2648
URL: https://issues.apache.org/jira/browse/HUDI-2648
Project: Apache Hudi
Issue Type: Task
Reporter: Yue Zhang
Hoodie will do lots of list/get/put/delete etc actions on filesystem.
Sometimes will meet the fileSystem performance issue to response like
Using hoodie on S3, and will throw lots of exception like this
{code:java}
com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 503, AWS
Service: Amazon S3, AWS Request ID: J58R9W4E324DWJ1Y, AWS Error Code: SlowDown,
AWS Error Message: Please reduce your request rate., S3 Extended Request ID:
Pkd6VCJKTuq3NJ6vUQJB1TUznKGvAfjIZkmJOxt+j6oAD5roz+ojKf9rtcBOgUBw4sWeDwxdTv4=com.amazonaws.services.s3.model.AmazonS3Exception:
Status Code: 503, AWS Service: Amazon S3, AWS Request ID: J58R9W4E324DWJ1Y,
AWS Error Code: SlowDown, AWS Error Message: Please reduce your request rate.,
S3 Extended Request ID:
Pkd6VCJKTuq3NJ6vUQJB1TUznKGvAfjIZkmJOxt+j6oAD5roz+ojKf9rtcBOgUBw4sWeDwxdTv4= at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at
com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393) at
org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:1191)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:1168)
at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:871) at
org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1881) at
org.apache.hudi.common.fs.HoodieWrapperFileSystem.lambda$mkdirs$25(HoodieWrapperFileSystem.java:632)
at
org.apache.hudi.common.fs.HoodieWrapperFileSystem.executeFuncWithTimeMetrics(HoodieWrapperFileSystem.java:100)
at
org.apache.hudi.common.fs.HoodieWrapperFileSystem.mkdirs(HoodieWrapperFileSystem.java:631)
at org.apache.hudi.table.MarkerFiles.create(MarkerFiles.java:195) at
org.apache.hudi.io.HoodieWriteHandle.createMarkerFile(HoodieWriteHandle.java:129)
at
org.apache.hudi.io.HoodieCreateHandle.lambda$new$1(HoodieCreateHandle.java:97)
at org.apache.hudi.common.util.RetryHelper.start(RetryHelper.java:61) at
org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:100) at
org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:67) at
org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:34) at
org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:83)
at
org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
at
org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
It's because of meet the throttle of S3 list/get/put/delete.
This action will cause app crashed, which can be solved by retry wrapper.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
