[
https://issues.apache.org/jira/browse/OAK-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rishabh Daim updated OAK-12088:
-------------------------------
Issue Type: Task (was: Bug)
> Fix timeout issue while uploading multiple files on AWS
> -------------------------------------------------------
>
> Key: OAK-12088
> URL: https://issues.apache.org/jira/browse/OAK-12088
> Project: Jackrabbit Oak
> Issue Type: Task
> Reporter: Rishabh Daim
> Assignee: Rishabh Daim
> Priority: Major
>
> We are getting connection time while uploading multiple files to AWS S3.
> Exception is:
> {quote}Caused by: software.amazon.awssdk.core.exception.SdkClientException:
> Unable to execute HTTP request: Acquire operation took longer than the
> configured maximum time. This indicates that a request cannot get a
> connection from the pool within the specified maximum time. This can be due
> to high request rate._Consider taking any of the following actions to
> mitigate the issue: increase max connections, increase acquire timeout, or
> slowing the request rate._Increasing the max connections can increase client
> throughput (unless the network interface is already fully utilized), but can
> eventually start to hit operation system limitations on the number of file
> descriptors used by the process. If you already are fully utilizing your
> network interface or cannot further increase your connection count,
> increasing the acquire timeout gives extra time for requests to acquire a
> connection before timing out. If the connections doesn't free up, the
> subsequent requests will still timeout._If the above mechanisms are not able
> to fix the issue, try smoothing out your requests so that large traffic
> bursts cannot overload the client, being more efficient with the number of
> times you need to call AWS, or by increasing the number of hosts sending
> requests. (SDK Attempt Count: 1)
> at
> software.amazon.awssdk.core.exception.SdkClientException<span>BuilderImpl.build(SdkClientException.java:130)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> software.amazon.awssdk.core.exception.SdkClientException</span>BuilderImpl.build(SdkClientException.java:95)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage<span>RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:139)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage</span>RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:157)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage<span>RetryingExecutor.lambda</span>attemptExecute<span>1(AsyncRetryableStage.java:117)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> software.amazon.awssdk.utils.CompletableFutureUtils.lambda<span>forwardExceptionTo</span>0(CompletableFutureUtils.java:78)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture<span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda</span>execute<span>0(MakeAsyncHttpRequestStage.java:108)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda<span>executeHttpRequest</span>3(MakeAsyncHttpRequestStage.java:167)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
> at
> java.base/java.util.concurrent.CompletableFuture<span>UniHandle.tryFire(CompletableFuture.java:911)
> at
> java.base/java.util.concurrent.CompletableFuture</span>Completion.run(CompletableFuture.java:482)
> ... 3 common frames omitted
> Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Failed
> to send multipart upload requests
> at
> software.amazon.awssdk.core.exception.SdkClientException<span>BuilderImpl.build(SdkClientException.java:130)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> software.amazon.awssdk.services.s3.internal.multipart.GenericMultipartHelper.handleException(GenericMultipartHelper.java:66)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> software.amazon.awssdk.services.s3.internal.multipart.MultipartUploadHelper.failRequestsElegantly(MultipartUploadHelper.java:126)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> software.amazon.awssdk.services.s3.internal.multipart.KnownContentLengthAsyncRequestBodySubscriber.lambda</span>onNext<span>5(KnownContentLengthAsyncRequestBodySubscriber.java:190)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> software.amazon.awssdk.utils.CompletableFutureUtils.lambda<span>forwardExceptionTo</span>0(CompletableFutureUtils.java:78)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture<span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.lambda</span>execute<span>0(AsyncApiCallMetricCollectionStage.java:56)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.lambda<span>execute</span>2(AsyncApiCallTimeoutTrackingStage.java:67)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture<span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> software.amazon.awssdk.utils.CompletableFutureUtils.lambda</span>forwardExceptionTo<span>0(CompletableFutureUtils.java:78)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> ... 25 common frames omitted
> Caused by: java.lang.Throwable: Acquire operation took longer than the
> configured maximum time. This indicates that a request cannot get a
> connection from the pool within the specified maximum time. This can be due
> to high request rate._Consider taking any of the following actions to
> mitigate the issue: increase max connections, increase acquire timeout, or
> slowing the request rate._Increasing the max connections can increase client
> throughput (unless the network interface is already fully utilized), but can
> eventually start to hit operation system limitations on the number of file
> descriptors used by the process. If you already are fully utilizing your
> network interface or cannot further increase your connection count,
> increasing the acquire timeout gives extra time for requests to acquire a
> connection before timing out. If the connections doesn't free up, the
> subsequent requests will still timeout._If the above mechanisms are not able
> to fix the issue, try smoothing out your requests so that large traffic
> bursts cannot overload the client, being more efficient with the number of
> times you need to call AWS, or by increasing the number of hosts sending
> requests.
> at
> software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.decorateException(NettyUtils.java:80)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:307)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequestListener(NettyRequestExecutor.java:188)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:603)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:570)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.DefaultPromise.access<span>200(DefaultPromise.java:36)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> io.netty.util.concurrent.DefaultPromise</span>1.run(DefaultPromise.java:516)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor<span>4.run(SingleThreadEventExecutor.java:998)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> io.netty.util.internal.ThreadExecutorMap</span>2.run(ThreadExecutorMap.java:74)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> ... 1 common frames omitted
> Caused by: java.util.concurrent.TimeoutException: Acquire operation took
> longer than 10000 milliseconds.
> at
> software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.timeoutAcquire(HealthCheckedChannelPool.java:77)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.lambda<span>acquire</span>0(HealthCheckedChannelPool.java:67)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:160)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> ... 7 common frames omitted
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
