[
https://issues.apache.org/jira/browse/OAK-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rishabh Daim updated OAK-12088:
-------------------------------
Description:
We are getting connection time while uploading multiple files to AWS S3.
Exception is:
{quote}Caused by: software.amazon.awssdk.core.exception.SdkClientException:
Unable to execute HTTP request: Acquire operation took longer than the
configured maximum time. This indicates that a request cannot get a connection
from the pool within the specified maximum time. This can be due to high
request rate._Consider taking any of the following actions to mitigate the
issue: increase max connections, increase acquire timeout, or slowing the
request rate._Increasing the max connections can increase client throughput
(unless the network interface is already fully utilized), but can eventually
start to hit operation system limitations on the number of file descriptors
used by the process. If you already are fully utilizing your network interface
or cannot further increase your connection count, increasing the acquire
timeout gives extra time for requests to acquire a connection before timing
out. If the connections doesn't free up, the subsequent requests will still
timeout._If the above mechanisms are not able to fix the issue, try smoothing
out your requests so that large traffic bursts cannot overload the client,
being more efficient with the number of times you need to call AWS, or by
increasing the number of hosts sending requests. (SDK Attempt Count: 1)
at
software.amazon.awssdk.core.exception.SdkClientException<span>BuilderImpl.build(SdkClientException.java:130)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
software.amazon.awssdk.core.exception.SdkClientException</span>BuilderImpl.build(SdkClientException.java:95)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage<span>RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:139)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage</span>RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:157)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage<span>RetryingExecutor.lambda</span>attemptExecute<span>1(AsyncRetryableStage.java:117)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at
java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(CompletableFuture.java:841)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at
software.amazon.awssdk.utils.CompletableFutureUtils.lambda<span>forwardExceptionTo</span>0(CompletableFutureUtils.java:78)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at
java.base/java.util.concurrent.CompletableFuture<span>UniWhenComplete.tryFire(CompletableFuture.java:841)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda</span>execute<span>0(MakeAsyncHttpRequestStage.java:108)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at
java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(CompletableFuture.java:841)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda<span>executeHttpRequest</span>3(MakeAsyncHttpRequestStage.java:167)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
at
java.base/java.util.concurrent.CompletableFuture<span>UniHandle.tryFire(CompletableFuture.java:911)
at
java.base/java.util.concurrent.CompletableFuture</span>Completion.run(CompletableFuture.java:482)
... 3 common frames omitted
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Failed to
send multipart upload requests
at
software.amazon.awssdk.core.exception.SdkClientException<span>BuilderImpl.build(SdkClientException.java:130)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
software.amazon.awssdk.services.s3.internal.multipart.GenericMultipartHelper.handleException(GenericMultipartHelper.java:66)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
software.amazon.awssdk.services.s3.internal.multipart.MultipartUploadHelper.failRequestsElegantly(MultipartUploadHelper.java:126)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
software.amazon.awssdk.services.s3.internal.multipart.KnownContentLengthAsyncRequestBodySubscriber.lambda</span>onNext<span>5(KnownContentLengthAsyncRequestBodySubscriber.java:190)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at
java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(CompletableFuture.java:841)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at
software.amazon.awssdk.utils.CompletableFutureUtils.lambda<span>forwardExceptionTo</span>0(CompletableFutureUtils.java:78)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at
java.base/java.util.concurrent.CompletableFuture<span>UniWhenComplete.tryFire(CompletableFuture.java:841)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.lambda</span>execute<span>0(AsyncApiCallMetricCollectionStage.java:56)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at
java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(CompletableFuture.java:841)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.lambda<span>execute</span>2(AsyncApiCallTimeoutTrackingStage.java:67)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at
java.base/java.util.concurrent.CompletableFuture<span>UniWhenComplete.tryFire(CompletableFuture.java:841)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at
software.amazon.awssdk.utils.CompletableFutureUtils.lambda</span>forwardExceptionTo<span>0(CompletableFutureUtils.java:78)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at
java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(CompletableFuture.java:841)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
... 25 common frames omitted
Caused by: java.lang.Throwable: Acquire operation took longer than the
configured maximum time. This indicates that a request cannot get a connection
from the pool within the specified maximum time. This can be due to high
request rate._Consider taking any of the following actions to mitigate the
issue: increase max connections, increase acquire timeout, or slowing the
request rate._Increasing the max connections can increase client throughput
(unless the network interface is already fully utilized), but can eventually
start to hit operation system limitations on the number of file descriptors
used by the process. If you already are fully utilizing your network interface
or cannot further increase your connection count, increasing the acquire
timeout gives extra time for requests to acquire a connection before timing
out. If the connections doesn't free up, the subsequent requests will still
timeout._If the above mechanisms are not able to fix the issue, try smoothing
out your requests so that large traffic bursts cannot overload the client,
being more efficient with the number of times you need to call AWS, or by
increasing the number of hosts sending requests.
at
software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.decorateException(NettyUtils.java:80)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:307)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequestListener(NettyRequestExecutor.java:188)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:603)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:570)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
io.netty.util.concurrent.DefaultPromise.access<span>200(DefaultPromise.java:36)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
io.netty.util.concurrent.DefaultPromise</span>1.run(DefaultPromise.java:516)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
io.netty.util.concurrent.SingleThreadEventExecutor<span>4.run(SingleThreadEventExecutor.java:998)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
io.netty.util.internal.ThreadExecutorMap</span>2.run(ThreadExecutorMap.java:74)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
... 1 common frames omitted
Caused by: java.util.concurrent.TimeoutException: Acquire operation took longer
than 10000 milliseconds.
at
software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.timeoutAcquire(HealthCheckedChannelPool.java:77)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.lambda<span>acquire</span>0(HealthCheckedChannelPool.java:67)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
at
io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:160)
[org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
... 7 common frames omitted
{quote}
was:
We are getting NPE while unmarshalling the InputStream during uploading of
metadata to AWS S3.
Exception is:
{quote}08.01.2026 14:46:29.770 *ERROR* [FelixStartLevel]
org.apache.jackrabbit.oak.blob.cloud.s3.S3Backend Exception in uploading
metadata file
java.util.concurrent.CompletionException:
software.amazon.awssdk.core.exception.SdkClientException
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:870)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
at
java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2253)
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.measureApiCallSuccess(BaseAsyncClientHandler.java:296)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.execute(BaseAsyncClientHandler.java:73)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.execute(AwsAsyncClientHandler.java:49)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.services.s3.DefaultS3AsyncClient.putObject(DefaultS3AsyncClient.java:12982)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.services.s3.DelegatingS3AsyncClient.lambda$putObject$89(DelegatingS3AsyncClient.java:10008)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.services.s3.internal.multipart.MultipartS3AsyncClient$1.invokeOperation(MultipartS3AsyncClient.java:108)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.services.s3.DelegatingS3AsyncClient.putObject(DelegatingS3AsyncClient.java:10008)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.services.s3.internal.multipart.MultipartUploadHelper.uploadInOneChunk(MultipartUploadHelper.java:155)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.services.s3.internal.multipart.UploadWithUnknownContentLengthHelper$UnknownContentLengthAsyncRequestBodySubscriber.onComplete(UploadWithUnknownContentLengthHelper.java:282)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.core.internal.async.SplittingPublisher$SplittingSubscriber.onComplete(SplittingPublisher.java:247)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.utils.async.InputStreamConsumingPublisher.doBlockingWrite(InputStreamConsumingPublisher.java:62)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody.writeInputStream(BlockingInputStreamAsyncRequestBody.java:95)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.doBlockingWrite(InputStreamWithExecutorAsyncRequestBody.java:112)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.lambda$subscribe$0(InputStreamWithExecutorAsyncRequestBody.java:80)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: null
at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.core.internal.util.ThrowableUtils.asSdkException(ThrowableUtils.java:98)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.doExecute(BaseAsyncClientHandler.java:248)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$execute$1(BaseAsyncClientHandler.java:80)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.measureApiCallSuccess(BaseAsyncClientHandler.java:294)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
... 24 common frames omitted
Caused by: java.lang.NullPointerException: null
at
software.amazon.awssdk.core.runtime.transform.AsyncStreamingRequestMarshaller.marshall(AsyncStreamingRequestMarshaller.java:53)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$finalizeSdkHttpFullRequest$0(BaseClientHandler.java:73)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:64)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.core.internal.handler.BaseClientHandler.finalizeSdkHttpFullRequest(BaseClientHandler.java:72)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.doExecute(BaseAsyncClientHandler.java:199)
[org.apache.jackrabbit.oak-blob-cloud:1.22.24.SNAPSHOT]
... 26 common frames omitted
{quote}
> Fix timeout issue while uploading multiple files on AWS
> -------------------------------------------------------
>
> Key: OAK-12088
> URL: https://issues.apache.org/jira/browse/OAK-12088
> Project: Jackrabbit Oak
> Issue Type: Bug
> Reporter: Rishabh Daim
> Assignee: Rishabh Daim
> Priority: Major
> Fix For: 1.22.24, 1.92.0
>
>
> We are getting connection time while uploading multiple files to AWS S3.
> Exception is:
> {quote}Caused by: software.amazon.awssdk.core.exception.SdkClientException:
> Unable to execute HTTP request: Acquire operation took longer than the
> configured maximum time. This indicates that a request cannot get a
> connection from the pool within the specified maximum time. This can be due
> to high request rate._Consider taking any of the following actions to
> mitigate the issue: increase max connections, increase acquire timeout, or
> slowing the request rate._Increasing the max connections can increase client
> throughput (unless the network interface is already fully utilized), but can
> eventually start to hit operation system limitations on the number of file
> descriptors used by the process. If you already are fully utilizing your
> network interface or cannot further increase your connection count,
> increasing the acquire timeout gives extra time for requests to acquire a
> connection before timing out. If the connections doesn't free up, the
> subsequent requests will still timeout._If the above mechanisms are not able
> to fix the issue, try smoothing out your requests so that large traffic
> bursts cannot overload the client, being more efficient with the number of
> times you need to call AWS, or by increasing the number of hosts sending
> requests. (SDK Attempt Count: 1)
> at
> software.amazon.awssdk.core.exception.SdkClientException<span>BuilderImpl.build(SdkClientException.java:130)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> software.amazon.awssdk.core.exception.SdkClientException</span>BuilderImpl.build(SdkClientException.java:95)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage<span>RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:139)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage</span>RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:157)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage<span>RetryingExecutor.lambda</span>attemptExecute<span>1(AsyncRetryableStage.java:117)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> software.amazon.awssdk.utils.CompletableFutureUtils.lambda<span>forwardExceptionTo</span>0(CompletableFutureUtils.java:78)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture<span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda</span>execute<span>0(MakeAsyncHttpRequestStage.java:108)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda<span>executeHttpRequest</span>3(MakeAsyncHttpRequestStage.java:167)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
> at
> java.base/java.util.concurrent.CompletableFuture<span>UniHandle.tryFire(CompletableFuture.java:911)
> at
> java.base/java.util.concurrent.CompletableFuture</span>Completion.run(CompletableFuture.java:482)
> ... 3 common frames omitted
> Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Failed
> to send multipart upload requests
> at
> software.amazon.awssdk.core.exception.SdkClientException<span>BuilderImpl.build(SdkClientException.java:130)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> software.amazon.awssdk.services.s3.internal.multipart.GenericMultipartHelper.handleException(GenericMultipartHelper.java:66)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> software.amazon.awssdk.services.s3.internal.multipart.MultipartUploadHelper.failRequestsElegantly(MultipartUploadHelper.java:126)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> software.amazon.awssdk.services.s3.internal.multipart.KnownContentLengthAsyncRequestBodySubscriber.lambda</span>onNext<span>5(KnownContentLengthAsyncRequestBodySubscriber.java:190)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> software.amazon.awssdk.utils.CompletableFutureUtils.lambda<span>forwardExceptionTo</span>0(CompletableFutureUtils.java:78)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture<span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.lambda</span>execute<span>0(AsyncApiCallMetricCollectionStage.java:56)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.lambda<span>execute</span>2(AsyncApiCallTimeoutTrackingStage.java:67)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture<span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> software.amazon.awssdk.utils.CompletableFutureUtils.lambda</span>forwardExceptionTo<span>0(CompletableFutureUtils.java:78)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> at
> java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(CompletableFuture.java:841)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> ... 25 common frames omitted
> Caused by: java.lang.Throwable: Acquire operation took longer than the
> configured maximum time. This indicates that a request cannot get a
> connection from the pool within the specified maximum time. This can be due
> to high request rate._Consider taking any of the following actions to
> mitigate the issue: increase max connections, increase acquire timeout, or
> slowing the request rate._Increasing the max connections can increase client
> throughput (unless the network interface is already fully utilized), but can
> eventually start to hit operation system limitations on the number of file
> descriptors used by the process. If you already are fully utilizing your
> network interface or cannot further increase your connection count,
> increasing the acquire timeout gives extra time for requests to acquire a
> connection before timing out. If the connections doesn't free up, the
> subsequent requests will still timeout._If the above mechanisms are not able
> to fix the issue, try smoothing out your requests so that large traffic
> bursts cannot overload the client, being more efficient with the number of
> times you need to call AWS, or by increasing the number of hosts sending
> requests.
> at
> software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.decorateException(NettyUtils.java:80)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:307)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequestListener(NettyRequestExecutor.java:188)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:603)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:570)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.DefaultPromise.access<span>200(DefaultPromise.java:36)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> io.netty.util.concurrent.DefaultPromise</span>1.run(DefaultPromise.java:516)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor<span>4.run(SingleThreadEventExecutor.java:998)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002] at
> io.netty.util.internal.ThreadExecutorMap</span>2.run(ThreadExecutorMap.java:74)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> ... 1 common frames omitted
> Caused by: java.util.concurrent.TimeoutException: Acquire operation took
> longer than 10000 milliseconds.
> at
> software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.timeoutAcquire(HealthCheckedChannelPool.java:77)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.lambda<span>acquire</span>0(HealthCheckedChannelPool.java:67)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> at
> io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:160)
> [org.apache.jackrabbit.oak-blob-cloud:1.88.1.B002]
> ... 7 common frames omitted
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
