[
https://issues.apache.org/jira/browse/HADOOP-19007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794670#comment-17794670
]
Ahmar Suhail commented on HADOOP-19007:
---------------------------------------
So I think what it means to pass in the executor has changed b/w V1 and V2.
With V1, that executor pool would be used to make requests to S3, documented
[here|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html#TransferManager-com.amazonaws.services.s3.AmazonS3-java.util.concurrent.ExecutorService-]
With V2, if you pass it in, it's only used for certain background tasks before
calling the S3AsyncClient such as visiting file tree in a
S3TransferManager.uploadDirectory(UploadDirectoryRequest) operation, I don't
think it's relevant for our usecase of copy. Documented
[here|https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/transfer/s3/S3TransferManager.Builder.html#executor(java.util.concurrent.Executor)]
It'll end up using the executor of the S3AsyncClient. Currently that client
creates it's own executor pool, but we can also pass in our own if required.
That behaviour is documented
[here|https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/asynchronous.html]
Do you think there is an advantage here of passing the in the boundedThreadPool
to the S3AsyncClient?
> S3A: transfer manager not wired up to s3a executor pool
> -------------------------------------------------------
>
> Key: HADOOP-19007
> URL: https://issues.apache.org/jira/browse/HADOOP-19007
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Priority: Major
>
> S3ClientFactory.createS3TransferManager() doesn't use the executor declared
> in S3ClientCreationParameters.transferManagerExecutor
> * method needs to take S3ClientCreationParameters
> * and set the transfer manager executor
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
