[
https://issues.apache.org/jira/browse/HADOOP-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794302#comment-17794302
]
Steve Loughran commented on HADOOP-18915:
-----------------------------------------
looking at the AWS SDK source, it looks like the s3express createsession code
doesn't pick up the expanded settings as the timeout is fixed at 10s.
What does that mean? It means if you are working with S3Express buckets *you
had better have a large http connection pool and no connectivity issues*
we may want to think about to test/handle this best in our own code. We should
be retrying.
> Tune/extend S3A http connection and thread pool settings
> --------------------------------------------------------
>
> Key: HADOOP-18915
> URL: https://issues.apache.org/jira/browse/HADOOP-18915
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Ahmar Suhail
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.7-aws
>
>
> Increases existing pool sizes, as with server scale and vector
> IO, larger pools are needed
> fs.s3a.connection.maximum 200
> fs.s3a.threads.max 96
> Adds new configuration options for v2 sdk internal timeouts,
> both with default of 60s:
> fs.s3a.connection.acquisition.timeout
> fs.s3a.connection.idle.time
> All the pool/timoeut options are covered in performance.md
> Moves all timeout/duration options in the s3a FS to taking
> temporal units (h, m, s, ms,...); retaining the previous default
> unit (normally millisecond)
> Adds a minimum duration for most of these, in order to recover from
> deployments where a timeout has been set on the assumption the unit
> was seconds, not millis.
> Uses java.time.Duration throughout the codebase;
> retaining the older numeric constants in
> org.apache.hadoop.fs.s3a.Constants for backwards compatibility;
> these are now deprecated.
> Adds new class AWSApiCallTimeoutException to be raised on
> sdk-related methods and also gateway timeouts. This is a subclass
> of org.apache.hadoop.net.ConnectTimeoutException to support
> existing retry logic.
> + reverted default value of fs.s3a.create.performance to false;
> inadvertently set to true during testing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
