[
https://issues.apache.org/jira/browse/HADOOP-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-18915.
-------------------------------------
Fix Version/s: 3.4.0
Resolution: Fixed
> Tune/extend S3A http connection and thread pool settings
> --------------------------------------------------------
>
> Key: HADOOP-18915
> URL: https://issues.apache.org/jira/browse/HADOOP-18915
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Ahmar Suhail
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Increases existing pool sizes, as with server scale and vector
> IO, larger pools are needed
> fs.s3a.connection.maximum 200
> fs.s3a.threads.max 96
> Adds new configuration options for v2 sdk internal timeouts,
> both with default of 60s:
> fs.s3a.connection.acquisition.timeout
> fs.s3a.connection.idle.time
> All the pool/timoeut options are covered in performance.md
> Moves all timeout/duration options in the s3a FS to taking
> temporal units (h, m, s, ms,...); retaining the previous default
> unit (normally millisecond)
> Adds a minimum duration for most of these, in order to recover from
> deployments where a timeout has been set on the assumption the unit
> was seconds, not millis.
> Uses java.time.Duration throughout the codebase;
> retaining the older numeric constants in
> org.apache.hadoop.fs.s3a.Constants for backwards compatibility;
> these are now deprecated.
> Adds new class AWSApiCallTimeoutException to be raised on
> sdk-related methods and also gateway timeouts. This is a subclass
> of org.apache.hadoop.net.ConnectTimeoutException to support
> existing retry logic.
> + reverted default value of fs.s3a.create.performance to false;
> inadvertently set to true during testing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
