This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 736ea5e04b5af9e42081d2c7d5c381b69413b7d1
Author: Mukund Thakur <mtha...@cloudera.com>
AuthorDate: Tue Jan 16 17:06:28 2024 -0600
HADOOP-19015. Increase fs.s3a.connection.maximum to 500 to minimize risk
of Timeout waiting for connection from pool. (#6372)
HADOOP-19015. Increase fs.s3a.connection.maximum to 500 to minimize the
risk of Timeout waiting for connection from the pool
Contributed By: Mukund Thakur
---
.../hadoop-common/src/main/resources/core-default.xml | 3 ++-
.../hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java | 2 +-
.../hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md | 2 +-
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
index 5a5171056d04..29ec06db6598 100644
--- a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
+++ b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
@@ -1530,7 +1530,7 @@
<property>
<name>fs.s3a.connection.maximum</name>
- <value>200</value>
+ <value>500</value>
<description>Controls the maximum number of simultaneous connections to S3.
This must be bigger than the value of fs.s3a.threads.max so as to stop
threads being blocked waiting for new HTTPS connections.
@@ -1538,6 +1538,7 @@
</description>
</property>
+
<property>
<name>fs.s3a.connection.ssl.enabled</name>
<value>true</value>
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
index e33f762cdfcf..636438afef25 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
@@ -173,7 +173,7 @@ public final class Constants {
* Future releases are likely to increase this value.
* Keep in sync with the value in {@code core-default.xml}
*/
- public static final int DEFAULT_MAXIMUM_CONNECTIONS = 200;
+ public static final int DEFAULT_MAXIMUM_CONNECTIONS = 500;
/**
* Configuration option to configure expiration time of
diff --git
a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md
b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md
index 4d506b6bfc49..4c03cca17161 100644
--- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md
+++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md
@@ -226,7 +226,7 @@ The choice is yours. Generally recovery is better, but
sometimes fail-fast is mo
| Property | Default | V2 | Meaning
|
|-----------------------------------------|---------|:----|-------------------------------------------------------|
-| `fs.s3a.connection.maximum` | `200` | | Connection pool
size |
+| `fs.s3a.connection.maximum` | `500` | | Connection pool
size |
| `fs.s3a.connection.keepalive` | `false` | `*` | Use TCP keepalive
on open channels |
| `fs.s3a.connection.acquisition.timeout` | `60s` | `*` | Timeout for
waiting for a connection from the pool. |
| `fs.s3a.connection.establish.timeout` | `30s` | | Time to establish
the TCP/TLS connection |
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
