[
https://issues.apache.org/jira/browse/HADOOP-18845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754718#comment-17754718
]
ASF GitHub Bot commented on HADOOP-18845:
-----------------------------------------
mukund-thakur commented on code in PR #5948:
URL: https://github.com/apache/hadoop/pull/5948#discussion_r1294714622
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:
##########
@@ -154,6 +156,11 @@ private Constants() {
public static final String MAXIMUM_CONNECTIONS = "fs.s3a.connection.maximum";
public static final int DEFAULT_MAXIMUM_CONNECTIONS = 96;
+ // Expiration time of s3 http connection from the connection pool.
Review Comment:
make javadoc; and use @value
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md:
##########
@@ -1782,6 +1782,23 @@ will attempt to retry the operation; it may just be a
transient event. If there
are many such exceptions in logs, it may be a symptom of connectivity or
network
problems.
+Above error could be because of a stale http connections. By default,
connections
+in the http connection pool are reused indefinitely. To discard connections
after
+a specific period of time please configure fs.s3a.connection.ttl.
+
+```xml
+<property>
+ <name>fs.s3a.connection.ttl</name>
+ <value>-1</value>
+ <description>
+ Expiration time for a connection in the connection pool in milliseconds.
+ When a connection is retrieved from the connection pool,
+ this parameter is checked to see if the connection can be reused.
+ Default value is set to -1 (infinite) which means connection
Review Comment:
Default in sdk is -1.
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md:
##########
@@ -1782,6 +1782,23 @@ will attempt to retry the operation; it may just be a
transient event. If there
are many such exceptions in logs, it may be a symptom of connectivity or
network
problems.
+Above error could be because of a stale http connections. By default,
connections
+in the http connection pool are reused indefinitely. To discard connections
after
Review Comment:
new default is 5 mins.
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:
##########
@@ -154,6 +156,11 @@ private Constants() {
public static final String MAXIMUM_CONNECTIONS = "fs.s3a.connection.maximum";
public static final int DEFAULT_MAXIMUM_CONNECTIONS = 96;
+ // Expiration time of s3 http connection from the connection pool.
+ // See {@code com.amazonaws.ClientConfiguration#setConnectionTTL}
+ public static final String CONNECTION_TTL = "fs.s3a.connection.ttl";
+ public static final long DEFAULT_CONNECTION_TTL =
ClientConfiguration.DEFAULT_CONNECTION_TTL;
Review Comment:
set the default to 5 mins.
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md:
##########
@@ -1782,6 +1782,23 @@ will attempt to retry the operation; it may just be a
transient event. If there
are many such exceptions in logs, it may be a symptom of connectivity or
network
problems.
+Above error could be because of a stale http connections. By default,
connections
Review Comment:
*the above.
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:
##########
@@ -154,6 +156,11 @@ private Constants() {
public static final String MAXIMUM_CONNECTIONS = "fs.s3a.connection.maximum";
public static final int DEFAULT_MAXIMUM_CONNECTIONS = 96;
+ // Expiration time of s3 http connection from the connection pool.
+ // See {@code com.amazonaws.ClientConfiguration#setConnectionTTL}
Review Comment:
cut the comment... v2 upgrade obsolete.
> Add ability to configure ConnectionTTL of http connections while creating S3
> Client.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-18845
> URL: https://issues.apache.org/jira/browse/HADOOP-18845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.6
> Reporter: Mukund Thakur
> Assignee: Mukund Thakur
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.9
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
