mukund-thakur commented on code in PR #5948:
URL: https://github.com/apache/hadoop/pull/5948#discussion_r1294714622
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:
##########
@@ -154,6 +156,11 @@ private Constants() {
public static final String MAXIMUM_CONNECTIONS = "fs.s3a.connection.maximum";
public static final int DEFAULT_MAXIMUM_CONNECTIONS = 96;
+ // Expiration time of s3 http connection from the connection pool.
Review Comment:
make javadoc; and use @value
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md:
##########
@@ -1782,6 +1782,23 @@ will attempt to retry the operation; it may just be a
transient event. If there
are many such exceptions in logs, it may be a symptom of connectivity or
network
problems.
+Above error could be because of a stale http connections. By default,
connections
+in the http connection pool are reused indefinitely. To discard connections
after
+a specific period of time please configure fs.s3a.connection.ttl.
+
+```xml
+<property>
+ <name>fs.s3a.connection.ttl</name>
+ <value>-1</value>
+ <description>
+ Expiration time for a connection in the connection pool in milliseconds.
+ When a connection is retrieved from the connection pool,
+ this parameter is checked to see if the connection can be reused.
+ Default value is set to -1 (infinite) which means connection
Review Comment:
Default in sdk is -1.
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md:
##########
@@ -1782,6 +1782,23 @@ will attempt to retry the operation; it may just be a
transient event. If there
are many such exceptions in logs, it may be a symptom of connectivity or
network
problems.
+Above error could be because of a stale http connections. By default,
connections
+in the http connection pool are reused indefinitely. To discard connections
after
Review Comment:
new default is 5 mins.
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:
##########
@@ -154,6 +156,11 @@ private Constants() {
public static final String MAXIMUM_CONNECTIONS = "fs.s3a.connection.maximum";
public static final int DEFAULT_MAXIMUM_CONNECTIONS = 96;
+ // Expiration time of s3 http connection from the connection pool.
+ // See {@code com.amazonaws.ClientConfiguration#setConnectionTTL}
+ public static final String CONNECTION_TTL = "fs.s3a.connection.ttl";
+ public static final long DEFAULT_CONNECTION_TTL =
ClientConfiguration.DEFAULT_CONNECTION_TTL;
Review Comment:
set the default to 5 mins.
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md:
##########
@@ -1782,6 +1782,23 @@ will attempt to retry the operation; it may just be a
transient event. If there
are many such exceptions in logs, it may be a symptom of connectivity or
network
problems.
+Above error could be because of a stale http connections. By default,
connections
Review Comment:
*the above.
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:
##########
@@ -154,6 +156,11 @@ private Constants() {
public static final String MAXIMUM_CONNECTIONS = "fs.s3a.connection.maximum";
public static final int DEFAULT_MAXIMUM_CONNECTIONS = 96;
+ // Expiration time of s3 http connection from the connection pool.
+ // See {@code com.amazonaws.ClientConfiguration#setConnectionTTL}
Review Comment:
cut the comment... v2 upgrade obsolete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
