[
https://issues.apache.org/jira/browse/HADOOP-18410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17581783#comment-17581783
]
Steve Loughran commented on HADOOP-18410:
-----------------------------------------
interesting fact, the connection pool wait timeout is set from connection
timeout option {{fs.s3a.connection.request.timeout}}, whose default is actually
0. we should set it to a realistic number.
aws class to go from ClientConfig to HttpClientConfig is
{{com.amazonaws.http.settings.HttpClientSettings}}
{code}
public int getConnectionTimeout() {
return config.getConnectionTimeout();
}
public int getConnectionPoolRequestTimeout() {
return config.getConnectionTimeout();
}
{code}
my test setup does have a default value, but it way too big, probably so i can
debug things without timeouts
{code}
<property>
<name>fs.s3a.connection.request.timeout</name>
<value>6000s</value>
</property>
{code}
we could set it in test/resources/core-site to something small; just to see
what happens.
> S3AInputStream async drain not releasing http connections
> ---------------------------------------------------------
>
> Key: HADOOP-18410
> URL: https://issues.apache.org/jira/browse/HADOOP-18410
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.9
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> Impala tcp-ds setup to s3 is hitting problems with timeout fetching http
> connections from the s3a fs pool. Disabling s3a async drain makes this
> problem *go away*. assumption, either those async ops are blocking, or they
> are not releasing references properly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
