Steve Loughran created HADOOP-13637:
---------------------------------------
Summary: improve setting of max connections in AWS client
Key: HADOOP-13637
URL: https://issues.apache.org/jira/browse/HADOOP-13637
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 2.8.0
Reporter: Steve Loughran
Priority: Minor
things can go badly wrong if the S3A FS creates a thread pool for IO > than the
number of pooled AWS http connections (set by property MAXIMUM_CONNECTIONS);
you also need some for any other IO requests coming in.
The max connections property is currently independent of thread pool size, and
has a default value of 1.
this is why there is a troubleshooting section in the docs showing the stack
trace and instructions to fix".
Better: have a dynamic minimum like thread pool size + n, for a value of n to
be chosen.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
