Konstantin Bereznyakov created HADOOP-20044:
-----------------------------------------------
Summary: S3A: enable shared AWS client schedulers by default and
fix pool-test isolation
Key: HADOOP-20044
URL: https://issues.apache.org/jira/browse/HADOOP-20044
Project: Hadoop Common
Issue Type: Improvement
Components: fs/s3
Reporter: Konstantin Bereznyakov
HADOOP-19862 introduced shared scheduled thread pools for S3A’s AWS clients.
During review, we agreed to keep them opt-in initially and change the defaults
in a follow-up. This is that follow-up.
Enable all four pools—S3 sync, S3 async, STS and KMS—by default. Keep the
existing flags as opt-outs and leave pool sizes and idle timeouts unchanged.
The same PR should fix the enabled/disabled tests’ isolation. Their pool
settings are remembered for the life of the JVM, so reusing a test JVM can make
the result depend on which class runs first. The Javadocs edits from
HADOOP-19862 also overstate when separate forks are guaranteed.
h2. The change scope:
* Consistent XML and Java defaults, with tests for explicit opt-out.
* Reliable on/off tests in both execution orders and sequential/parallel
profiles.
* [Possibly] A regression check for multipart-copy auditing, including retries.
h2. A follow-up review also raised these questions:
* Should we change the test-only accessor visibility? STS and KMS accessors
are public, while the S3 equivalents are package-private.
* Can the thread-counting helper avoid collecting stack frames when it only
needs thread names?
* Would simplifying getters or sharing test scaffolding help the isolation
correction?
* Is parallel filesystem creation worthwhile in the test helper?
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
