Konstantin Bereznyakov created HADOOP-20041:
-----------------------------------------------
Summary: S3A: AWS SDK extension callbacks can observe another
filesystem user's UGI
Key: HADOOP-20041
URL: https://issues.apache.org/jira/browse/HADOOP-20041
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.5.0
Reporter: Konstantin Bereznyakov
h2. Background
During review of HADOOP-19862 ([apache/hadoop PR
8426|https://github.com/apache/hadoop/pull/8426]), [~stevel] has [raised the
concern about shared UGI subjects on shared-user
systems|https://github.com/apache/hadoop/pull/8426#issuecomment-4802130168].
That discussion concerned the proposed shared scheduled executors for AWS SDK
clients and whether to enable them by default.
Subsequent investigation demonstrated related ambient-UGI behaviour on the
SDK's existing Netty event-loop and completion threads, independently of those
shared scheduled executors. It reproduces in released Hadoop 3.5.0, which does
not contain HADOOP-19862, and on current trunk with all four shared-scheduler
flags disabled.
This report follows up on that review concern. It is not reporting a regression
introduced by HADOOP-19862. Enabling the shared schedulers adds a separate
execution path requiring consideration; keeping them disabled does not
eliminate the existing behaviour described here.
h2. Summary
S3A extension code invoked by the AWS SDK can observe the UGI of another
filesystem user when running on SDK Netty event-loop or completion threads.
This was reproduced with Hadoop 3.5.0 and freshly built trunk at
6980beafb9ed7d672c0a61bd4eec1a3246db1ae1, using AWS SDK 2.35.4.
h2. Reproduction
The standalone Maven reproduction uses a local HTTP S3 stand-in and synthetic
credentials. No AWS account or real credentials are required.
* Create separate Alice and Bob UGIs.
* Under Alice.doAs, create enough separate S3A filesystem instances and
perform asynchronous copies to start the SDK's shared Netty event-loop threads.
Keep those filesystem instances open.
* Under Bob.doAs, create another filesystem instance and rename an object. The
local endpoint returns one 503 response so that the SDK also exercises a retry.
* Record the executing thread and UserGroupInformation.getCurrentUser() in an
execution interceptor, a custom signer delegating to AwsS3V4Signer, and a
credentials provider.
* Repeat with the classic input stream and a 15 MiB object, using a 5 MiB
multipart threshold and part size, so the copy uses multipart follow-up
requests.
* As a control, repeat the same scenarios but close Alice's filesystem
instances before starting Bob's operation.
The credentials provider returns a distinct, configuration-bound synthetic
access key for each filesystem. The endpoint records the access key used to
sign every request.
h2. Observed behaviour
With Alice's filesystem instances still open, Bob's callbacks on
aws-java-sdk-NettyEventLoop threads observe Alice's UGI. Callbacks on Bob's
per-client sdk-async-response completion threads also observe Alice.
For the multipart copy, the completion threads execute 10 credentials-provider
lookups and 5 signer calls under Alice's ambient UGI.
||Hadoop version||JDK||Alice's filesystems remain open||Alice's filesystems
closed before Bob starts||
|3.5.0|17, 21|Alice on Netty/completion callbacks|Bob|
|3.5.0|25|Login user on SDK worker callbacks|Login user|
|Trunk 6980beafb9e|17, 21, 25|Alice on Netty/completion callbacks|Bob|
The caller-thread callbacks run as Bob. On trunk with the shared schedulers
disabled, per-client retry signing also runs as Bob.
Every recorded Bob request is signed with Bob's synthetic access key: 10
requests in the non-multipart scenario and 15 in the multipart scenario. The
demonstrated mismatch is the ambient UGI seen by extension code, not use of
Alice's signing credentials.
h2. Thread-context mechanism
The SDK uses a JVM-wide shared Netty event-loop group. Its threads can retain
the context inherited when they were created.
Although the SDK completion executor is per client, its workers are created
lazily and can be created from an event-loop thread carrying another user's
context. A per-client completion executor therefore does not ensure that its
worker threads carry that client's filesystem-owner UGI.
HADOOP-19906 accounts for the JDK 25 difference above: trunk restores Subject
inheritance on newer JDKs, while released 3.5.0 falls back to the login user in
these worker-thread calls.
h2. Expected behaviour and impact
An extension selecting per-filesystem or per-user state should not
inadvertently select another filesystem user's state because an unrelated
client started the SDK threads first.
This matters for extensions that consult the current UGI. Hadoop's
ITestCustomSigner.CustomSigner.sign() itself uses
UserGroupInformation.getCurrentUser() to look up bucket/user-specific state, so
this is an existing S3A extension pattern.
If preserving the filesystem owner's UGI is not the intended callback contract,
the integration needs a supported way for such extensions to identify the
filesystem owner independently of the executing thread.
h2. Scope
* The issue is reproducible without enabling the shared scheduled executors.
* Both non-multipart and classic-stream multipart copies were exercised.
* Fresh trunk verification covered four scenarios on each of JDK 17, 21 and 25.
* No incorrect AWS signing key, unauthorized access or data corruption was
demonstrated.
* The standalone reproduction establishes the UGI observations above. Separate
earlier trunk characterizations observed context-class-loader and CallerContext
carryover, but those are not measured by this standalone reproduction.
* These observations do not establish an independent defect in Netty or
UserGroupInformation itself.
* No concrete patch is proposed yet. The intended context contract and the
appropriate boundary for supplying filesystem-owner context to S3A extensions
may need discussion.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
