[
https://issues.apache.org/jira/browse/HADOOP-18993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18031961#comment-18031961
]
Brandon edited comment on HADOOP-18993 at 10/22/25 6:03 AM:
------------------------------------------------------------
Tried this with Hadoop 3.4.1. It appears the `fs.s3a.classloader.isolation`
property does not have the desired effect, because the credentials providers
are loaded using the classloader of S3AUtils class:
[CredentialProviderListFactory#createAWSV2CredentialProvider|https://github.com/apache/hadoop/blob/4f9efcb3bc10d11d6c81457bc51e59dfe0791041/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/CredentialProviderListFactory.java#L299-L305]
calls
[S3AUtils#getInstanceFromReflection|https://github.com/apache/hadoop/blob/4f9efcb3bc10d11d6c81457bc51e59dfe0791041/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java#L642],
which uses the classloader that loaded the S3AUtils class, not the classloader
set on the Configuration
was (Author: brandonvin):
Tried this with Hadoop 3.4.1. It appears the `fs.s3a.classloader.isolation`
property does not have the desired effect, because the credentials providers
are loaded using the classloader of S3AUtils class instead of the Configuration
class:
[[https://github.com/apache/hadoop/blob/4f9efcb3bc10d11d6c81457bc51e59dfe0791041/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/CredentialProviderListFactory.java#L299-L305|http://example.com]|https://github.com/apache/hadoop/blob/4f9efcb3bc10d11d6c81457bc51e59dfe0791041/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/CredentialProviderListFactory.java#L299-L305]
[https://github.com/apache/hadoop/blob/4f9efcb3bc10d11d6c81457bc51e59dfe0791041/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java#L642]
> S3A: Add option fs.s3a.classloader.isolation (#6301)
> ----------------------------------------------------
>
> Key: HADOOP-18993
> URL: https://issues.apache.org/jira/browse/HADOOP-18993
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 3.3.6
> Reporter: Antonio Murgia
> Assignee: Antonio Murgia
> Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.5.0, 3.4.1
>
>
> In HADOOP-17372 the S3AFileSystem forces the configuration classloader to be
> the same as the one that loaded S3AFileSystem. This leads to the
> impossibility in Spark applications to load third party credentials providers
> as user jars.
> The option fs.s3a.classloader.isolation (default: true) can be set to false
> to disable s3a classloader isolation;
> This can assist in using custom credential providers and other extension
> points.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
