[
https://issues.apache.org/jira/browse/HADOOP-19733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18032340#comment-18032340
]
ASF GitHub Bot commented on HADOOP-19733:
-----------------------------------------
brandonvin opened a new pull request, #8048:
URL: https://github.com/apache/hadoop/pull/8048
…lassloader
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
Follow-up to
[HADOOP-18993](https://issues.apache.org/jira/browse/HADOOP-18993) and
[HADOOP-19733](https://issues.apache.org/jira/browse/HADOOP-19733) before it.
With `fs.s3a.classloader.isolation` set to `false` in a Spark application,
it was still impossible to load a credentials provider class from the Spark
application jar.
`fs.s3a.classloader.isolation` works by saving a reference to the intended
classloader in the `Configuration`.
However, loading credentials providers goes through
`S3AUtils#getInstanceFromReflection`, which always used the classloader that
loaded `S3AUtils`.
With this patch, credentials providers will be loaded using the
`Configuration`'s classloader.
### How was this patch tested?
Unit tests in
`org.apache.hadoop.fs.s3a.ITestS3AFileSystemIsolatedClassloader`.
Manual testing in a Spark application.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [x] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> S3A: Credentials provider classes not found despite setting
> `fs.s3a.classloader.isolation` to `false`
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-19733
> URL: https://issues.apache.org/jira/browse/HADOOP-19733
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.4.2
> Reporter: Brandon
> Priority: Minor
>
> HADOOP-18993 added the option `fs.s3a.classloader.isolation` to support, for
> example, a Spark job using an AWS credentials provider class that is bundled
> into the Spark job JAR. In testing this, the AWS credentials provider classes
> are still not found.
> I think the cause is:
> * `fs.s3a.classloader.isolation` is implemented by setting (or not setting)
> a classloader on the `Configuration`
> * However, code paths to load AWS credential provider call
> `S3AUtils.getInstanceFromReflection`, which uses the classloader that loaded
> the S3AUtils class. That's likely to be the built-in application classloader,
> which won't be able to load classes in a Spark job JAR.
> And the fix seems small:
> * Change `S3AUtils.getInstanceFromReflection` to load classes using the
> `Configuration`'s classloader. Luckily we already have the Configuration in
> this method.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
