[
https://issues.apache.org/jira/browse/HADOOP-19833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18063813#comment-18063813
]
ASF GitHub Bot commented on HADOOP-19833:
-----------------------------------------
deepujain opened a new pull request, #8303:
URL: https://github.com/apache/hadoop/pull/8303
# PR description — copy into GitHub when opening the PR (do not commit this
file)
## HADOOP-19833. Fix fs.s3a.classloader.isolation=false by using classloader
from Configuration in getInstanceFromReflection
### Problem
The configuration `fs.s3a.classloader.isolation=false` (introduced in
HADOOP-18993) does not work as intended. The isolation flag is stored correctly
in the Hadoop Configuration, but when creating instances via
`S3AUtils.getInstanceFromReflection`, the classloader was always taken from
`conf.getClassLoader()` without considering the isolation flag. When isolation
is disabled, the Configuration may not have had its classloader set (e.g.
`maybeIsolateClassloader` does not set it when isolation=false), so we must
explicitly respect the flag and, when isolation is false, use the
Configuration's classloader when set, otherwise the thread context classloader
so that loading is not forced to use an isolated S3A classloader.
### Change
In `S3AUtils.getInstanceFromReflection`:
- When `conf != null` and `fs.s3a.classloader.isolation` is true (default):
keep using `conf.getClassLoader()` (set by `maybeIsolateClassloader` to the S3A
classloader).
- When `conf != null` and `fs.s3a.classloader.isolation` is false: use the
classloader from the Configuration (`conf.getClassLoader()`); if that is null,
use `Thread.currentThread().getContextClassLoader()` so that classes are loaded
from the application classpath.
- If the resolved classloader is still null, fall back to
`S3AUtils.class.getClassLoader()` for backward compatibility.
This ensures that when users set `fs.s3a.classloader.isolation=false`,
instance creation uses the Configuration's classloader (or the context
classloader) instead of always using the S3A isolated classloader.
### Tests
- Existing test `ITestS3AFileSystemIsolatedClassloader` covers
isolation=true and isolation=false; the test expects that when isolation=false,
`fs.getConf().getClassLoader()` is not the S3A classloader. This fix ensures
the classloader used in `getInstanceFromReflection` is consistent with that
expectation.
### JIRA
Fixes HADOOP-19833
> fs.s3a.classloader.isolation=false configuration does not work - classloader
> not retrieved from Configuration object
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-19833
> URL: https://issues.apache.org/jira/browse/HADOOP-19833
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 3.4.0, 3.5.0, 3.4.1, 3.4.2, 3.4.3, 3.4.4, 3.5.1
> Reporter: Yunzhao Yang
> Priority: Minor
> Fix For: 3.4.0, 3.5.0, 3.4.1, 3.4.2, 3.4.3, 3.4.4, 3.5.1
>
>
> The configuration {{fs.s3a.classloader.isolation=false (introduced from
> https://issues.apache.org/jira/browse/HADOOP-18993)}} does not work as
> intended. While the configuration is correctly set in the Hadoop
> Configuration object, the S3A classloader is never retrieved from the Hadoop
> Configuration object, resulting in classloader isolation always being enabled
> regardless of the {{fs.s3a.classloader.isolation}} value.
>
> The bug occurs in two steps:
> *Configuration is set correctly* - The isolation flag is properly stored in
> the Hadoop Configuration object. Code link:
> [https://github.com/apache/hadoop/blob/release-3.5.0-RC0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java#L1756]
>
> *Classloader is not retrieved from Configuration* - When creating instances,
> the classloader is not read from the Configuration object, instead it is
> always using the S3AUtils's classloader. Code link:
> [https://github.com/apache/hadoop/blob/release-3.5.0-RC0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java#L660]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
