[
https://issues.apache.org/jira/browse/HADOOP-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481871#comment-17481871
]
mkv edited comment on HADOOP-18090 at 1/25/22, 2:57 PM:
--------------------------------------------------------
[~busbey] For the second question why the class isn't reachable, it seems
hadoop-client-api has a compile time dependency on hadoop-common which gets
shaded because of pattern.
However there are many transitive dependencies of hadoop-common which are
excluded. The list of exlusions comprise {_}com.jcraft:jsch{_}.
Hence in SFTPFileSystem, the pattern matching rewrites the imports to relocated
path while at the same time the classes which were supposed to be relocated
were excluded from POM.
Sheer text replacement without any class reloaction.
Regarding first point, SFTPFileSystem is a part of hadoop-common. Now it would
be good to have it land in its own module unlike hadoop-aws.
For fixes, I am new in this space that's why was just checking if this is
indeed a bug. I know in OSS community solves the issues. Not asking you to fix
:).
was (Author: manojkumarvohra9):
[~busbey] For the second question why the class isn't reachable, it seems
hadoop-client-api has a compile time dependency on hadoop-common which gets
shaded because of pattern.
However there are many transitive dependencies of hadoop-common which are
excluded. The list of inclusion include _com.jcraft:jsch_.
Hence in SFTPFileSystem, the pattern matching rewrites the imports to relocated
path while at the same time the classes which were supposed to be relocated
were excluded from POM.
Sheer text replacement without any class reloaction.
Regarding first point, SFTPFileSystem is a part of hadoop-common. Now it would
be good to have it land in its own module unlike hadoop-aws.
For fixes, I am new in this space that's why was just checking if this is
indeed a bug. I know in OSS community solves the issues. Not asking you to fix
:).
> Exclude com/jcraft/jsch classes from being shaded/relocated
> -----------------------------------------------------------
>
> Key: HADOOP-18090
> URL: https://issues.apache.org/jira/browse/HADOOP-18090
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 3.3.1
> Reporter: mkv
> Priority: Major
>
> Spark 3.2.0 transitively introduces hadoop-client-api and
> hadoop-client-runtime dependencies.
> When we create a SFTPFileSystem instance
> (org.apache.hadoop.fs.sftp.SFTPFileSystem) it tries to load the relocated
> classes from _com.jcraft.jsch_ package.
> The filesystem instance creation fails with error:
> {code:java}
> java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.com.jcraft.jsch.SftpException
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) {code}
> Excluding client from transitive load of spark and directly using
> hadoop-common/hadoop-client is the way its working for us.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
