[
https://issues.apache.org/jira/browse/HADOOP-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mkv updated HADOOP-18090:
-------------------------
Description:
Spark 3.2.0 transitively introduces hadoop-client-api and hadoop-client-runtime
dependencies.
When we create a SFTPFileSystem instance
(org.apache.hadoop.fs.sftp.SFTPFileSystem) it tries to load the relocated
classes from _com.jcraft.jsch_ package.
The filesystem instance creation fails with error:
{code:java}
java.lang.ClassNotFoundException:
org.apache.hadoop.shaded.com.jcraft.jsch.SftpException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) {code}
Excluding client from transitive load of spark and directly using
hadoop-common/hadoop-client is the way its working for us.
was:
Spark 3.2.0 transitively introduces hadoop-client-api and hadoop-client-runtime
dependencies.
When we create a SFTPFileSystem instance
(org.apache.hadoop.fs.sftp.SFTPFileSystem) it tries to load the relocated
classes from _com.jcraft.jsch_ package.
The filesystem instance creation fails with error:
{code:java}
java.lang.ClassNotFoundException:
org.apache.hadoop.shaded.com.jcraft.jsch.SftpException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) {code}
> Exclude com/jcraft/jsch classes from being shaded/relocated
> -----------------------------------------------------------
>
> Key: HADOOP-18090
> URL: https://issues.apache.org/jira/browse/HADOOP-18090
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 3.3.1
> Reporter: mkv
> Priority: Major
>
> Spark 3.2.0 transitively introduces hadoop-client-api and
> hadoop-client-runtime dependencies.
> When we create a SFTPFileSystem instance
> (org.apache.hadoop.fs.sftp.SFTPFileSystem) it tries to load the relocated
> classes from _com.jcraft.jsch_ package.
> The filesystem instance creation fails with error:
> {code:java}
> java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.com.jcraft.jsch.SftpException
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) {code}
> Excluding client from transitive load of spark and directly using
> hadoop-common/hadoop-client is the way its working for us.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
