[
https://issues.apache.org/jira/browse/KUDU-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Evans updated KUDU-3157:
-----------------------------
Description:
It seems that sometime between versions {{1.7.0}} and {{1.12.0}} of
{{org.apache.kudu:kudu-spark2_2.11}}, some SLF4J classes started being included.
These should be removed from the jar, or at least relocated so as not to clash
with Spark vendor dependencies, which can cause {{LinkageError}} in Spark
clusters. Real life example of this problem, seen on a Spark job started on
CDH 6.3 cluster:
{code}
Exception in thread "main" java.lang.LinkageError: loader constraint violation:
when resolving method
"org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;"
the class loader (instance of org/apache/spark/util/ChildFirstURLClassLoader)
of the current class, org/slf4j/LoggerFactory, and the class loader (instance
of sun/misc/Launcher$AppClassLoader) for the method\'s defining class,
org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type
org/slf4j/ILoggerFactory used in the signature
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:423)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
{code}
was:
It seems that sometime between versions {{1.7.0}} and {{1.12.0}} of
{{org.apache.kudu:kudu-spark2_2.11}}, some SLF4J classes started begin included.
These should be removed from the jar, or at least relocated so as not to clash
with Spark vendor dependencies, which can cause {{LinkageError}} in Spark
clusters. Real life example of this problem, seen on a Spark job started on
CDH 6.3 cluster:
{code}
Exception in thread "main" java.lang.LinkageError: loader constraint violation:
when resolving method
"org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;"
the class loader (instance of org/apache/spark/util/ChildFirstURLClassLoader)
of the current class, org/slf4j/LoggerFactory, and the class loader (instance
of sun/misc/Launcher$AppClassLoader) for the method\'s defining class,
org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type
org/slf4j/ILoggerFactory used in the signature
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:423)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
{code}
> kudu-spark2 jar is now including slf4j classes
> ----------------------------------------------
>
> Key: KUDU-3157
> URL: https://issues.apache.org/jira/browse/KUDU-3157
> Project: Kudu
> Issue Type: Bug
> Components: spark
> Affects Versions: 1.12.0
> Reporter: Jeff Evans
> Priority: Major
>
> It seems that sometime between versions {{1.7.0}} and {{1.12.0}} of
> {{org.apache.kudu:kudu-spark2_2.11}}, some SLF4J classes started being
> included.
> These should be removed from the jar, or at least relocated so as not to
> clash with Spark vendor dependencies, which can cause {{LinkageError}} in
> Spark clusters. Real life example of this problem, seen on a Spark job
> started on CDH 6.3 cluster:
> {code}
> Exception in thread "main" java.lang.LinkageError: loader constraint
> violation: when resolving method
> "org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;"
> the class loader (instance of
> org/apache/spark/util/ChildFirstURLClassLoader) of the current class,
> org/slf4j/LoggerFactory, and the class loader (instance of
> sun/misc/Launcher$AppClassLoader) for the method\'s defining class,
> org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type
> org/slf4j/ILoggerFactory used in the signature
> at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:423)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
