Hi Marcelo, When I add this exclusion rule to my pom: <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>1.4.1</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency>
The SparkRunner class works fine (from IntelliJ) but when I build a jar and submit it to spark-submit: I get this error: Caused by: java.lang.ClassCastException: org.slf4j.impl.Log4jLoggerFactory cannot be cast to ch.qos.logback.classic.LoggerContext at com.opentable.logging.AssimilateForeignLogging.assimilate(AssimilateForeignLogging.java:68) at com.opentable.logging.AssimilateForeignLoggingHook.automaticAssimilationHook(AssimilateForeignLoggingHook.java:28) at com.opentable.logging.Log.<clinit>(Log.java:31) Which is this here (our logging lib is open sourced): https://github.com/opentable/otj-logging/blob/master/logging/src/main/java/com/opentable/logging/AssimilateForeignLogging.java#L68 Thanks, -Utkarsh On Mon, Aug 24, 2015 at 3:04 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > Hi Utkarsh, > > Unfortunately that's not going to be easy. Since Spark bundles all > dependent classes into a single fat jar file, to remove that > dependency you'd need to modify Spark's assembly jar (potentially in > all your nodes). Doing that per-job is even trickier, because you'd > probably need some kind of script to inject the correct binding into > Spark's classpath. > > That being said, that message is not an error, it's more of a noisy > warning. I'd expect slf4j to use the first binding available - in your > case, logback-classic. Is that not the case? > > > On Mon, Aug 24, 2015 at 2:50 PM, Utkarsh Sengar <utkarsh2...@gmail.com> > wrote: > > Continuing this discussion: > > > http://apache-spark-user-list.1001560.n3.nabble.com/same-log4j-slf4j-error-in-spark-9-1-td5592.html > > > > I am getting this error when I use logback-classic. > > > > SLF4J: Class path contains multiple SLF4J bindings. > > SLF4J: Found binding in > > > [jar:file:.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] > > SLF4J: Found binding in > > > [jar:file:.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > > > > I need to use logback-classic for my current project, so I am trying to > > ignore "slf4j-log4j12" from spark: > > <dependency> > > <groupId>org.apache.spark</groupId> > > <artifactId>spark-core_2.10</artifactId> > > <version>1.4.1</version> > > <exclusions> > > <exclusion> > > <groupId>org.slf4j</groupId> > > <artifactId>slf4j-log4j12</artifactId> > > </exclusion> > > </exclusions> > > </dependency> > > > > Now, when I run my job from Intellij (which sets the classpath), things > work > > perfectly. > > > > But when I run my job via spark-submit: > > ~/spark-1.4.1-bin-hadoop2.4/bin/spark-submit --class runner.SparkRunner > > spark-0.1-SNAPSHOT-jar-with-dependencies.jar > > My job fails because spark-submit sets up the classpath and it re-adds > the > > slf4j-log4j12. > > > > I am not adding spark jar to the uber-jar via the maven assembly plugin: > > <dependencySets> > > <dependencySet> > > ...... > > <useTransitiveDependencies>false</useTransitiveDependencies> > > <excludes> > > <exclude>org.apache.spark:spark-core_2.10</exclude> > > </excludes> > > </dependencySet> > > </dependencySets> > > > > So how can I exclude "slf4j-log4j12.jar" when I submit a job via > > spark-submit (on a per job basis)? > > > > -- > > Thanks, > > -Utkarsh > > > > -- > Marcelo > -- Thanks, -Utkarsh