Thanks for the tip. That worked. When would one use the assembly? On Wed, Mar 29, 2017 at 7:13 PM, Tathagata Das <tathagata.das1...@gmail.com> wrote:
> Try depending on "spark-streaming-kafka-0-10_2.11" (not the assembly) > > On Wed, Mar 29, 2017 at 9:59 AM, Srikanth <srikanth...@gmail.com> wrote: > >> Hello, >> >> I'm trying to use "org.json4s" % "json4s-native" library in a spark >> streaming + kafka direct app. >> When I use the latest version of the lib I get an error similar to this >> <https://github.com/json4s/json4s/issues/316> >> The work around suggest there is to use version 3.2.10. As spark has a >> hard dependency on this version. >> >> I forced this version in SBT with >> dependencyOverrides += "org.json4s" %% "json4s-native" % "3.2.10" >> >> But now it seems to have some conflict with spark-streaming-kafka-0-10-ass >> embly >> >> [error] (*:assembly) deduplicate: different file contents found in the >> following: >> >> [error] C:\Users\stati\.ivy2\cache\org.apache.spark\spark-streaming- >> kafka-0-10-assembly_2.11\jars\spark-streaming-kafka-0-10- >> assembly_2.11-2.1.0.jar:scala/util/parsing/combinator/Implic >> itConversions$$anonfun$flatten2$1.class >> >> [error] C:\Users\stati\.ivy2\cache\org.scala-lang.modules\scala-pars >> er-combinators_2.11\bundles\scala-parser-combinators_2.11- >> 1.0.4.jar:scala/util/parsing/combinator/ImplicitConversions >> $$anonfun$flatten2$1.class >> >> DependencyTree didn't show spark-streaming-kafka-0-10-assembly pulling >> json4s-native. >> Any idea how to resolve this? I'm using spark version 2.1.0 >> >> Thanks, >> Srikanth >> > >
