Try depending on "spark-streaming-kafka-0-10_2.11" (not the assembly)
On Wed, Mar 29, 2017 at 9:59 AM, Srikanth <srikanth...@gmail.com> wrote: > Hello, > > I'm trying to use "org.json4s" % "json4s-native" library in a spark > streaming + kafka direct app. > When I use the latest version of the lib I get an error similar to this > <https://github.com/json4s/json4s/issues/316> > The work around suggest there is to use version 3.2.10. As spark has a > hard dependency on this version. > > I forced this version in SBT with > dependencyOverrides += "org.json4s" %% "json4s-native" % "3.2.10" > > But now it seems to have some conflict with spark-streaming-kafka-0-10- > assembly > > [error] (*:assembly) deduplicate: different file contents found in the > following: > > [error] C:\Users\stati\.ivy2\cache\org.apache.spark\spark- > streaming-kafka-0-10-assembly_2.11\jars\spark-streaming- > kafka-0-10-assembly_2.11-2.1.0.jar:scala/util/parsing/combinator/ > ImplicitConversions$$anonfun$flatten2$1.class > > [error] C:\Users\stati\.ivy2\cache\org.scala-lang.modules\scala- > parser-combinators_2.11\bundles\scala-parser-combinators_2.11-1.0.4.jar: > scala/util/parsing/combinator/ImplicitConversions$$anonfun$ > flatten2$1.class > > DependencyTree didn't show spark-streaming-kafka-0-10-assembly pulling > json4s-native. > Any idea how to resolve this? I'm using spark version 2.1.0 > > Thanks, > Srikanth >
