Re: Cassandra spark connector exception: "NoSuchMethodError: com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;"
Thanks Helena. I think I will wait for the new release and try it. Again thanks, /Shahab On Tue, Nov 11, 2014 at 3:41 PM, Helena Edelson wrote: > Hi, > It looks like you are building from master > (spark-cassandra-connector-assembly-1.2.0). > - Append this to your com.google.guava declaration: % "provided" > - Be sure your version of the connector dependency is the same as the > assembly build. For instance, if you are using 1.1.0-beta1, build your > assembly with that vs master. > - You can upgrade your version of cassandra if that is plausible for your > deploy environment, to 2.1.0. Side note: we are releasing 1.1.0-beta2 today > or tomorrow which allows usage of Cassandra 2.1.1 and fixes any guava issues > - Make your version of cassandra server + dependencies match your > cassandra driver version. You currently have 2.0.9 with 2.0.4 > > > - Helena > @helenaedelson > > > On Nov 11, 2014, at 6:13 AM, shahab wrote: > > Hi, > > I have a spark application which uses Cassandra > "connectorspark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar" to load > data from Cassandra into spark. > > Everything works fine in the local mode, when I run in my IDE. But when I > submit the application to be executed in standalone Spark server, I get the > following exception, (which is apparently related to Guava versions ???!). > Does any one know how to solve this? > > I create a jar file of my spark application using assembly.bat, and the > followings is the dependencies I used: > > I put the "connectorspark-cassandra-connector-assembly-1.2.0-SNAPSHOT.ja" > in the "lib/" folder of my eclipse project thats why it is not included in > the dependencies > > libraryDependencies ++= Seq( > > "org.apache.spark"%% "spark-catalyst"% "1.1.0" % > "provided", > > "org.apache.cassandra" % "cassandra-all" % "2.0.9" intransitive(), > > "org.apache.cassandra" % "cassandra-thrift" % "2.0.9" intransitive(), > > "net.jpountz.lz4" % "lz4" % "1.2.0", > > "org.apache.thrift" % "libthrift" % "0.9.1" exclude("org.slf4j", > "slf4j-api") exclude("javax.servlet", "servlet-api"), > > "com.datastax.cassandra" % "cassandra-driver-core" % "2.0.4" > intransitive(), > > "org.apache.spark" %% "spark-core" % "1.1.0" % "provided" > exclude("org.apache.hadoop", "hadoop-core"), > > "org.apache.spark" %% "spark-streaming" % "1.1.0" % "provided", > > "org.apache.hadoop" % "hadoop-client" % "1.0.4" % "provided", > > "com.github.nscala-time" %% "nscala-time" % "1.0.0", > > "org.scalatest" %% "scalatest" % "1.9.1" % "test", > > "org.apache.spark" %% "spark-sql" % "1.1.0" % "provided", > > "org.apache.spark" %% "spark-hive" % "1.1.0" % "provided", > > "org.json4s" %% "json4s-jackson" % "3.2.5", > > "junit" % "junit" % "4.8.1" % "test", > > "org.slf4j" % "slf4j-api" % "1.7.7", > > "org.slf4j" % "slf4j-simple" % "1.7.7", > > "org.clapper" %% "grizzled-slf4j" % "1.0.2", > > "log4j" % "log4j" % "1.2.17", > > "com.google.guava" % "guava" % "16.0" > >) > > best, > > /Shahab > And this is the exception I get: > > Exception in thread "main" > com.google.common.util.concurrent.ExecutionError: > java.lang.NoSuchMethodError: > com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set; > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2261) > at com.google.common.cache.LocalCache.get(LocalCache.java:4000) > at > com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004) > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874) > at > org.apache.spark.sql.cassandra.CassandraCatalog.lookupRelation(CassandraCatalog.scala:39) > at org.apache.spark.sql.cassandra.CassandraSQLContext$$anon$2.org > $apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$lookupRelation(CassandraSQLContext.scala:60) > at > org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:123) > at > org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:123) > at scala.Option.getOrElse(Option.scala:120) > at > org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:123) > at > org.apache.spark.sql.cassandra.CassandraSQLContext$$anon$2.lookupRelation(CassandraSQLContext.scala:65) > > >
Re: Cassandra spark connector exception: "NoSuchMethodError: com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;"
Hi, It looks like you are building from master (spark-cassandra-connector-assembly-1.2.0). - Append this to your com.google.guava declaration: % "provided" - Be sure your version of the connector dependency is the same as the assembly build. For instance, if you are using 1.1.0-beta1, build your assembly with that vs master. - You can upgrade your version of cassandra if that is plausible for your deploy environment, to 2.1.0. Side note: we are releasing 1.1.0-beta2 today or tomorrow which allows usage of Cassandra 2.1.1 and fixes any guava issues - Make your version of cassandra server + dependencies match your cassandra driver version. You currently have 2.0.9 with 2.0.4 - Helena @helenaedelson On Nov 11, 2014, at 6:13 AM, shahab wrote: > Hi, > > I have a spark application which uses Cassandra > "connectorspark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar" to load data > from Cassandra into spark. > > Everything works fine in the local mode, when I run in my IDE. But when I > submit the application to be executed in standalone Spark server, I get the > following exception, (which is apparently related to Guava versions ???!). > Does any one know how to solve this? > > I create a jar file of my spark application using assembly.bat, and the > followings is the dependencies I used: > > I put the "connectorspark-cassandra-connector-assembly-1.2.0-SNAPSHOT.ja" in > the "lib/" folder of my eclipse project thats why it is not included in the > dependencies > libraryDependencies ++= Seq( > > "org.apache.spark"%% "spark-catalyst"% "1.1.0" % > "provided", > > "org.apache.cassandra" % "cassandra-all" % "2.0.9" intransitive(), > > "org.apache.cassandra" % "cassandra-thrift" % "2.0.9" intransitive(), > > "net.jpountz.lz4" % "lz4" % "1.2.0", > > "org.apache.thrift" % "libthrift" % "0.9.1" exclude("org.slf4j", > "slf4j-api") exclude("javax.servlet", "servlet-api"), > > "com.datastax.cassandra" % "cassandra-driver-core" % "2.0.4" > intransitive(), > > "org.apache.spark" %% "spark-core" % "1.1.0" % "provided" > exclude("org.apache.hadoop", "hadoop-core"), > > "org.apache.spark" %% "spark-streaming" % "1.1.0" % "provided", > > "org.apache.hadoop" % "hadoop-client" % "1.0.4" % "provided", > > "com.github.nscala-time" %% "nscala-time" % "1.0.0", > > "org.scalatest" %% "scalatest" % "1.9.1" % "test", > > "org.apache.spark" %% "spark-sql" % "1.1.0" % "provided", > > "org.apache.spark" %% "spark-hive" % "1.1.0" % "provided", > > "org.json4s" %% "json4s-jackson" % "3.2.5", > > "junit" % "junit" % "4.8.1" % "test", > > "org.slf4j" % "slf4j-api" % "1.7.7", > > "org.slf4j" % "slf4j-simple" % "1.7.7", > > "org.clapper" %% "grizzled-slf4j" % "1.0.2", > > "log4j" % "log4j" % "1.2.17", > > "com.google.guava" % "guava" % "16.0" > > >) > > > best, > > /Shahab > And this is the exception I get: > > Exception in thread "main" com.google.common.util.concurrent.ExecutionError: > java.lang.NoSuchMethodError: > com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set; > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2261) > at com.google.common.cache.LocalCache.get(LocalCache.java:4000) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004) > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874) > at > org.apache.spark.sql.cassandra.CassandraCatalog.lookupRelation(CassandraCatalog.scala:39) > at > org.apache.spark.sql.cassandra.CassandraSQLContext$$anon$2.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$lookupRelation(CassandraSQLContext.scala:60) > at > org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:123) > at > org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:123) > at scala.Option.getOrElse(Option.scala:120) > at > org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:123) > at > org.apache.spark.sql.cassandra.CassandraSQLContext$$anon$2.lookupRelation(CassandraSQLContext.scala:65)