For others who may be having a similar problem: The error below occurs when using Yarn, which uses an earlier version of Guava compared to Spark 1.1.0. When packaging using Maven, if you put the Yarn dependency above the Spark dependency, the earlier version of guava is the one that gets recognized, leading to the NoSuchMethodError. I solved the problem by simply switching the order of the dependencies..
Rahul From: Akhil Das <[email protected]<mailto:[email protected]>> Date: Wednesday, December 3, 2014 at 4:02 AM To: Rahul Swaminathan <[email protected]<mailto:[email protected]>> Cc: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: WordCount fails in .textFile() method Resent-From: <[email protected]<mailto:[email protected]>> dump your classpath, looks like you have multiple versions of guava jars in the classpath. Thanks Best Regards On Wed, Dec 3, 2014 at 2:30 PM, Rahul Swaminathan <[email protected]<mailto:[email protected]>> wrote: I've tried that and the same error occurs. Do you have any other suggestions? Thanks! Rahul From: Akhil Das <[email protected]<mailto:[email protected]>> Date: Wednesday, December 3, 2014 at 3:55 AM To: Rahul Swaminathan <[email protected]<mailto:[email protected]>> Cc: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: WordCount fails in .textFile() method Resent-From: <[email protected]<mailto:[email protected]>> Try running it in local mode. Looks like a jar conflict/missing. SparkConf conf = new SparkConf().setAppName("JavaWordCount"); conf.set("spark.io.compression.codec","org.apache.spark.io.LZ4CompressionCodec"); conf.setMaster("local[2]").setSparkHome(System.getenv("SPARK_HOME")); JavaSparkContext jsc = new JavaSparkContext(conf); jsc.addJar("myJar.jar"); new JavaWordCount(jsc).doJob(); Thanks Best Regards On Wed, Dec 3, 2014 at 2:49 AM, Rahul Swaminathan <[email protected]<mailto:[email protected]>> wrote: Hi, I am trying to run JavaWordCount without using the spark-submit script. I have copied the source code for JavaWordCount and am using a JavaSparkContext with the following: SparkConf conf = new SparkConf().setAppName("JavaWordCount"); conf.set("spark.io.compression.codec","org.apache.spark.io.LZ4CompressionCodec"); conf.setMaster("spark://127.0.0.1:7077<http://127.0.0.1:7077>").setSparkHome(System.getenv("SPARK_HOME")); JavaSparkContext jsc = new JavaSparkContext(conf); jsc.addJar("myJar.jar"); new JavaWordCount(jsc).doJob(); I am getting the following error in the .textFile() method: Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.hash.HashFunction.hashInt(I)Lcom/google/common/hash/HashCode; at org.apache.spark.util.collection.OpenHashSet.org<http://org.apache.spark.util.collection.OpenHashSet.org>$apache$spark$util$collection$OpenHashSet$$hashcode(OpenHashSet.scala:261) What can I do to solve this issue? It works fine when running from command line with spark-submit script. Thanks, Rahul
