Running Spark Job on Yarn from Java Code

2014-12-16 Thread Rahul Swaminathan
Hi all, I am trying to run a simple Spark_Pi application through Yarn from Java code. I have the Spark_Pi class and everything works fine if I run on Spark. However, when I set master to "yarn-client" and set yarn mode to true, I keep getting exceptions. I suspect this has something to do with

Re: WordCount fails in .textFile() method

2014-12-03 Thread Rahul Swaminathan
recognized, leading to the NoSuchMethodError. I solved the problem by simply switching the order of the dependencies.. Rahul From: Akhil Das mailto:ak...@sigmoidanalytics.com>> Date: Wednesday, December 3, 2014 at 4:02 AM To: Rahul Swaminathan mailto:rahul.swaminat...@duke.edu>

Re: WordCount fails in .textFile() method

2014-12-03 Thread Rahul Swaminathan
I've tried that and the same error occurs. Do you have any other suggestions? Thanks! Rahul From: Akhil Das mailto:ak...@sigmoidanalytics.com>> Date: Wednesday, December 3, 2014 at 3:55 AM To: Rahul Swaminathan mailto:rahul.swaminat...@duke.edu>> Cc: "u...@spark.incubat

WordCount fails in .textFile() method

2014-12-02 Thread Rahul Swaminathan
Hi, I am trying to run JavaWordCount without using the spark-submit script. I have copied the source code for JavaWordCount and am using a JavaSparkContext with the following: SparkConf conf = new SparkConf().setAppName("JavaWordCount"); conf.set("spark.io.compression.codec","org.apache.spark.i

WordCount fails in .textFile() method

2014-12-02 Thread Rahul Swaminathan
Hi, I am trying to run JavaWordCount without using the spark-submit script. I have copied the source code for JavaWordCount and am using a JavaSparkContext with the following: SparkConf conf = new SparkConf().setAppName("JavaWordCount"); conf.set("spark.io.compression.codec","org.apache.spark.i