Hi Patrick, Sorry I don’t have access to web UI. So I have been running these jobs on larger servers and letting them run.. I have observed that when I run a job with “local[12]”, it runs for some time on full throttle at 1200% CPU consumptions, but after some this processing goes to 0%. After few seconds it again starts processing and goes to high percentage of CPU utilization. This cycle repeats till the job is completed. Ironically I observed similar behavior simple “local” jobs.
Is it the nature of the job that is causing this? I am processing a 70GB file and performing simple map and reduce operations. I am sufficient 100GB ram. Any thoughts? Vijay Gaikwad University of Washington MSIM [email protected] (206) 261-5828 On Nov 25, 2013, at 11:43 AM, Patrick Wendell <[email protected]> wrote: > When it gets stuck, what does it show in the web UI? Also, can you run > a jstack on the process and attach the output... that might explain > what's going on. > > On Mon, Nov 25, 2013 at 11:30 AM, Vijay Gaikwad <[email protected]> wrote: >> I am using apache spark 0.8.0 to process a large data file and perform some >> basic .map and.reduceByKey operations on the RDD. >> >> Since I am using a single machine with multiple processors, I mention >> local[8] in the Master URL field while creating SparkContext >> >> val sc = new SparkContext("local[8]", "Tower-Aggs", SPARK_HOME ) >> >> But whenever I mention multiple processors, the job gets stuck >> (pauses/halts) randomly. There is no definite place where it gets stuck, its >> just random. Sometimes it won't happen at all. I am not sure if it continues >> after that but it gets stuck for a long time after which I abort the job. >> >> But when I just use local in place of local[8], the job runs seamlessly >> without getting stuck ever. >> >> val sc = new SparkContext("local", "Tower-Aggs", SPARK_HOME ) >> >> I am not able to understand where is the problem. >> >> I am using Scala 2.9.3 and sbt to build and run the application >> >> >> - >> http://stackoverflow.com/questions/20187048/apache-spark-localk-master-url-job-gets-stuck >> >> Thx >> Vijay Gaikwad >> University of Washington MSIM >> [email protected] >> (206) 261-5828 >>
