Hello group, Any ideas on this issue ? Any pointers would be appreciated.
Regards, Ramkumar Chokkalingam, Masters Student, University of Washington || 206-747-3515 <http://www.linkedin.com/in/mynameisram> On Mon, Oct 14, 2013 at 12:30 PM, Ramkumar Chokkalingam < [email protected]> wrote: > Hello Group, > > I'm using Spark 0.8.0 with Scala 2.9.3. > > Two issues - > > > *1) Job hangs when the number of files increases > 4000 * > > First, I was using "local" as an argument for for the Master URL like > here, > > *val sc = new SparkContext("local", "AnonApp", "/usr/local/spark/") > * > *// Read all files in a directory * > *var t = sc.textFile(fileName)* > *t. map(each_line => some_functions(each_line)).saveAsTextFile("/output/" > + filename)* > > My job runs fine for sample inputs (~20 files ) but when the number of > input files increases (~7000 files ) the program execution stops around > 4000 files [once hanged at 4200 files and once at 4216 files ]. > > This is my console , > > "13/10/14 11:54:08 INFO mapred.FileOutputCommitter: Saved output of task > 'attempt_201310141154_0000_m_000000_4212' to > file:/usr/local/spark/ram_examples/AnonApp/sample-ANON/MSC/06/MAZ0320111206074848911831.cdr.gz > 13/10/14 11:54:08 INFO mapred.FileInputFormat: Total input paths to > process : 1 > 13/10/14 11:54:08 INFO mapred.FileOutputCommitter: Saved output of task > 'attempt_201310141154_0000_m_000000_4213' to > file:/usr/local/spark/ram_examples/AnonApp/sample-ANON/MSC/06/KBL0320111206154040475980.cdr.gz > 13/10/14 11:54:08 INFO mapred.FileInputFormat: Total input paths to > process : 1" > __<Control waits here> > > When the job hangs, I checked the output folder , the _temporary file is > created but I'm not sure why the program hangs there. The control > stops/waits like this, > > I saw one post on user group and it suggested me to increase my *ulimit*(on > number of open files) - but my ulimit is already set to unlimited. > > 2 ) When I change the Master URI to local[2], where I have 2 cores. > > My earlier said works fine for sample inputs of 20 files. But the same > program when changed from local to local[2] in SparkContext, hangs in the > same fashion like the one shown above. While making the change (local -> > local[2]) am I expected to make any other change ? > > > Is there any pattern between both these failures ? Apart from the console > logs ? Is there a place where I can see the logs to understand what is > going on when the program hangs ? > > > > Regards, > > Ram. > >
