Hey Sparkies... I have an odd "bug".
I am running Spark 0.9.2 on Amazon EC2 machines as a job (i.e. not in REPL) After a bunch of processing, I tell spark to save my rdd to S3 using: rdd.saveAsSequenceFile(uri,codec) That line of code hangs. By hang I mean (a) Spark stages UI shows no update on that task succeeded (b) Pushing into that stage shows "No task have reported metrics yet" (c) Ganglia shows the cpu, network access etc at nil (d) No error logs on master or slave. The system just does nothing. What makes it weirder is if I modify the code to either.. (1) rdd.coalesce(41, true) // i.e. increase the num partitions by 1 OR (2) rdd.coalesce(39, false) // decrease by 1 with no shuffle It runs through like a charm... ?? Any ideas for debugging? shay
