Hi, I might be wrong, but need your help.
My understanding in Giraph is that, it doesn't write the intermediate data to disk while sending messages to different machines. But in SPARK, I see that intermediate map outputs gets written to disk. Why does SPARK write intermediate data to disk ? What happens at reducer side ? Does SPARK write the data again to disk ? How does it differ from Hadoop MR ? Can't SPARK communicate everything in memory ? If my understanding is wrong. Please do correct me. Regards, Suman Bharadwaj S
