Re: Spark ReduceByKey - Working in Java

2014-08-02 Thread Sean Owen
I think your questions revolve around the reduce function here, which is a function of 2 arguments returning 1, whereas in a Reducer, you implement a function of many-to-many. This API is simpler if less general. Here you provide an associative operation that can reduce any 2 values down to 1 (e.g

Spark ReduceByKey - Working in Java

2014-08-02 Thread Anil Karamchandani
Hi, I am a complete newbie to spark and map reduce frameworks and have a basic question on the reduce function. I was working on the word count example and was stuck at the reduce stage where the sum happens. I am trying to understand the working of the reducebykey in Spark using java as the prog