Thanks DB. I will work with mapPartition for now.
Question to the community in general: should we consider adding such an
operation to RDDs especially as a developer API?
On Sun, May 4, 2014 at 1:41 AM, DB Tsai wrote:
> You could easily achieve this by mapPartition. However, it seems that it
You could easily achieve this by mapPartition. However, it seems that it
can not be done by using aggregate type of operation. I can see that it's a
general useful operation. For now, you could use mapPartition.
Sincerely,
DB Tsai
---
My Blog:
I am currently using the RDD aggregate operation to reduce (fold) per
partition and then combine using the RDD aggregate operation.
def aggregate[U: ClassTag](zeroValue: U)(seqOp: (U, T) => U, combOp: (U, U)
=> U): U
I need to perform a transform operation after the seqOp and before the
combOp. Th
