I don't think directy .aggregateByKey() can be done, because we will need
count of keys (for average). Maybe we can use .countByKey() which returns a
map and .foldByKey(0)(_+_) (or aggregateByKey()) which gives sum of values
per key. I myself ain't getting how to proceed.
Regards
On Fri, Oct 31, 2014 at 1:26 PM, qinwei wei@dewmobile.net wrote:
Hi, everyone
I have an RDD filled with data like
(k1, v11)
(k1, v12)
(k1, v13)
(k2, v21)
(k2, v22)
(k2, v23)
...
I want to calculate the average and standard deviation of (v11, v12,
v13) and (v21, v22, v23) group by there keys
for the moment, i have done that by using groupByKey and map, I notice
that groupByKey is very expensive, but i can not figure out how to do it
by using aggregateByKey, so i wonder is there any better way to do this?
Thanks!
--
qinwei
