2 is added every time the final partition aggregator is called. The result
of summing the elements across partitions is 9 of course. If you force a
single partition (using spark-shell in local mode):
scala val data = sc.parallelize(List(2,3,4),1)
scala data.aggregate(0)((x,y) = x+y,(x,y) = 2+x+y)
res0: Int = 11
The 2nd function is still called, even though there is only one partition
(presumably either x or y is set to 0).
For every additional partition you specify as the 2nd arg. to parallelize,
the 2nd function will be called again:
scala val data = sc.parallelize(List(2,3,4),1)
scala data.aggregate(0)((x,y) = x+y,(x,y) = 2+x+y)
res0: Int = 11
scala val data = sc.parallelize(List(2,3,4),2)
scala data.aggregate(0)((x,y) = x+y,(x,y) = 2+x+y)
res0: Int = 13
scala val data = sc.parallelize(List(2,3,4),3)
scala data.aggregate(0)((x,y) = x+y,(x,y) = 2+x+y)
res0: Int = 15
scala val data = sc.parallelize(List(2,3,4),4)
scala data.aggregate(0)((x,y) = x+y,(x,y) = 2+x+y)
res0: Int = 17
Hence, it appears that not specifying the 2nd argument resulted in 4
partitions, even though you only had three elements in the list.
If p_i is the ith partition, the final sum appears to be:
(2 + ... (2 + (2 + (2 + 0 + p_1) + p_2) + p_3) ...)
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
http://shop.oreilly.com/product/0636920033073.do (O'Reilly)
Typesafe http://typesafe.com
@deanwampler http://twitter.com/deanwampler
http://polyglotprogramming.com
On Sun, Mar 22, 2015 at 8:05 AM, ashish.usoni ashish.us...@gmail.com
wrote:
Hi ,
I am not able to understand how aggregate function works, Can some one
please explain how below result came
I am running spark using cloudera VM
The result in below is 17 but i am not able to find out how it is
calculating 17
val data = sc.parallelize(List(2,3,4))
data.aggregate(0)((x,y) = x+y,(x,y) = 2+x+y)
*res21: Int = 17*
Also when i try to change the 2nd parameter in sc.parallelize i get
different result
val data = sc.parallelize(List(2,3,4),2)
data.aggregate(0)((x,y) = x+y,(x,y) = 2+x+y)
*res21: Int = 13*
Thanks for the help.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-Does-aggregate-work-tp22179.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
