RE: mapPartitions versus map overhead?

2013-12-23 Thread Shao, Saisai
-- From: Huan Dao [mailto:huan...@me.com] Sent: Tuesday, December 24, 2013 1:15 PM To: user@spark.incubator.apache.org Subject: mapPartitions versus map overhead? Hi all, is there any overhead of mapPartitions versus overhead, if I implement an algorithm using map -> reduce versus mapPartitio

mapPartitions versus map overhead?

2013-12-23 Thread Huan Dao
Hi all, is there any overhead of mapPartitions versus overhead, if I implement an algorithm using map -> reduce versus mapPartitions -> reduce. Thanks, Huan Dao