there is no way to avoid shuffle if you use combine by key, no matter if
your data is cached in memory, because the shuffle write must write the
data into disk. And It seem that spark can not guarantee the similar
key(K1) goes to the Container_X.
you can use the tmpfs for your shuffle dir, this can improve your shuffle
write speed.
If the number of worker nodes is enough, then hundreds of GB is not quite
big to deal with.
On Wed, Jan 14, 2015 at 5:30 AM, Puneet Kapoor puneet.cse.i...@gmail.com
wrote:
Hi,
I have a usecase where in I have hourly spark job which creates hourly
RDDs, which are partitioned by keys.
At the end of the day I need to access all of these RDDs and combine the
Key/Value pairs over the day.
If there is a key K1 in RDD0 (1st hour of day), RDD1 ... RDD23(last hour
of the day); we need to combine all the values of this K1 using some logic.
What I want to do is to avoid the shuffling at the end of the day since
the data in huge ~ hundreds of GB.
Questions
---
1.) Is there a way that i can persist hourly RDDs with partition
information and then while reading back the RDDs the partition information
is restored.
2.) Can i ensure that partitioning is similar for different hours. Like if
K1 goes to container_X, it would go to the same container in the next hour
and so on.
Regards
Puneet
