Hi Steve, I have implemented repartitions on dataframe to 1. It helped the performance but not to a great extent. I am also looking for answers from the experts.
Thanks, Asmath On Wed, Apr 12, 2017 at 9:45 AM, Steve Robinson < [email protected]> wrote: > Hi, > > > Does anyone have any optimisation tips or could propose an alternative way > to perform the below: > > > val groupedUserItems1 = userItems1.groupByKey{_.customer_id} > val groupedUserItems2 = userItems2.groupByKey{_.customer_id} > groupedUserItems1.cogroup(groupedUserItems2){ > case (_, userItems1, userItems2) => > processSingleUser(userItems1, userItems2) > } > } > > The userItems1 and userItems2 datasets are quite large (100's millions of > records) so I'm finding the shuffle stage is shuffing Gigabytes of data. > > Any help would be greatly appreciated. > > Thanks, > > > Steve Robinson > > [email protected] > 0131 290 2300 > > > www.aquilainsight.com > linkedin.com/aquilainsight > <https://www.linkedin.com/company/aquila-insight> > > twitter.com/aquilainsight >
