This is the basic design of spark that it runs all actions in different stages... Not sure you can achieve what you r looking for. On Jul 3, 2015 12:43 PM, "Marius Danciu" <[email protected]> wrote:
> Hi all, > > If I have something like: > > rdd.join(...).mapPartitionToPair(...) > > It looks like mapPartitionToPair runs in a different stage then join. Is > there a way to piggyback this computation inside the join stage ? ... such > that each result partition after join is passed to > the mapPartitionToPair function, all running in the same state without any > other costs. > > Best, > Marius >
