I'm not sure what your use case is, but perhaps you could use mapPartitions to reduce across the individual partitions and apply your filtering. Then you can finish with a reduceByKey.
On Thu, Feb 19, 2015 at 9:21 AM, Debasish Das <[email protected]> wrote: > Hi, > > Before I send out the keys for network shuffle, in reduceByKey after map + > combine are done, I would like to filter the keys based on some threshold... > > Is there a way to get the key, value after map+combine stages so that I > can run a filter on the keys ? > > Thanks. > Deb > -- Daniel Siegmann, Software Developer Velos Accelerating Machine Learning 54 W 40th St, New York, NY 10018 E: [email protected] W: www.velos.io
