mridulm edited a comment on pull request #30392: URL: https://github.com/apache/spark/pull/30392#issuecomment-729318977
> I think so. In some cases, unnecessary executor-side reduce might invoke an additional map task although it just returns the single element. So this is just a minor concern for me. There will not be an additional map task - it will get pipelined with the `mapPartitions` - with the `iter.reduceLeft` in `reduce` working on a single element. Essentially, I am not sure what this change is buying us. If the concern had been that the driver is handling all the priority queue's - I can see that being an issue (that is a general critique on reduce itself). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
