JoshRosen commented on issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to be deserialized every time URL: https://github.com/apache/spark/pull/22065#issuecomment-511240848 I agree that the added complexity does not seem justified by the reported performance gains: the change here makes the code harder to understand (and the code comments are no longer clear; they'd need to be updated). As a result, I'm going to close this PR and JIRA as "Won't Fix". In general, if we want to optimize task launch latency / throughput then I think the right way to go about that is to develop a set of good microbenchmarks and use those to drive the optimization efforts; end-to-end benchmarks may be too noisy / non-reproducible for this type of change.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
