vanzin commented on issue #24083: [SPARK-24432] Support dynamic allocation without external shuffle service URL: https://github.com/apache/spark/pull/24083#issuecomment-473444559 Ping? I'm seeing two or 3 separate PRs here; the ExecutorAllocationManager changes + tracking of active stages being the most obvious. "Shuffle biased scheduling" builds on top of the active stage tracking, but feels separate from the dynamic allocation changes. And the k8s changes definitely should be a separate PR. Also, instead of adding YARN tests, I'd do that in core. `YarnClusterSuite` does not run when you don't change YARN code, so it often doesn't run when changes are made to the core code in this area. Using a `local-cluster[...]` for this works fine (I've been using it in some code I've been playing with), and even runs a bit faster than tests in the YARN suite.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
