Marcelo Vanzin created SPARK-27963:
--------------------------------------
Summary: Allow dynamic allocation without an external shuffle
service
Key: SPARK-27963
URL: https://issues.apache.org/jira/browse/SPARK-27963
Project: Spark
Issue Type: New Feature
Components: Spark Core
Affects Versions: 3.0.0
Reporter: Marcelo Vanzin
It would be useful for users to be able to enable dynamic allocation without
the need to provision an external shuffle service. One immediate use case is
the ability to use dynamic allocation on Kubernetes, which doesn't yet have
that service.
This has been suggested before (e.g.
https://github.com/apache/spark/pull/24083, which was attached to the
k8s-specific SPARK-24432), and can actually be done without affecting the
internals of the Spark scheduler (aside from the dynamic allocation code).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
