Xingbo Jiang created SPARK-30978:
------------------------------------
Summary: Remove multiple workers on the same host support from
Standalone backend
Key: SPARK-30978
URL: https://issues.apache.org/jira/browse/SPARK-30978
Project: Spark
Issue Type: Task
Components: Spark Core
Affects Versions: 3.0.0, 3.1.0
Reporter: Xingbo Jiang
Assignee: Xingbo Jiang
Based on our experience, there is no scenario that necessarily requires
deploying multiple Workers on the same node with Standalone backend. A worker
should book all the resources reserved to Spark on the host it is launched,
then it can allocate those resources to one or more executors launched by this
worker. Since each executor runs in a separated JVM, we can limit the memory of
each executor to avoid long GC pause.
The remaining concern is the local-cluster mode is implemented by launching
multiple workers on the local host, we might need to re-implement
LocalSparkCluster to launch only one Worker and multiple executors. It should
be fine because local-cluster mode is only used in running Spark unit test
cases, thus end users should not be affected by this change.
Removing multiple workers on the same host support could simplify the deploy
model of Standalone backend, and also reduce the burden to support legacy
deploy pattern in the future feature developments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
