[
https://issues.apache.org/jira/browse/SPARK-30978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243831#comment-17243831
]
Hyukjin Kwon commented on SPARK-30978:
--------------------------------------
I changed this to 3.2.0, [~jiangxb1987]
> Remove multiple workers on the same host support from Standalone backend
> ------------------------------------------------------------------------
>
> Key: SPARK-30978
> URL: https://issues.apache.org/jira/browse/SPARK-30978
> Project: Spark
> Issue Type: Task
> Components: Spark Core
> Affects Versions: 3.1.0
> Reporter: Xingbo Jiang
> Assignee: Xingbo Jiang
> Priority: Major
>
> Based on our experience, there is no scenario that necessarily requires
> deploying multiple Workers on the same node with Standalone backend. A worker
> should book all the resources reserved to Spark on the host it is launched,
> then it can allocate those resources to one or more executors launched by
> this worker. Since each executor runs in a separated JVM, we can limit the
> memory of each executor to avoid long GC pause.
> The remaining concern is the local-cluster mode is implemented by launching
> multiple workers on the local host, we might need to re-implement
> LocalSparkCluster to launch only one Worker and multiple executors. It should
> be fine because local-cluster mode is only used in running Spark unit test
> cases, thus end users should not be affected by this change.
> Removing multiple workers on the same host support could simplify the deploy
> model of Standalone backend, and also reduce the burden to support legacy
> deploy pattern in the future feature developments.
> The proposal is to update the document to deprecate the support of system
> environment `SPARK_WORKER_INSTANCES` in 3.0, and remove the support in the
> next major version (3.1.0).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
