[ https://issues.apache.org/jira/browse/SPARK-30978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243831#comment-17243831 ]
Hyukjin Kwon commented on SPARK-30978: -------------------------------------- I changed this to 3.2.0, [~jiangxb1987] > Remove multiple workers on the same host support from Standalone backend > ------------------------------------------------------------------------ > > Key: SPARK-30978 > URL: https://issues.apache.org/jira/browse/SPARK-30978 > Project: Spark > Issue Type: Task > Components: Spark Core > Affects Versions: 3.1.0 > Reporter: Xingbo Jiang > Assignee: Xingbo Jiang > Priority: Major > > Based on our experience, there is no scenario that necessarily requires > deploying multiple Workers on the same node with Standalone backend. A worker > should book all the resources reserved to Spark on the host it is launched, > then it can allocate those resources to one or more executors launched by > this worker. Since each executor runs in a separated JVM, we can limit the > memory of each executor to avoid long GC pause. > The remaining concern is the local-cluster mode is implemented by launching > multiple workers on the local host, we might need to re-implement > LocalSparkCluster to launch only one Worker and multiple executors. It should > be fine because local-cluster mode is only used in running Spark unit test > cases, thus end users should not be affected by this change. > Removing multiple workers on the same host support could simplify the deploy > model of Standalone backend, and also reduce the burden to support legacy > deploy pattern in the future feature developments. > The proposal is to update the document to deprecate the support of system > environment `SPARK_WORKER_INSTANCES` in 3.0, and remove the support in the > next major version (3.1.0). -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org