Github user skyluc commented on a diff in the pull request:
https://github.com/apache/spark/pull/11047#discussion_r51846148
--- Diff: docs/running-on-mesos.md ---
@@ -246,15 +246,15 @@ In either case, HDFS runs separately from Hadoop
MapReduce, without being schedu
# Dynamic Resource Allocation with Mesos
-Mesos supports dynamic allocation only with coarse grain mode, which can
resize the number of executors based on statistics
-of the application. While dynamic allocation supports both scaling up and
scaling down the number of executors, the coarse grain scheduler only supports
scaling down
-since it is already designed to run one executor per slave with the
configured amount of resources. However, after scaling down the number of
executors the coarse grain scheduler
-can scale back up to the same amount of executors when Spark signals more
executors are needed.
-
-Users that like to utilize this feature should launch the Mesos Shuffle
Service that
-provides shuffle data cleanup functionality on top of the Shuffle Service
since Mesos doesn't yet support notifying another framework's
-termination. To launch/stop the Mesos Shuffle Service please use the
provided sbin/start-mesos-shuffle-service.sh and
sbin/stop-mesos-shuffle-service.sh
-scripts accordingly.
+Mesos supports dynamic allocation only with coarse grain mode, which can
resize the number of
+executors based on statistics of the application. For general information,
+see [Dynamic Resource
Allocation](job-scheduling.html#dynamic-resource-allocation).
+
+To be able to use the dynamic allocation support, an External Shuffle
Service is required.
+On Mesos, users should launch the Mesos Shuffle Service. It provides
shuffle data cleanup functionality
--- End diff --
The whole does add one piece of information: that a tweaked version of the
External Shuffle service should be used, to have a full support in Mesos.
I'll try to merge the 2 last paragraphs together.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
