Github user mgummelt commented on a diff in the pull request:
https://github.com/apache/spark/pull/11047#discussion_r51766078
--- Diff: docs/running-on-mesos.md ---
@@ -246,15 +246,15 @@ In either case, HDFS runs separately from Hadoop
MapReduce, without being schedu
# Dynamic Resource Allocation with Mesos
-Mesos supports dynamic allocation only with coarse grain mode, which can
resize the number of executors based on statistics
-of the application. While dynamic allocation supports both scaling up and
scaling down the number of executors, the coarse grain scheduler only supports
scaling down
-since it is already designed to run one executor per slave with the
configured amount of resources. However, after scaling down the number of
executors the coarse grain scheduler
-can scale back up to the same amount of executors when Spark signals more
executors are needed.
-
-Users that like to utilize this feature should launch the Mesos Shuffle
Service that
-provides shuffle data cleanup functionality on top of the Shuffle Service
since Mesos doesn't yet support notifying another framework's
-termination. To launch/stop the Mesos Shuffle Service please use the
provided sbin/start-mesos-shuffle-service.sh and
sbin/stop-mesos-shuffle-service.sh
-scripts accordingly.
+Mesos supports dynamic allocation only with coarse grain mode, which can
resize the number of
+executors based on statistics of the application. For general information,
+see [Dynamic Resource
Allocation](job-scheduling.html#dynamic-resource-allocation).
+
+To be able to use the dynamic allocation support, an External Shuffle
Service is required.
+On Mesos, users should launch the Mesos Shuffle Service. It provides
shuffle data cleanup functionality
--- End diff --
Saying they should run it "On Mesos" is a big confusing, since they don't
technically need to. They could run it out of band. The instructions in the
"Job Scheduling" page are pretty clear, so maybe we should say something like
that here:
In Mesos coarse-grained mode, run start-mesos-shuffle-service.sh on all
slave nodes. For instance, you may do so through Marathon.
I'd actually prefer leaving this entire paragraph out, since it just
repeats instructions from the "Job Scheduling" page linked above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
