Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3731#discussion_r22044849
--- Diff: docs/job-scheduling.md ---
@@ -56,6 +56,112 @@ the same RDDs. For example, the
[Shark](http://shark.cs.berkeley.edu) JDBC serve
queries. In future releases, in-memory storage systems such as
[Tachyon](http://tachyon-project.org) will
provide another approach to share RDDs.
+## Dynamic Resource Allocation
+
+Spark 1.2 introduces the ability to dynamically scale the set of cluster
resources allocated to
+your application up and down based on the workload. This means that your
application may give
+resources back to the cluster if they are no longer used and request them
again later when there
+is demand. This feature is particularly useful if multiple applications
share resources in your
+Spark cluster. If a subset of the resources allocated to an application
becomes idle, it can be
+returned to the cluster's pool of resources and acquired by other
applications. In Spark, dynamic
+resource allocation is performed on the granularity of the executor and
can be enabled through
+`spark.dynamicAllocation.enabled`.
+
+This feature is currently disabled by default and available only on
[YARN](running-on-yarn.html).
+A future release will extend this to [standalone
mode](spark-standalone.html) and
+[Mesos coarse-grained mode](running-on-mesos.html#mesos-run-modes). Note
that although Spark on
+Mesos already has a similar notion of dynamic resource sharing in
fine-grained mode, enabling
+dynamic allocation allows your Mesos application to take advantage of
coarse-grained low-latency
+scheduling while sharing cluster resources efficiently.
+
+Lastly, it is worth noting that Spark's dynamic resource allocation
mechanism is cooperative.
+This means if a Spark application enables this feature, other applications
on the same cluster
+are also expected to do so. Otherwise, the cluster's resources will end up
being unfairly
+distributed to the applications that do not voluntarily give up unused
resources they have
+acquired.
+
+### Configuration and Setup
+
+All configurations used by this feature live under the
`spark.dynamicAllocation.*` namespace.
+To enable this feature, your application must set
`spark.dynamicAllocation.enabled` to `true` and
+provide lower and upper bounds for the number of executors through
+`spark.dynamicAllocation.minExecutors` and
`spark.dynamicAllocation.maxExecutors`. Other relevant
+configurations are described on the [configurations
page](configuration.html#dynamic-allocation)
+and in the subsequent sections in detail.
+
+Additionally, your application must use an external shuffle service
(described below). To enable
+this, set `spark.shuffle.service.enabled` to `true`. In YARN, this
external shuffle service is
+implemented in `org.apache.spark.yarn.network.YarnShuffleService` that
runs in each `NodeManager`
--- End diff --
Should this be broken out into a separate section for users that don't care
about dynamic allocation, but want to learn how to use the external shuffle
service?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
