Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/7991#discussion_r36421604
--- Diff: docs/configuration.md ---
@@ -474,6 +474,22 @@ Apart from these, the following properties are also
available, and may be useful
</td>
</tr>
<tr>
+ <td><code>spark.shuffle.service.enabled</code></td>
+ <td>false</td>
+ <td>
+ Enables the external shuffle service. This service preserves the
shuffle files written by
+ executors so the executors can be safely removed. This must be enabled
if
--- End diff --
http://spark.apache.org/docs/latest/job-scheduling.html#configuration-and-setup
So on YARN atleast, you have to install the shuffle jar into the YARN
nodemanager as an auxiliary service. Then when you run your application on yarn
you have to turn this config to true to tell it to use that service. If you
haven't installed it then turning this to true will cause the application to
fail.
I haven't looked to see how this is installed in standalone (if its
automatically there).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
