mridulm commented on code in PR #37853:
URL: https://github.com/apache/spark/pull/37853#discussion_r968078477
##########
docs/spark-standalone.md:
##########
@@ -322,9 +322,9 @@ SPARK_WORKER_OPTS supports the following system properties:
<td>true</td>
<td>
Store External Shuffle service state on local disk so that when the
external shuffle service is restarted, it will
- automatically reload info on current executors. This only affects
standalone mode (yarn always has this behavior
- enabled). You should also enable
<code>spark.worker.cleanup.enabled</code>, to ensure that the state
- eventually gets cleaned up. This config may be removed in the future.
+ automatically reload info on current executors. This only affects
standalone mode. You should also enable
+ <code>spark.worker.cleanup.enabled</code>, to ensure that the state
eventually gets cleaned up.
+ This config may be removed in the future.
Review Comment:
Why are we removing the yarn related blurb from here ? Essentially, this
boolean does not control the behavior in yarn - for yarn, that is configured
for the cluster, and inherits the behavior for spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
