LuciferYang commented on code in PR #37853:
URL: https://github.com/apache/spark/pull/37853#discussion_r968124459
##########
docs/spark-standalone.md:
##########
@@ -322,9 +322,9 @@ SPARK_WORKER_OPTS supports the following system properties:
<td>true</td>
<td>
Store External Shuffle service state on local disk so that when the
external shuffle service is restarted, it will
- automatically reload info on current executors. This only affects
standalone mode (yarn always has this behavior
- enabled). You should also enable
<code>spark.worker.cleanup.enabled</code>, to ensure that the state
- eventually gets cleaned up. This config may be removed in the future.
+ automatically reload info on current executors. This only affects
standalone mode. You should also enable
+ <code>spark.worker.cleanup.enabled</code>, to ensure that the state
eventually gets cleaned up.
+ This config may be removed in the future.
Review Comment:
- when `yarn.nodemanager.recovery.enabled` is true, `_recoveryPath` and
`registeredExecutorFile` in `YarnShuffleService` will not null, then
`YarnShuffleService` persist data into Level/RocksDB
- when `yarn.nodemanager.recovery.enabled` is false, `_recoveryPath` and
`registeredExecutorFile` in `YarnShuffleService` will null, then
`YarnShuffleService` not persist data into diskstore
The persist behavior of YarnShuffleService is controlled by Yarn's
configuration. It seems that it not related to
`spark.shuffle.service.db.enabled`, so I don't think it is necessary to mention
`yarn always has this behavior enabled` in this configuration description.
If we need add yarn related descriptions here, should we also need mention
`mesos always has this behavior disabled` here...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
