weixiuli commented on a change in pull request #23393: [SPARK-26288][CORE]add
initRegisteredExecutorsDB
URL: https://github.com/apache/spark/pull/23393#discussion_r266181384
##########
File path: docs/spark-standalone.md
##########
@@ -260,6 +261,17 @@ SPARK_WORKER_OPTS supports the following system
properties:
especially if you run jobs very frequently.
</td>
</tr>
+<tr>
+ <td><spark.shuffle.service.db.enabled</code></td>
+ <td>true</td>
+ <td>
+ Enable record RegisteredExecutors information by leveldb, which can be
reloaded and
+ used again when the external shuffle service is restarted. Note that this
only affects standalone
+ mode, its has always on for yarn. We should Enable
`spark.worker.cleanup.enabled` to remove the entry
+ (It will leave an entry in the DB forever when an application is stopped
while the external shuffle
+ service is down) in the leveldb with WorkDirCleanup. It may be removed in
the future.
Review comment:
Very clear description, thank you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
