srowen commented on a change in pull request #28258:
URL: https://github.com/apache/spark/pull/28258#discussion_r418974650
##########
File path: docs/spark-standalone.md
##########
@@ -374,6 +374,24 @@ To run an interactive Spark shell against the cluster, run
the following command
You can also pass an option `--total-executor-cores <numCores>` to control the
number of cores that spark-shell uses on the cluster.
+#Spark Properties
+
+Spark applications supports the following configuration properties specific to
standalone Mode:
+<table class="table">
Review comment:
Nit: Mode -> mode
and newline before the table.
##########
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##########
@@ -61,6 +61,10 @@ private class ClientEndpoint(
private val lostMasters = new HashSet[RpcAddress]
private var activeMasterEndpoint: RpcEndpointRef = null
+ private val waitAppCompletion =
conf.getBoolean("spark.standalone.submit.waitAppCompletion",
Review comment:
Hm, do we use the standard parameters classes for these in this part of
the code?
##########
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##########
@@ -124,38 +128,58 @@ private class ClientEndpoint(
}
}
- /* Find out driver status then exit the JVM */
+ /**
+ * Find out driver status then exit the JVM. If the waitAppCompletion is set
to true, monitors
+ * the application until it finishes, fails or is killed.
+ */
def pollAndReportStatus(driverId: String): Unit = {
// Since ClientEndpoint is the only RpcEndpoint in the process, blocking
the event loop thread
// is fine.
logInfo("... waiting before polling master for driver state")
Thread.sleep(5000)
logInfo("... polling master for driver state")
- val statusResponse =
-
activeMasterEndpoint.askSync[DriverStatusResponse](RequestDriverStatus(driverId))
- if (statusResponse.found) {
- logInfo(s"State of $driverId is ${statusResponse.state.get}")
- // Worker node, if present
- (statusResponse.workerId, statusResponse.workerHostPort,
statusResponse.state) match {
- case (Some(id), Some(hostPort), Some(DriverState.RUNNING)) =>
- logInfo(s"Driver running on $hostPort ($id)")
- case _ =>
- }
- // Exception, if present
- statusResponse.exception match {
- case Some(e) =>
- logError(s"Exception from cluster was: $e")
- e.printStackTrace()
- System.exit(-1)
- case _ =>
- System.exit(0)
+ while (true) {
+ val statusResponse =
+
activeMasterEndpoint.askSync[DriverStatusResponse](RequestDriverStatus(driverId))
+ if (statusResponse.found) {
+ logInfo(s"State of $driverId is ${statusResponse.state.get}")
+ // Worker node, if present
+ (statusResponse.workerId, statusResponse.workerHostPort,
statusResponse.state) match {
+ case (Some(id), Some(hostPort), Some(DriverState.RUNNING)) =>
+ logInfo(s"Driver running on $hostPort ($id)")
+ case _ =>
+ }
+ // Exception, if present
+ statusResponse.exception match {
+ case Some(e) =>
+ logError(s"Exception from cluster was: $e")
+ e.printStackTrace()
+ System.exit(-1)
+ case _ =>
+ if (!waitAppCompletion) {
+ logInfo(s"spark-submit not configured to wait for completion, " +
+ s"exiting spark-submit JVM.")
+ System.exit(0)
+ } else {
+ statusResponse.state.get match {
+ case DriverState.FINISHED | DriverState.FAILED |
+ DriverState.ERROR | DriverState.KILLED =>
+ logInfo(s"State of $driverId is
${statusResponse.state.get}, " +
+ s"exiting spark-submit JVM.")
Review comment:
Indent this continuation
##########
File path: docs/spark-standalone.md
##########
@@ -374,6 +374,24 @@ To run an interactive Spark shell against the cluster, run
the following command
You can also pass an option `--total-executor-cores <numCores>` to control the
number of cores that spark-shell uses on the cluster.
+#Spark Properties
+
+Spark applications supports the following configuration properties specific to
standalone Mode:
+<table class="table">
+ <tr><th style="width:21%">Property Name</th><th>Default
Value</th><th>Meaning</th><th>Since Version</th></tr>
+ <tr>
+ <td><code>spark.standalone.submit.waitAppCompletion</code></td>
+ <td><code>false</code></td>
+ <td>
+ In Standalone cluster mode, controls whether the client waits to exit until
the application completes.
Review comment:
Standalone -> standalone
reporting -> polling
##########
File path: docs/spark-standalone.md
##########
@@ -374,6 +374,24 @@ To run an interactive Spark shell against the cluster, run
the following command
You can also pass an option `--total-executor-cores <numCores>` to control the
number of cores that spark-shell uses on the cluster.
+#Spark Properties
Review comment:
Space after #. Maybe call this "Client Properties"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
