shockdm commented on a change in pull request #29496:
URL: https://github.com/apache/spark/pull/29496#discussion_r482203205
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala
##########
@@ -177,4 +188,27 @@ private[k8s] class LoggingPodStatusWatcherImpl(
private def formatTime(time: String): String = {
if (time != null || time != "") time else "N/A"
}
+
+ override def watchOrStop(sId: String): Boolean = if (hasCompleted()) {
Review comment:
@jkleckner
This line is different in the `master` branch. It check whether the client
was set to wait for the completion of the driver, not simply checking
`hasCompleted()`, which should normally return false - causing `spark-submit`
to terminate immediately.
For `2.4.7` port you may want to add additional parameter to the
`LoggingPodStatusWatcherImpl`, specifically `waitForCompletion: Boolean`, that
can be checked in this if statement. In current master they are using
`KubernetesDriverConf` instead of passing parameters separately:
https://github.com/apache/spark/blob/5e6173ebefcbf3d85ba10234b35fe043f8497d32/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala#L40.
However in `2.4.6`, `KubernetesDriverConf` is not yet introduced, and
instead the `appId` and `maybeLoggingInterval` are passed individually. Simple
solution is to add `waitForCompletion` as a parameter, and use it in the above
if statement.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
