vanzin commented on a change in pull request #25648: [SPARK-28947][K8S] Status
logging not happens at an interval for liveness
URL: https://github.com/apache/spark/pull/25648#discussion_r329670993
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala
##########
@@ -22,43 +22,41 @@ import io.fabric8.kubernetes.api.model.Pod
import io.fabric8.kubernetes.client.{KubernetesClientException, Watcher}
import io.fabric8.kubernetes.client.Watcher.Action
+import org.apache.spark.deploy.k8s.Config._
+import org.apache.spark.deploy.k8s.KubernetesDriverConf
import org.apache.spark.deploy.k8s.KubernetesUtils._
import org.apache.spark.internal.Logging
import org.apache.spark.util.ThreadUtils
private[k8s] trait LoggingPodStatusWatcher extends Watcher[Pod] {
- def awaitCompletion(): Unit
+ def watchOrStop(submissionId: String): Unit
}
/**
* A monitor for the running Kubernetes pod of a Spark application. Status
logging occurs on
* every state change and also at an interval for liveness.
*
- * @param appId application ID.
- * @param maybeLoggingInterval ms between each state request. If provided,
must be a positive
- * number.
+ * @param conf kubernetes driver conf.
*/
-private[k8s] class LoggingPodStatusWatcherImpl(
- appId: String,
- maybeLoggingInterval: Option[Long])
+private[k8s] class LoggingPodStatusWatcherImpl(conf: KubernetesDriverConf)
extends LoggingPodStatusWatcher with Logging {
+ private val appId = conf.appId
+
private val podCompletedFuture = new CountDownLatch(1)
+
// start timer for periodic logging
- private val scheduler =
-
ThreadUtils.newDaemonSingleThreadScheduledExecutor("logging-pod-status-watcher")
- private val logRunnable: Runnable = () => logShortStatus()
+ private lazy val maybeLoggingService = if
(conf.get(WAIT_FOR_APP_COMPLETION)) {
Review comment:
@erikerlandson not sure what question you're referring to, but I don't
remember asking that question. This seems to be the same behavior as before,
which is fine by me.
(This could even be done inside `watchOrStop` now instead of having a lazy
val. Or you could even avoid the extra thread by using a loop, calling
`podCompletedFuture.await` with a timeout and logging while it's not signaled.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
