GitHub user moelhoussein created a discussion: Should Kyuubi Batch support restart policies for always-on Spark streaming applications?
In addition to the mature Structured Streaming micro-batch engine, Spark 4.1 added the first official support for [Real-Time Mode](https://issues.apache.org/jira/browse/SPARK-53736), initially for stateless Scala queries. We are seeing more interest from our team in running always-on Spark streaming applications through Kyuubi. I would like to understand how the community sees Kyuubi's role in managing the driver lifecycle for these applications. Kyuubi Batch can submit an application, monitor its state, persist metadata and kill it. If the driver terminates or is evicted, the batch becomes terminal and Kyuubi does not resubmit it. This is especially relevant on Kubernetes. A healthy driver pod cannot be expected to remain on the same node indefinitely. Node upgrades, reimaging, autoscaler consolidation, drift and other maintenance events can eventually evict it. Kubernetes pod restart policies only restart containers inside an existing pod. They do not recreate an evicted or deleted driver pod on another node. The [Kubeflow Spark Operator](https://www.kubeflow.org/docs/components/spark-operator/user-guide/working-with-sparkapplication/) provides one precedent. Its restart policy resubmits the Spark application and creates a new driver when the previous attempt fails. Longer term, I think Kyuubi should ideally manage more of the workload lifecycle itself, perhaps through gateway-level extension points similar in spirit to [PR #7530](https://github.com/apache/kyuubi/pull/7530) which covers interactive statement interception rather than batch or streaming lifecycle, so I am only citing it as an extension pattern. For this narrower question, assume that the Spark application remains responsible for its own Structured Streaming lifecycle. This includes durable checkpoints, source and sink recovery, query health, and failing the driver when a critical query terminates. Kyuubi would not inspect or restart individual `StreamingQuery` instances. It would only decide whether to resubmit the application after a terminal failure. Would [Batch v2](https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiBatchService.scala) be the right place for an optional, cluster-manager-neutral restart policy? It already persists the submission request and uses a database-backed scheduler. A policy might preserve the current behavior by default with `NEVER`, while allowing an opt-in `ON_FAILURE` mode with bounded attempts and backoff. Are others running always-on Spark applications through Kyuubi today? Should application resubmission remain the responsibility of an external orchestrator, or would a bounded restart policy be useful in Batch v2? If Kyuubi should own it, what semantics would the community expect for attempts, cancellation, HA ownership, resource cleanup and application history? GitHub link: https://github.com/apache/kyuubi/discussions/7754 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
