HwangDongJun opened a new issue, #7749: URL: https://github.com/apache/kyuubi/issues/7749
### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the bug When submitting a Spark batch job via the Batch REST API (`POST /api/v1/batches`) with `spark.master=k8s://...`, Kyuubi does not validate that `spark.submit.deployMode` is `cluster`. If `spark.submit.deployMode` is explicitly set to `client`, or simply left unset (Spark's own default is `client`), Kyuubi still proceeds to submit the job. Spark then runs the driver as a subprocess of the Kyuubi server itself (no separate Kubernetes driver pod is created), which breaks the entire premise of the Batch API: - Kyuubi's `kyuubi-unique-tag`-based tracking mechanism waits for a driver pod that will never be created, and eventually reports a misleading `ERROR` / `UNKNOWN` state. - The batch driver process shares CPU/memory with the Kyuubi server pod itself instead of being isolated in its own pod. - Users see errors that give no indication of the actual root cause, and have to read server-side logs or source code to figure out that `spark.submit.deployMode` was the actual problem. **Root cause (found by reading the code):** There are two compounding issues. 1. `BatchJobSubmission` never calls `builder.validateConf()` at all. `Validator.validateConf()` already exists specifically to fail fast on invalid Spark/K8s configuration (currently it validates `spark.kubernetes.executor.podNamePrefix` length), but it is only invoked from `EngineRef.scala` (used for launching interactive SQL engines). The batch job submission path (`kyuubi-server/.../operation/BatchJobSubmission.scala`) goes straight to `builder.start` and never calls `validateConf()`: ```scala // BatchJobSubmission.scala, submitAndMonitorBatchJob() try { info(s"Submitting $batchType batch[$batchId] job:\n$builder") val process = builder.start // <-- validateConf() is never called before this ... ``` This means the existing validation (`podNamePrefix` length) has never actually been exercised for any batch job submitted through Kyuubi. 2. Even if `validateConf()` were called, nothing checks the `deployMode` for `k8s://` masters. ```scala // kyuubi-common/.../util/Validator.scala def validateConf(kyuubiConf: KyuubiConf): Unit = { validateSparkK8sConf(kyuubiConf) } private def validateSparkK8sConf(kyuubiConf: KyuubiConf): Unit = { val prefix = kyuubiConf.getOption(KUBERNETES_EXECUTOR_POD_NAME_PREFIX).orNull if (prefix != null && !isValidExecutorPodNamePrefix(prefix)) { ... } // no check at all for spark.master / spark.submit.deployMode combination } ``` `SparkProcessBuilder` already has all the helpers needed to detect this (`clusterManager()`, `deployMode()`, `isClusterMode()`, `isK8sClusterMode`); they are just never used for this purpose. ### Affects Version(s) 1.11.1, reproduced again on master (1.13.0-SNAPSHOT) ### Kyuubi Server Log Output ```logtalk INFO ... org.apache.kyuubi.operation.BatchJobSubmission: Submitting PYSPARK batch[<batchId>] job: --conf spark.kyuubi.batch.id=<batchId> ... --proxy-user testuser local:///path/to/user_script.py SELECT 1 WARN ... org.apache.kyuubi.engine.KubernetesApplicationOperation: Waiting for driver pod with label: kyuubi-unique-tag=<batchId> to be created, elapsed time: 0ms, return UNKNOWN status INFO ... org.apache.kyuubi.engine.KubernetesApplicationOperation: Found 0 existing pods with label kyuubi-unique-tag WARN ... org.apache.kyuubi.engine.KubernetesApplicationOperation: Waiting for driver pod with label: kyuubi-unique-tag=<batchId> to be created, elapsed time: 1167ms, return UNKNOWN status ERROR ... org.apache.kyuubi.operation.BatchJobSubmission: Error operating BatchJobSubmission: org.apache.kyuubi.KyuubiException: Process exit with value 1, application info: Some(ApplicationInfo(null,null,UNKNOWN,None,None,None)) ERROR ... org.apache.kyuubi.operation.BatchJobSubmission: Batch <batchId> state is ERROR, but the application state is UNKNOWN and not terminated, set to UNKNOWN. Batch status API response for the same batch: {"id":"<batchId>","user":"testuser","batchType":"PYSPARK","name":null,"appStartTime":0,"appId":null,"appUrl":null,"appState":"UNKNOWN","appDiagnostic":null,"kyuubiInstance":"0.0.0.0:10099","state":"ERROR","createTime":1700000000000,"endTime":1700000001000,"batchInfo":{}} ``` ### Kyuubi Engine Log Output ```logtalk N/A for this batch job (no engine/driver pod is ever created). For reference, the local spark-submit client process log captured on the Kyuubi server pod itself ($KYUUBI_WORK_DIR_ROOT/<user>/kyuubi-spark-batch-submit.log.0) shows the driver code running locally instead of submitting to Kubernetes: python3: can't open file '/path/to/user_script.py': [Errno 2] No such file or directory Exception in thread "main" java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1969) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:184) ... Caused by: org.apache.spark.SparkUserAppException: User application exited with 2 at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:127) at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala) ... ``` ### Kyuubi Server Configurations ```yaml kyuubi.frontend.protocols=REST kyuubi.frontend.rest.bind.host=0.0.0.0 kyuubi.frontend.rest.bind.port=10099 kyuubi.engine.type=SPARK_SQL kyuubi.engine.share.level=CONNECTION kyuubi.kubernetes.master.address=https://kubernetes.default.svc.cluster.local:443 ``` ### Kyuubi Engine Configurations ```yaml # $SPARK_CONF_DIR/spark-defaults.conf on the Kyuubi server pod, which is what # SparkProcessBuilder.defaultsConf falls back to when a conf key isn't explicitly # passed per-batch: spark.master k8s://https://kubernetes.default.svc.cluster.local:443 spark.submit.deployMode client ``` ### Additional context - This is not a one-off misconfiguration; we hit this exact class of problem three separate times across different clusters over about two months of running Kyuubi in production (each time initially suspecting a different cause — tracking bug, pod-template issue — before tracing it back to the same root cause). - We already have a fix implemented and verified locally: - Added an override of `validateConf()` in `SparkBatchProcessBuilder` that throws a clear `KyuubiException` when `spark.master` starts with `k8s` and `deployMode` is not `cluster`. - Added the missing `builder.validateConf()` call in `BatchJobSubmission.submitAndMonitorBatchJob()`. - Added 4 new unit tests to `SparkBatchProcessBuilderSuite` covering: reject when `client`, reject when unset, allow when `cluster`, and confirm non-k8s masters (e.g. `local[*]`, `yarn` client) are unaffected. - Ran the full `kyuubi-server` test suite locally (`mvn test -pl kyuubi-server -am -P spark-3.5`) — all 6 tests in `SparkBatchProcessBuilderSuite` pass, including the 2 pre-existing tests (no regression). - Note: wiring up `validateConf()` for batches for the first time also means the pre-existing `podNamePrefix` length check will now apply to batch submissions where it previously didn't. We consider this a bug fix (the check was silently dead code for all batches), but flagging it here since it is technically a behavior change for anyone currently relying on the check being skipped. ### Are you willing to submit PR? - [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix. - [ ] No. I cannot submit a PR at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
