tnk-dev opened a new issue, #4629: URL: https://github.com/apache/kyuubi/issues/4629
### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the bug Installing the vanilla helm deployment according to the docs on k8s ```bash helm install kyuubi ${KYUUBI_HOME}/charts/kyuubi -n kyuubi --create-namespace kubectl port-forward svc/kyuubi-thrift-binary 10009:10009 -n kyuubi ``` and then using the jdbc url in dbeaver or beeline results in the following error ### DBEAVER LOGS ``` Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10009: org.apache.kyuubi.KyuubiSQLException: Timeout(180000 ms, you can modify kyuubi.session.engine.initialize.timeout to change it) to launched SPARK_SQL engine with /opt/kyuubi/externals/spark-3.3.2-bin-hadoop3/bin/spark-submit \ --class org.apache.kyuubi.engine.spark.SparkSQLEngine \ --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \ --conf spark.kyuubi.client.ipAddress=192.168.178.43 \ --conf spark.kyuubi.client.version=1.7.0 \ --conf spark.kyuubi.engine.submit.time=1680000909614 \ --conf spark.kyuubi.frontend.protocols=THRIFT_BINARY \ --conf spark.kyuubi.ha.addresses=kyuubi-684c9ccb7-4zdsf:2181 \ --conf spark.kyuubi.ha.engine.ref.id=f7398328-da1e-4fc8-a34f-3f0481e40ca2 \ --conf spark.kyuubi.ha.namespace=/kyuubi_1.8.0-SNAPSHOT_USER_SPARK_SQL/anonymous/default \ --conf spark.kyuubi.ha.zookeeper.auth.type=NONE \ --conf spark.kyuubi.kubernetes.namespace=kyuubi \ --conf spark.kyuubi.server.ipAddress=127.0.0.1 \ --conf spark.kyuubi.session.connection.url=localhost:10009 \ --conf spark.kyuubi.session.real.user=anonymous \ --conf spark.app.name=kyuubi_USER_SPARK_SQL_anonymous_default_f7398328-da1e-4fc8-a34f-3f0481e40ca2 \ --conf spark.kubernetes.driver.label.kyuubi-unique-tag=f7398328-da1e-4fc8-a34f-3f0481e40ca2 \ --conf spark.master=k8s://https://172.20.0.1:443 \ --conf spark.kubernetes.driverEnv.SPARK_USER_NAME=anonymous \ --conf spark.executorEnv.SPARK_USER_NAME=anonymous \ --proxy-user anonymous /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.8.0-SNAPSHOT.jar. (false,Target Pod(tag: f7398328-da1e-4fc8-a34f-3f0481e40ca2) is not found, due to pod have been deleted or not created) at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:250) at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.tryWithLock(ZookeeperDiscoveryClient.scala:180) at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:171) at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:176) at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:276) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:276) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2(KyuubiSessionImpl.scala:147) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2$adapted(KyuubiSessionImpl.scala:123) at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:123) at org.apache.kyuubi.session.KyuubiSession.handleSessionException(KyuubiSession.scala:49) at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:123) at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:60) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.20.0.1/api/v1/namespaces/default/pods?labelSelector=spark-app-selector%3Dspark-ac1644aa5038439092c7bdef0a00eedc%2Cspark-role%3Dexecutor&allowWatchBookmarks=true&watch=true. Message: Forbidden. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:682) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:661) at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.lambda$run$2(WatchConnectionManager.java:126) at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) at io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$BuilderImpl$1.onFailure(OkHttpWebSocketImpl.java:66) at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) See more: /opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.1 at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.engine.ProcBuilder.$anonfun$start$1(ProcBuilder.scala:229) at java.lang.Thread.run(Thread.java:750) . FYI: The last 10 line(s) of log are: at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.engine.ProcBuilder.getError(ProcBuilder.scala:275) at org.apache.kyuubi.engine.ProcBuilder.getError$(ProcBuilder.scala:264) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.getError(SparkProcessBuilder.scala:37) at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:253) ... 18 more org.apache.kyuubi.KyuubiSQLException: Timeout(180000 ms, you can modify kyuubi.session.engine.initialize.timeout to change it) to launched SPARK_SQL engine with /opt/kyuubi/externals/spark-3.3.2-bin-hadoop3/bin/spark-submit \ --class org.apache.kyuubi.engine.spark.SparkSQLEngine \ --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \ --conf spark.kyuubi.client.ipAddress=192.168.178.43 \ --conf spark.kyuubi.client.version=1.7.0 \ --conf spark.kyuubi.engine.submit.time=1680000909614 \ --conf spark.kyuubi.frontend.protocols=THRIFT_BINARY \ --conf spark.kyuubi.ha.addresses=kyuubi-684c9ccb7-4zdsf:2181 \ --conf spark.kyuubi.ha.engine.ref.id=f7398328-da1e-4fc8-a34f-3f0481e40ca2 \ --conf spark.kyuubi.ha.namespace=/kyuubi_1.8.0-SNAPSHOT_USER_SPARK_SQL/anonymous/default \ --conf spark.kyuubi.ha.zookeeper.auth.type=NONE \ --conf spark.kyuubi.kubernetes.namespace=kyuubi \ --conf spark.kyuubi.server.ipAddress=127.0.0.1 \ --conf spark.kyuubi.session.connection.url=localhost:10009 \ --conf spark.kyuubi.session.real.user=anonymous \ --conf spark.app.name=kyuubi_USER_SPARK_SQL_anonymous_default_f7398328-da1e-4fc8-a34f-3f0481e40ca2 \ --conf spark.kubernetes.driver.label.kyuubi-unique-tag=f7398328-da1e-4fc8-a34f-3f0481e40ca2 \ --conf spark.master=k8s://https://172.20.0.1:443 \ --conf spark.kubernetes.driverEnv.SPARK_USER_NAME=anonymous \ --conf spark.executorEnv.SPARK_USER_NAME=anonymous \ --proxy-user anonymous /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.8.0-SNAPSHOT.jar. (false,Target Pod(tag: f7398328-da1e-4fc8-a34f-3f0481e40ca2) is not found, due to pod have been deleted or not created) at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:250) at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.tryWithLock(ZookeeperDiscoveryClient.scala:180) at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:171) at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:176) at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:276) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:276) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2(KyuubiSessionImpl.scala:147) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2$adapted(KyuubiSessionImpl.scala:123) at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:123) at org.apache.kyuubi.session.KyuubiSession.handleSessionException(KyuubiSession.scala:49) at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:123) at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:60) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.20.0.1/api/v1/namespaces/default/pods?labelSelector=spark-app-selector%3Dspark-ac1644aa5038439092c7bdef0a00eedc%2Cspark-role%3Dexecutor&allowWatchBookmarks=true&watch=true. Message: Forbidden. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:682) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:661) at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.lambda$run$2(WatchConnectionManager.java:126) at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) at io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$BuilderImpl$1.onFailure(OkHttpWebSocketImpl.java:66) at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) See more: /opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.1 at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.engine.ProcBuilder.$anonfun$start$1(ProcBuilder.scala:229) at java.lang.Thread.run(Thread.java:750) . FYI: The last 10 line(s) of log are: at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.engine.ProcBuilder.getError(ProcBuilder.scala:275) at org.apache.kyuubi.engine.ProcBuilder.getError$(ProcBuilder.scala:264) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.getError(SparkProcessBuilder.scala:37) at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:253) ... 18 more org.apache.kyuubi.KyuubiSQLException: Timeout(180000 ms, you can modify kyuubi.session.engine.initialize.timeout to change it) to launched SPARK_SQL engine with /opt/kyuubi/externals/spark-3.3.2-bin-hadoop3/bin/spark-submit \ --class org.apache.kyuubi.engine.spark.SparkSQLEngine \ --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \ --conf spark.kyuubi.client.ipAddress=192.168.178.43 \ --conf spark.kyuubi.client.version=1.7.0 \ --conf spark.kyuubi.engine.submit.time=1680000909614 \ --conf spark.kyuubi.frontend.protocols=THRIFT_BINARY \ --conf spark.kyuubi.ha.addresses=kyuubi-684c9ccb7-4zdsf:2181 \ --conf spark.kyuubi.ha.engine.ref.id=f7398328-da1e-4fc8-a34f-3f0481e40ca2 \ --conf spark.kyuubi.ha.namespace=/kyuubi_1.8.0-SNAPSHOT_USER_SPARK_SQL/anonymous/default \ --conf spark.kyuubi.ha.zookeeper.auth.type=NONE \ --conf spark.kyuubi.kubernetes.namespace=kyuubi \ --conf spark.kyuubi.server.ipAddress=127.0.0.1 \ --conf spark.kyuubi.session.connection.url=localhost:10009 \ --conf spark.kyuubi.session.real.user=anonymous \ --conf spark.app.name=kyuubi_USER_SPARK_SQL_anonymous_default_f7398328-da1e-4fc8-a34f-3f0481e40ca2 \ --conf spark.kubernetes.driver.label.kyuubi-unique-tag=f7398328-da1e-4fc8-a34f-3f0481e40ca2 \ --conf spark.master=k8s://https://172.20.0.1:443 \ --conf spark.kubernetes.driverEnv.SPARK_USER_NAME=anonymous \ --conf spark.executorEnv.SPARK_USER_NAME=anonymous \ --proxy-user anonymous /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.8.0-SNAPSHOT.jar. (false,Target Pod(tag: f7398328-da1e-4fc8-a34f-3f0481e40ca2) is not found, due to pod have been deleted or not created) at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:250) at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.tryWithLock(ZookeeperDiscoveryClient.scala:180) at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:171) at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:176) at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:276) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:276) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2(KyuubiSessionImpl.scala:147) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2$adapted(KyuubiSessionImpl.scala:123) at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:123) at org.apache.kyuubi.session.KyuubiSession.handleSessionException(KyuubiSession.scala:49) at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:123) at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:60) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.20.0.1/api/v1/namespaces/default/pods?labelSelector=spark-app-selector%3Dspark-ac1644aa5038439092c7bdef0a00eedc%2Cspark-role%3Dexecutor&allowWatchBookmarks=true&watch=true. Message: Forbidden. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:682) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:661) at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.lambda$run$2(WatchConnectionManager.java:126) at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) at io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$BuilderImpl$1.onFailure(OkHttpWebSocketImpl.java:66) at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) See more: /opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.1 at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.engine.ProcBuilder.$anonfun$start$1(ProcBuilder.scala:229) at java.lang.Thread.run(Thread.java:750) . FYI: The last 10 line(s) of log are: at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.engine.ProcBuilder.getError(ProcBuilder.scala:275) at org.apache.kyuubi.engine.ProcBuilder.getError$(ProcBuilder.scala:264) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.getError(SparkProcessBuilder.scala:37) at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:253) ... 18 more ``` ### K8s Pod Logs ``` 2023-03-28 10:54:31.901 INFO org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[45f2f1fc-e5ba-490d-a509-473185e6b8d8]: PENDING_STATE -> RUNNING_STATE, statement: LaunchEngine 2023-03-28 10:54:31.909 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: Starting 2023-03-28 10:54:31.910 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=kyuubi-684c9ccb7-4zdsf:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@5d50b78d 2023-03-28 10:54:31.919 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server kyuubi-684c9ccb7-4zdsf/10.2.41.162:2181. Will not attempt to authenticate using SASL (unknown error) 2023-03-28 10:54:31.926 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.2.41.162:48604 2023-03-28 10:54:31.926 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to kyuubi-684c9ccb7-4zdsf/10.2.41.162:2181, initiating session 2023-03-28 10:54:31.928 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.2.41.162:48604 2023-03-28 10:54:31.943 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x100667d547a0002 with negotiated timeout 60000 for client /10.2.41.162:48604 2023-03-28 10:54:31.944 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server kyuubi-684c9ccb7-4zdsf/10.2.41.162:2181, sessionid = 0x100667d547a0002, negotiated timeout = 60000 2023-03-28 10:54:31.944 INFO org.apache.curator.framework.state.ConnectionStateManager: State change: CONNECTED 2023-03-28 10:55:09.605 WARN org.apache.kyuubi.engine.KubernetesApplicationOperation: Get Tag: 465bd0da-e10c-4a42-b658-fae16f2985ca Driver Pod In Kubernetes size: 0, we expect 1 2023-03-28 10:55:09.617 INFO org.apache.kyuubi.engine.EngineRef: Launching engine: --conf spark.kyuubi.frontend.protocols=THRIFT_BINARY \ --conf spark.kyuubi.ha.addresses=kyuubi-684c9ccb7-4zdsf:2181 \ --conf spark.kyuubi.ha.engine.ref.id=f7398328-da1e-4fc8-a34f-3f0481e40ca2 \ --conf spark.kyuubi.ha.namespace=/kyuubi_1.8.0-SNAPSHOT_USER_SPARK_SQL/anonymous/default \ --conf spark.kyuubi.ha.zookeeper.auth.type=NONE \ --conf spark.kyuubi.kubernetes.namespace=kyuubi \ --conf spark.kyuubi.server.ipAddress=127.0.0.1 \ --conf spark.kyuubi.session.connection.url=localhost:10009 \ --conf spark.kyuubi.session.real.user=anonymous \ --conf spark.app.name=kyuubi_USER_SPARK_SQL_anonymous_default_f7398328-da1e-4fc8-a34f-3f0481e40ca2 \ --conf spark.kubernetes.driver.label.kyuubi-unique-tag=f7398328-da1e-4fc8-a34f-3f0481e40ca2 \ --conf spark.master=k8s://https://172.20.0.1:443 \ --conf spark.kubernetes.driverEnv.SPARK_USER_NAME=anonymous \ --conf spark.executorEnv.SPARK_USER_NAME=anonymous \ --proxy-user anonymous /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.8.0-SNAPSHOT.jar 2023-03-28 10:55:09.620 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting 2023-03-28 10:55:09.621 INFO org.apache.kyuubi.engine.ProcBuilder: Logging to /opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.1 2023-03-28 10:55:09.626 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x100667d547a0001 2023-03-28 10:55:09.630 INFO org.apache.zookeeper.ZooKeeper: Session: 0x100667d547a0001 closed 2023-03-28 10:55:09.632 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.2.41.162:33976 which had sessionid 0x100667d547a0001 2023-03-28 10:55:09.630 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x100667d547a0001 2023-03-28 10:55:09.647 INFO org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[bd0bec2c-a9cd-418a-8274-bf57388ad9a7]: RUNNING_STATE -> ERROR_STATE, time taken: 182.128 seconds 2023-03-28 10:55:09.726 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [465bd0da-e10c-4a42-b658-fae16f2985ca] 2023-03-28 10:55:09.733 INFO org.apache.kyuubi.session.KyuubiSessionManager: anonymous's session with SessionHandle [465bd0da-e10c-4a42-b658-fae16f2985ca] is closed, current opening sessions 1 2023-03-28 10:55:09.801 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [465bd0da-e10c-4a42-b658-fae16f2985ca] 2023-03-28 10:58:10.029 WARN org.apache.kyuubi.engine.KubernetesApplicationOperation: Get Tag: f7398328-da1e-4fc8-a34f-3f0481e40ca2 Driver Pod In Kubernetes size: 0, we expect 1 2023-03-28 10:58:10.036 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting 2023-03-28 10:58:10.056 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x100667d547a0002 2023-03-28 10:58:10.059 INFO org.apache.zookeeper.ZooKeeper: Session: 0x100667d547a0002 closed 2023-03-28 10:58:10.059 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x100667d547a0002 2023-03-28 10:58:10.065 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.2.41.162:48604 which had sessionid 0x100667d547a0002 2023-03-28 10:58:10.066 INFO org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[45f2f1fc-e5ba-490d-a509-473185e6b8d8]: RUNNING_STATE -> ERROR_STATE, time taken: 218.165 seconds 2023-03-28 10:58:10.166 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [f7398328-da1e-4fc8-a34f-3f0481e40ca2] 2023-03-28 10:58:10.166 INFO org.apache.kyuubi.session.KyuubiSessionManager: anonymous's session with SessionHandle [f7398328-da1e-4fc8-a34f-3f0481e40ca2] is closed, current opening sessions 0 2023-03-28 10:58:10.168 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [f7398328-da1e-4fc8-a34f-3f0481e40ca2] ``` Any Ideas? ### Affects Version(s) 1.7.0 ### Kyuubi Server Log Output _No response_ ### Kyuubi Engine Log Output _No response_ ### Kyuubi Server Configurations _No response_ ### Kyuubi Engine Configurations _No response_ ### Additional context _No response_ ### Are you willing to submit PR? - [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix. - [X] No. I cannot submit a PR at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
