GitHub user justlucknb edited a discussion: Spark Driver/Executors dont start
in K8S via Kyuubi
Hello. I have a problem with start Kyuubi session in K8S..
I started session via beeline. But i has no effect in create spark
driver/executors via spark-submit.
ServiceAccount for statefull set have a full permissions in Namespace.
Network access to k8s master via pod working.
Kyuubi deployed via official helm chart.
Logs:
```
kyuubi@kyuubi-0:/opt/kyuubi$ ./bin/beeline -u
jdbc:hive2://localhost:10009/default
Warn: Not find kyuubi environment file /opt/kyuubi/conf/kyuubi-env.sh, using
default ones...
Connecting to jdbc:hive2://localhost:10009/default
2025-06-11 12:17:59.379 INFO KyuubiSessionManager-exec-pool: Thread-68
org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's
query[48fe5799-f97e-4387-8253-a02f15cbf565]: PENDING_STATE -> RUNNING_STATE,
statement:
LaunchEngine
2025-06-11 12:17:59.380 INFO KyuubiSessionManager-exec-pool: Thread-68
org.apache.kyuubi.shaded.curator.framework.imps.CuratorFrameworkImpl: Starting
2025-06-11 12:17:59.381 INFO KyuubiSessionManager-exec-pool: Thread-68
org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Initiating client connection,
connectString=100.74.1.85:2181 sessionTimeout=60000
watcher=org.apache.kyuubi.shaded.curator.ConnectionState@4ac4c6e0
2025-06-11 12:17:59.382 INFO KyuubiSessionManager-exec-pool:
Thread-68-SendThread(kyuubi-0.kyuubi-headless.spark-test.svc.k8s-xx:2181)
org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Opening socket connection to
server kyuubi-0.kyuubi-headless.spark-test.svc.xx/100.74.1.85:2181. Will not
attempt to authenticate using SASL (unknown error)
2025-06-11 12:17:59.383 INFO KyuubiSessionManager-exec-pool:
Thread-68-SendThread(kyuubi-0.kyuubi-headless.spark-test.svc.xx:2181)
org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Socket connection established to
kyuubi-0.kyuubi-headless.spark-test.svc.xx/100.74.1.85:2181, initiating session
2025-06-11 12:17:59.384 INFO KyuubiSessionManager-exec-pool:
Thread-68-SendThread(kyuubi-0.kyuubi-headless.spark-test.svc.xx:2181)
org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Session establishment complete
on server kyuubi-0.kyuubi-headless.spark-test.svc.xx/100.74.1.85:2181,
sessionid = 0x110988064e60002, negotiated timeout = 60000
2025-06-11 12:17:59.384 INFO KyuubiSessionManager-exec-pool:
Thread-68-EventThread
org.apache.kyuubi.shaded.curator.framework.state.ConnectionStateManager: State
change: CONNECTED
2025-06-11 12:17:59.391 INFO KyuubiSessionManager-exec-pool: Thread-68
org.apache.kyuubi.engine.ProcBuilder: Logging to
/opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.1
2025-06-11 12:17:59.392 INFO KyuubiSessionManager-exec-pool: Thread-68
org.apache.kyuubi.engine.EngineRef: Launching engine:
/opt/kyuubi/externals/spark-3.4.2-bin-hadoop3/bin/spark-submit \
--class org.apache.kyuubi.engine.spark.SparkSQLEngine \
--conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
--conf spark.kyuubi.client.ipAddress=xxx \
--conf spark.kyuubi.client.version=1.8.3 \
--conf
spark.kyuubi.engine.engineLog.path=/opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.1
\
--conf spark.kyuubi.engine.share.level=USER \
--conf spark.kyuubi.engine.submit.time=1749644279388 \
--conf spark.kyuubi.engine.type=SPARK_SQL \
--conf spark.kyuubi.ha.addresses=xxx:2181 \
--conf
spark.kyuubi.ha.engine.ref.id=515cc8cd-18ef-48e2-a12e-69392c846e23 \
--conf
spark.kyuubi.ha.namespace=/spark-test_1.8.3_USER_SPARK_SQL/anonymous/default \
--conf spark.kyuubi.ha.zookeeper.auth.type=NONE \
--conf spark.kyuubi.ha.zookeeper.namespace=spark-test \
--conf spark.kyuubi.kubernetes.namespace=spark-test \
--conf spark.kyuubi.server.ipAddress=0.0.0.0 \
--conf spark.kyuubi.session.connection.url=0.0.0.0:10009 \
--conf spark.kyuubi.session.engine.initialize.timeout=100000 \
--conf spark.kyuubi.session.engine.open.max.attempts=1 \
--conf spark.kyuubi.session.engine.share.level=CONNECTION \
--conf spark.kyuubi.session.real.user=anonymous \
--conf
spark.app.name=kyuubi_USER_SPARK_SQL_anonymous_default_515cc8cd-18ef-48e2-a12e-69392c846e23
\
--conf spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp
-Divy.home=/tmp \
--conf spark.driver.host=kyuubi-hs.spark-test.svc \
--conf spark.driver.memory=2g \
--conf spark.driver.port=44104 \
--conf spark.dynamicAllocation.enabled=true \
--conf spark.dynamicAllocation.executorAllocationRatio=1 \
--conf spark.dynamicAllocation.initialExecutors=2 \
--conf spark.dynamicAllocation.maxExecutors=20 \
--conf spark.dynamicAllocation.minExecutors=2 \
--conf spark.dynamicAllocation.shuffleTracking.enabled=true \
--conf spark.executor.instances=2 \
--conf spark.hadoop.fs.s3a.access.key=xxx \
--conf
spark.hadoop.fs.s3a.endpoint=http://minio-test.minio-common-test:9000 \
--conf spark.hadoop.fs.s3a.fast.upload=true \
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3a.path.style.access=true \
--conf spark.hadoop.fs.s3a.secret.key=xxx \
--conf
spark.jars.packages=org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262
\
--conf
spark.kubernetes.authenticate.caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
\
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf
spark.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token
\
--conf spark.kubernetes.authenticate.serviceAccountName=kyuubi \
--conf spark.kubernetes.container.image=registry.xx/spark:3.4.0 \
--conf
spark.kubernetes.driver.label.kyuubi-unique-tag=515cc8cd-18ef-48e2-a12e-69392c846e23
\
--conf spark.kubernetes.driver.pod.name=$HOSTNAME \
--conf spark.kubernetes.driver.podNamePrefix=kyuubi \
--conf spark.kubernetes.executor.podNamePrefix=kyuubi \
--conf spark.kubernetes.file.upload.path=local:///opt/spark/tmp \
--conf spark.kubernetes.namespace=spark-test \
--conf spark.master=k8s://https://kubernetes.default.svc:443 \
--conf spark.submit.deployMode=cluster \
--conf spark.kubernetes.driverEnv.SPARK_USER_NAME=anonymous \
--conf spark.executorEnv.SPARK_USER_NAME=anonymous \
--proxy-user anonymous
/opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.8.3.jar
25/06/11 12:18:01 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
:: loading settings :: url =
jar:file:/opt/kyuubi/externals/spark-3.4.2-bin-hadoop3/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /home/kyuubi/.ivy2/cache
The jars for the packages stored in: /home/kyuubi/.ivy2/jars
org.apache.hadoop#hadoop-aws added as a dependency
com.amazonaws#aws-java-sdk-bundle added as a dependency
:: resolving dependencies ::
org.apache.spark#spark-submit-parent-8ca5fcf2-22dc-4c26-96a9-ab39e380d10f;1.0
confs: [default]
2025-06-11 12:19:39.518 INFO KyuubiSessionManager-exec-pool: Thread-68
org.apache.kyuubi.Utils: Loading Kyuubi properties from
/opt/spark/conf/spark-defaults.conf
2025-06-11 12:19:39.520 WARN KyuubiSessionManager-exec-pool: Thread-68
org.apache.kyuubi.engine.KubernetesApplicationOperation: No application info
found, trying to delete pod with label:
kyuubi-unique-tag=515cc8cd-18ef-48e2-a12e-69392c846e23
2025-06-11 12:19:40.915 INFO Curator-Framework-0
org.apache.kyuubi.shaded.curator.framework.imps.CuratorFrameworkImpl:
backgroundOperationsLoop exiting
2025-06-11 12:19:40.917 INFO KyuubiSessionManager-exec-pool: Thread-68
org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Session: 0x110988064e60002 closed
2025-06-11 12:19:40.917 INFO KyuubiSessionManager-exec-pool:
Thread-68-EventThread org.apache.kyuubi.shaded.zookeeper.ClientCnxn:
EventThread shut down for session: 0x110988064e60002
2025-06-11 12:19:40.919 INFO KyuubiSessionManager-exec-pool: Thread-68
org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's
query[48fe5799-f97e-4387-8253-a02f15cbf565]: RUNNING_STATE -> ERROR_STATE, time
taken: 101.539 seconds
Error: org.apache.kyuubi.KyuubiSQLException: Timeout(100000 ms, you can modify
kyuubi.session.engine.initialize.timeout to change it) to launched SPARK_SQL
engine with /opt/kyuubi/externals/spark-3.4.2-bin-hadoop3/bin/spark-submit \
--class org.apache.kyuubi.engine.spark.SparkSQLEngine \
--conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
--conf spark.kyuubi.client.ipAddress=xx \
--conf spark.kyuubi.client.version=1.8.3 \
--conf
spark.kyuubi.engine.engineLog.path=/opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.1
\
--conf spark.kyuubi.engine.share.level=USER \
--conf spark.kyuubi.engine.submit.time=1749644279388 \
--conf spark.kyuubi.engine.type=SPARK_SQL \
--conf spark.kyuubi.ha.addresses=xx:2181 \
--conf
spark.kyuubi.ha.engine.ref.id=515cc8cd-18ef-48e2-a12e-69392c846e23 \
--conf
spark.kyuubi.ha.namespace=/spark-test_1.8.3_USER_SPARK_SQL/anonymous/default \
--conf spark.kyuubi.ha.zookeeper.auth.type=NONE \
--conf spark.kyuubi.ha.zookeeper.namespace=spark-test \
--conf spark.kyuubi.kubernetes.namespace=spark-test \
--conf spark.kyuubi.server.ipAddress=0.0.0.0 \
--conf spark.kyuubi.session.connection.url=0.0.0.0:10009 \
--conf spark.kyuubi.session.engine.initialize.timeout=100000 \
--conf spark.kyuubi.session.engine.open.max.attempts=1 \
--conf spark.kyuubi.session.engine.share.level=CONNECTION \
--conf spark.kyuubi.session.real.user=anonymous \
--conf
spark.app.name=kyuubi_USER_SPARK_SQL_anonymous_default_515cc8cd-18ef-48e2-a12e-69392c846e23
\
--conf spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp
-Divy.home=/tmp \
--conf spark.driver.host=kyuubi-hs.spark-test.svc \
--conf spark.driver.memory=2g \
--conf spark.driver.port=44104 \
--conf spark.dynamicAllocation.enabled=true \
--conf spark.dynamicAllocation.executorAllocationRatio=1 \
--conf spark.dynamicAllocation.initialExecutors=2 \
--conf spark.dynamicAllocation.maxExecutors=20 \
--conf spark.dynamicAllocation.minExecutors=2 \
--conf spark.dynamicAllocation.shuffleTracking.enabled=true \
--conf spark.executor.instances=2 \
--conf spark.hadoop.fs.s3a.access.key=xx \
--conf
spark.hadoop.fs.s3a.endpoint=http://minio-test.minio-common-test:9000 \
--conf spark.hadoop.fs.s3a.fast.upload=true \
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3a.path.style.access=true \
--conf spark.hadoop.fs.s3a.secret.key=xx \
--conf
spark.jars.packages=org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262
\
--conf
spark.kubernetes.authenticate.caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
\
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf
spark.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token
\
--conf spark.kubernetes.authenticate.serviceAccountName=kyuubi \
--conf spark.kubernetes.container.image=registry.xx/spark:3.4.0 \
--conf
spark.kubernetes.driver.label.kyuubi-unique-tag=515cc8cd-18ef-48e2-a12e-69392c846e23
\
--conf spark.kubernetes.driver.pod.name=$HOSTNAME \
--conf spark.kubernetes.driver.podNamePrefix=kyuubi \
--conf spark.kubernetes.executor.podNamePrefix=kyuubi \
--conf spark.kubernetes.file.upload.path=local:///opt/spark/tmp \
--conf spark.kubernetes.namespace=spark-test \
--conf spark.master=k8s://https://kubernetes.default.svc:443 \
--conf spark.submit.deployMode=cluster \
--conf spark.kubernetes.driverEnv.SPARK_USER_NAME=anonymous \
--conf spark.executorEnv.SPARK_USER_NAME=anonymous \
--proxy-user anonymous
/opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.8.3.jar.
(false,[KubernetesInfo(None,Some(spark-test))] Operation of deleted pod with
label: kyuubi-unique-tag=515cc8cd-18ef-48e2-a12e-69392c846e23 is completed)
at
org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:70)
at
org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:243)
at
org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.tryWithLock(ZookeeperDiscoveryClient.scala:166)
at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:174)
at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:179)
at
org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:309)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:309)
at
org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2(KyuubiSessionImpl.scala:150)
at
org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2$adapted(KyuubiSessionImpl.scala:124)
at
org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
at
org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:124)
at
org.apache.kyuubi.session.KyuubiSession.handleSessionException(KyuubiSession.scala:49)
at
org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:124)
at
org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:60)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.kyuubi.KyuubiSQLException: Failed to detect the root
cause, please check /opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.1 at
server side if necessary. The last 10 line(s) of log are:
25/06/11 12:18:01 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
:: loading settings :: url =
jar:file:/opt/kyuubi/externals/spark-3.4.2-bin-hadoop3/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /home/kyuubi/.ivy2/cache
The jars for the packages stored in: /home/kyuubi/.ivy2/jars
org.apache.hadoop#hadoop-aws added as a dependency
com.amazonaws#aws-java-sdk-bundle added as a dependency
:: resolving dependencies ::
org.apache.spark#spark-submit-parent-8ca5fcf2-22dc-4c26-96a9-ab39e380d10f;1.0
confs: [default]
at
org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:70)
at org.apache.kyuubi.engine.ProcBuilder.getError(ProcBuilder.scala:275)
at org.apache.kyuubi.engine.ProcBuilder.getError$(ProcBuilder.scala:268)
at
org.apache.kyuubi.engine.spark.SparkProcessBuilder.getError(SparkProcessBuilder.scala:42)
at
org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:246)
... 18 more (state=,code=0)
```
GitHub link: https://github.com/apache/kyuubi/discussions/7096
----
This is an automatically sent email for notifications@kyuubi.apache.org.
To unsubscribe, please send an email to:
notifications-unsubscr...@kyuubi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscr...@kyuubi.apache.org
For additional commands, e-mail: notifications-h...@kyuubi.apache.org
