Swarvenstein opened a new issue, #4194: URL: https://github.com/apache/kyuubi/issues/4194
### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the bug Hello. We are running kyuubi in k8s environment and faced an issue with adjusting driver idle timeout. By default it's going to Completed state after 50s of inactivity. We've tried to use the following parameters in kyuubi-configmap.yaml\kyuubi-defaults.conf: ``` kyuubi.backend.engine.exec.pool.keepalive.time=PT10M kyuubi.backend.server.exec.pool.keepalive.time=PT10M kyuubi.batch.application.check.interval=PT10M kyuubi.engine.user.isolated.spark.session.idle.interval=PT10M kyuubi.frontend.thrift.worker.keepalive.time=PT10M kyuubi.ha.zookeeper.session.timeout=60000000 kyuubi.zookeeper.embedded.min.session.timeout=600000 kyuubi.zookeeper.embedded.max.session.timeout=6000000 kyuubi.zookeeper.embedded.tick.time=30000 kyuubi.session.engine.alive.timeout=PT10M kyuubi.session.engine.idle.timeout=PT10M kyuubi.session.engine.check.interval=PT10M kyuubi.session.engine.alive.probe.interval=PT10M kyuubi.session.engine.login.timeout=PT10M kyuubi.session.check.interval=PT10M kyuubi.session.idle.timeout=PT10M kyuubi.session.engine.startup.waitCompletion=true ``` None of them (we tried one by one and all of them together) changing the driver state changing timeout. Why it's needed - driver start takes about 20-30s and we want to keep it 5-10 minutes in idle state to prevent this time overhead for kyuubi users in each separate request. I will be very grateful if you have the opportunity and help to solve this issue. ### Affects Version(s) 1.6.1 ### Kyuubi Server Log Output ```logtalk 2023-01-19 13:18:47.765 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V10 2023-01-19 13:18:47.766 INFO org.apache.kyuubi.session.KyuubiSessionManager: Opening session for anonymous@192.168.21.137 2023-01-19 13:18:47.767 WARN org.apache.kyuubi.config.KyuubiConf: The Kyuubi config 's3kyuubi.ha.zookeeper.namespace' has been deprecated in Kyuubi v1.6.0 and may be removed in the future. Use kyuubi.ha.namespace instead 2023-01-19 13:18:47.767 WARN org.apache.kyuubi.config.KyuubiConf: The Kyuubi config 'kyuubi.frontend.bind.port' has been deprecated in Kyuubi v1.4.0 and may be removed in the future. Use kyuubi.frontend.thrift.binary.bind.port instead 2023-01-19 13:18:47.768 WARN org.apache.kyuubi.config.KyuubiConf: The Kyuubi config 'kyuubi.frontend.bind.port' has been deprecated in Kyuubi v1.4.0 and may be removed in the future. Use kyuubi.frontend.thrift.binary.bind.port instead 2023-01-19 13:18:47.769 INFO org.apache.kyuubi.operation.log.OperationLog: Creating operation log file /opt/kyuubi/work/server_operation_logs/35d34b4c-e204-485d-b559-8afde030e936/02f35cb1-348b-458d-b9da-8f833b60cf21 2023-01-19 13:18:47.770 INFO org.apache.kyuubi.session.KyuubiSessionManager: anonymous's session with SessionHandle [35d34b4c-e204-485d-b559-8afde030e936] is opened, current opening sessions 1 2023-01-19 13:18:47.770 INFO org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[02f35cb1-348b-458d-b9da-8f833b60cf21]: PENDING_STATE -> RUNNING_STATE, statement: LaunchEngine 2023-01-19 13:18:47.772 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: Starting 2023-01-19 13:18:47.772 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=kyuubi-zookeeper.zookeeper:2181 sessionTimeout=60000000 watcher=org.apache.curator.ConnectionState@117edab5 2023-01-19 13:18:47.777 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server kyuubi-zookeeper.zookeeper/10.108.54.123:2181. Will not attempt to authenticate using SASL (unknown error) 2023-01-19 13:18:47.779 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to kyuubi-zookeeper.zookeeper/10.108.54.123:2181, initiating session 2023-01-19 13:18:47.781 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server kyuubi-zookeeper.zookeeper/10.108.54.123:2181, sessionid = 0x300720fdb0619e3, negotiated timeout = 40000 2023-01-19 13:18:47.782 INFO org.apache.curator.framework.state.ConnectionStateManager: State change: CONNECTED 2023-01-19 13:18:47.786 INFO org.apache.kyuubi.engine.EngineRef: Launching engine: /opt/spark/bin/spark-submit \ --class org.apache.kyuubi.engine.spark.SparkSQLEngine \ --conf spark.kyuubi.frontend.connection.url.use.hostname=false \ --conf spark.dynamicAllocation.shuffleTracking.enabled=true \ --conf spark.sql.warehouse.dir=s3a://<bucket_name>/ \ --conf spark.kyuubi.frontend.thrift.worker.keepalive.time=PT10M \ --conf spark.kyuubi.session.engine.idle.timeout=PT10M \ --conf spark.hadoop.fs.s3a.connection.ssl.enabled=true \ --conf spark.kyuubi.session.engine.check.interval=PT10M \ --conf spark.kubernetes.namespace=kyuubi-jobs \ --conf spark.eventLog.enabled=true \ --conf spark.kubernetes.driver.podTemplateFile=/opt/kyuubi/conf/pod_template.yml \ --conf spark.sql.adaptive.enabled=true \ --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \ --conf spark.kyuubi.session.check.interval=PT10M \ --conf spark.kyuubi.backend.server.exec.pool.keepalive.time=PT10M \ --conf spark.kyuubi.session.idle.timeout=PT10M \ --conf spark.kyuubi.engine.user.isolated.spark.session.idle.interval=PT10M \ --conf spark.eventLog.rotation.minFileSize=100m \ --conf spark.kyuubi.session.engine.startup.waitCompletion=true \ --conf spark.kyuubi.engine.submit.time=1674134327784 \ --conf spark.kubernetes.driver.label.kyuubi-unique-tag=35d34b4c-e204-485d-b559-8afde030e936 \ --conf spark.hadoop.fs.s3a.downgrade.syncable.exceptions=true \ --conf spark.app.name=kyuubi_CONNECTION_SPARK_SQL_anonymous_35d34b4c-e204-485d-b559-8afde030e936 \ --conf spark.kubernetes.driver.podNamePrefix=kyuubi-anonymous-driver \ --conf spark.eventLog.rotation.maxFilesToRetain=2 \ --conf spark.kyuubi.ha.addresses=kyuubi-zookeeper.zookeeper:2181 \ --conf spark.kyuubi.zookeeper.embedded.tick.time=30000 \ --conf spark.driver.memory=2g \ --conf spark.kyuubi.metrics.reporters=PROMETHEUS \ --conf spark.kyuubi.ha.zookeeper.session.timeout=60000000 \ --conf spark.eventLog.rotation.enabled=true \ --conf spark.kubernetes.authenticate.serviceAccountName=de-kyuubi \ --conf spark.kyuubi.batch.application.check.interval=PT10M \ --conf spark.sql.parquet.mergeSchema=true \ --conf spark.kyuubi.backend.engine.exec.pool.keepalive.time=PT10M \ --conf spark.kyuubi.ha.engine.ref.id=35d34b4c-e204-485d-b559-8afde030e936 \ --conf spark.dynamicAllocation.maxExecutors=20 \ --conf spark.kubernetes.container.image=harbor.dwh.runit.cc/de-image-spark/spark:v3.0.65 \ --conf spark.kubernetes.executor.podTemplateFile=/opt/kyuubi/conf/pod_template.yml \ --conf spark.dynamicAllocation.executorAllocationRatio=1 \ --conf spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp \ --conf spark.sql.sources.partitionOverwriteMode=dynamic \ --conf spark.submit.deployMode=cluster \ --conf spark.hadoop.fs.s3a.change.detection.mode=warn \ --conf spark.sql.broadcastTimeout=30000 \ --conf spark.kyuubi.zookeeper.embedded.max.session.timeout=6000000 \ --conf spark.master=k8s://https://kubernetes.default.svc:443 \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=de-kyuubi \ --conf spark.kubernetes.executor.podNamePrefix=kyuubi-anonymous \ --conf spark.kyuubi.engine.share.level=CONNECTION \ --conf spark.kyuubi.ha.zookeeper.namespace=kyuubi-de \ --conf spark.eventLog.dir=s3a://<logs_bucket_name>/ \ --conf spark.kyuubi.session.engine.alive.timeout=PT10M \ --conf spark.dynamicAllocation.enabled=true \ --conf spark.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token \ --conf spark.sql.legacy.parquet.datetimeRebaseModeInWrite=LEGACY \ --conf spark.kyuubi.client.ipAddress=192.168.56.1 \ --conf spark.kyuubi.ha.enabled=true \ --conf spark.hadoop.fs.s3a.endpoint=<s3_endpoint> \ --conf spark.kyuubi.zookeeper.embedded.min.session.timeout=600000 \ --conf spark.kyuubi.session.engine.alive.probe.interval=PT10M \ --conf spark.kubernetes.authenticate.caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \ --conf spark.dynamicAllocation.minExecutors=2 \ --conf spark.dynamicAllocation.initialExecutors=2 \ --conf spark.kubernetes.file.upload.path=s3a://<bucket_name> \ --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \ --conf spark.eventLog.rotation.interval=3600 \ --conf spark.dynamicAllocation.executorIdleTimeout=600s \ --conf spark.kyuubi.session.engine.login.timeout=PT10M \ --conf spark.kyuubi.ha.namespace=/kyuubi-de_1.6.1-incubating_CONNECTION_SPARK_SQL/anonymous/35d34b4c-e204-485d-b559-8afde030e936 \ --conf spark.hadoop.fs.s3a.fast.upload=true \ --conf spark.kubernetes.driverEnv.SPARK_USER_NAME=anonymous \ --conf spark.executorEnv.SPARK_USER_NAME=anonymous \ --proxy-user anonymous /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.6.1-incubating.jar 2023-01-19 13:18:47.787 INFO org.apache.kyuubi.engine.ProcBuilder: Logging to /opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.5 2023-01-19 13:19:16.833 INFO org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient: Get service instance:10.10.46.26:37197 and version:Some(1.6.1-incubating) under /kyuubi-de_1.6.1-incubating_CONNECTION_SPARK_SQL/anonymous/35d34b4c-e204-485d-b559-8afde030e936 2023-01-19 13:19:17.178 INFO org.apache.kyuubi.session.KyuubiSessionImpl: [anonymous:192.168.56.1] SessionHandle [35d34b4c-e204-485d-b559-8afde030e936] - Connected to engine [10.10.46.26:37197]/[spark-85247ce9c71c432290615c69a192b85e] with SessionHandle [0d8be737-c71d-4870-954b-831835390f09]] 2023-01-19 13:19:17.178 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting 2023-01-19 13:19:17.182 INFO org.apache.zookeeper.ZooKeeper: Session: 0x300720fdb0619e3 closed 2023-01-19 13:19:17.182 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x300720fdb0619e3 2023-01-19 13:19:17.182 INFO org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[02f35cb1-348b-458d-b9da-8f833b60cf21]: RUNNING_STATE -> FINISHED_STATE, time taken: 29.411 seconds 2023-01-19 13:19:17.238 INFO org.apache.kyuubi.operation.log.OperationLog: Creating operation log file /opt/kyuubi/work/server_operation_logs/35d34b4c-e204-485d-b559-8afde030e936/e568af82-e989-4750-94da-c2ddc6f95df9 2023-01-19 13:19:17.238 INFO org.apache.kyuubi.session.KyuubiSessionImpl: [anonymous:192.168.56.1] SessionHandle [35d34b4c-e204-485d-b559-8afde030e936] - Starting to wait the launch engine operation finished 2023-01-19 13:19:17.238 INFO org.apache.kyuubi.session.KyuubiSessionImpl: [anonymous:192.168.56.1] SessionHandle [35d34b4c-e204-485d-b559-8afde030e936] - Engine has been launched, elapsed time: 0 s 2023-01-19 13:19:17.275 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing anonymous's query[e568af82-e989-4750-94da-c2ddc6f95df9]: PENDING_STATE -> RUNNING_STATE, statement: _GET_CATALOG 2023-01-19 13:19:17.316 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[e568af82-e989-4750-94da-c2ddc6f95df9] in FINISHED_STATE 2023-01-19 13:19:17.316 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing anonymous's query[e568af82-e989-4750-94da-c2ddc6f95df9]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.041 seconds 2023-01-19 13:19:17.467 INFO org.apache.kyuubi.client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:54 BD FF 01 92 47 49 30 99 65 C1 D9 25 33 8F 5F, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side 2023-01-19 13:19:17.600 INFO org.apache.kyuubi.operation.GetTypeInfo: Processing anonymous's query[0757545a-952b-4fd8-beef-67a454168c7c]: INITIALIZED_STATE -> RUNNING_STATE, statement: GetTypeInfo 2023-01-19 13:19:17.615 INFO org.apache.kyuubi.operation.GetTypeInfo: Processing anonymous's query[0757545a-952b-4fd8-beef-67a454168c7c]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.015 seconds 2023-01-19 13:19:17.720 INFO org.apache.kyuubi.client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:59 42 36 BD 4D 08 42 BE 8E AF 7E 8F 5F C2 B8 31, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:GET_TYPE_INFO, hasResultSet:true)) succeed on engine side 2023-01-19 13:19:17.736 INFO org.apache.kyuubi.operation.GetCatalogs: Processing anonymous's query[abf2867b-40c5-4618-88d9-d3130498de34]: INITIALIZED_STATE -> RUNNING_STATE, statement: GetCatalogs 2023-01-19 13:19:17.753 INFO org.apache.kyuubi.operation.GetCatalogs: Processing anonymous's query[abf2867b-40c5-4618-88d9-d3130498de34]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.017 seconds 2023-01-19 13:19:17.819 INFO org.apache.kyuubi.client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:FC BE F7 72 34 8C 45 23 87 08 00 D8 56 BA 21 21, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:GET_CATALOGS, hasResultSet:true)) succeed on engine side 2023-01-19 13:20:07.821 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Session [SessionHandle [35d34b4c-e204-485d-b559-8afde030e936]] disconnected without closing properly, close it now 2023-01-19 13:20:07.821 INFO org.apache.kyuubi.session.KyuubiSessionManager: SessionHandle [35d34b4c-e204-485d-b559-8afde030e936] is closed, current opening sessions 0 2023-01-19 13:20:07.875 INFO org.apache.kyuubi.engine.ProcBuilder: Destroy the process, since waitCompletion is false. ``` ### Kyuubi Engine Log Output ```logtalk ++ id -u + myuid=185 ++ id -g + mygid=0 + set +e ++ getent passwd 185 + uidentry= + set -e + '[' -z '' ']' + '[' -w /etc/passwd ']' + echo 185:x:185:0:anonymous:/opt/spark:/bin/false + '[' -z /usr/local/openjdk-11 ']' + SPARK_CLASSPATH=':/opt/spark/jars/*' + env + grep SPARK_JAVA_OPT_ + sort -t_ -k4 -n + sed 's/[^=]*=\(.*\)/\1/g' + readarray -t SPARK_EXECUTOR_JAVA_OPTS + '[' -n '' ']' + '[' -z ']' + '[' -z ']' + '[' -n '' ']' + '[' -z x ']' + SPARK_CLASSPATH='/opt/hadoop/conf::/opt/spark/jars/*' + '[' -z x ']' + SPARK_CLASSPATH='/opt/spark/conf:/opt/hadoop/conf::/opt/spark/jars/*' + case "$1" in + shift 1 + CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@") + exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.10.46.26 --deploy-mode client --proxy-user anonymous --properties-file /opt/spark/conf/spark.properties --class org.apache.kyuubi.engine.spark.SparkSQLEngine spark-internal 23/01/19 13:18:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 23/01/19 13:18:58 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties 23/01/19 13:19:01 INFO SignalRegister: Registering signal handler for TERM 23/01/19 13:19:01 INFO SignalRegister: Registering signal handler for HUP 23/01/19 13:19:01 INFO SignalRegister: Registering signal handler for INT 23/01/19 13:19:01 INFO HiveConf: Found configuration file file:/opt/hadoop/conf/hive-site.xml 23/01/19 13:19:01 WARN KyuubiConf: The Kyuubi config 'kyuubi.ha.zookeeper.namespace' has been deprecated in Kyuubi v1.6.0 and may be removed in the future. Use kyuubi.ha.namespace instead 23/01/19 13:19:01 INFO SparkContext: Running Spark version 3.3.0 23/01/19 13:19:01 INFO ResourceUtils: ============================================================== 23/01/19 13:19:01 INFO ResourceUtils: No custom resources configured for spark.driver. 23/01/19 13:19:01 INFO ResourceUtils: ============================================================== 23/01/19 13:19:01 INFO SparkContext: Submitted application: kyuubi_CONNECTION_SPARK_SQL_anonymous_35d34b4c-e204-485d-b559-8afde030e936 23/01/19 13:19:01 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) 23/01/19 13:19:01 INFO ResourceProfile: Limiting resource is cpus at 1 tasks per executor 23/01/19 13:19:01 INFO ResourceProfileManager: Added ResourceProfile id: 0 23/01/19 13:19:01 INFO SecurityManager: Changing view acls to: 185,anonymous 23/01/19 13:19:01 INFO SecurityManager: Changing modify acls to: 185,anonymous 23/01/19 13:19:01 INFO SecurityManager: Changing view acls groups to: 23/01/19 13:19:01 INFO SecurityManager: Changing modify acls groups to: 23/01/19 13:19:01 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(185, anonymous); groups with view permissions: Set(); users with modify permissions: Set(185, anonymous); groups with modify permissions: Set() 23/01/19 13:19:01 INFO Utils: Successfully started service 'sparkDriver' on port 7078. 23/01/19 13:19:01 INFO SparkEnv: Registering MapOutputTracker 23/01/19 13:19:01 INFO SparkEnv: Registering BlockManagerMaster 23/01/19 13:19:01 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 23/01/19 13:19:01 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 23/01/19 13:19:01 INFO SparkEnv: Registering BlockManagerMasterHeartbeat 23/01/19 13:19:01 INFO DiskBlockManager: Created local directory at /var/data/spark-250920f8-2aa9-4458-99de-9aa60e944508/blockmgr-fa1c6bde-d5ca-4a17-a151-07b2f72f47d4 23/01/19 13:19:02 INFO MemoryStore: MemoryStore started with capacity 1048.8 MiB 23/01/19 13:19:02 INFO SparkEnv: Registering OutputCommitCoordinator 23/01/19 13:19:02 INFO Utils: Successfully started service 'SparkUI' on port 4040. 23/01/19 13:19:02 INFO SparkContext: Added JAR file:/tmp/spark-aff3b193-2214-4183-b374-38e30d7a5b28/kyuubi-spark-sql-engine_2.12-1.6.1-incubating.jar at spark://spark-25d8f985ca2ea186-driver-svc.kyuubi-jobs.svc:7078/jars/kyuubi-spark-sql-engine_2.12-1.6.1-incubating.jar with timestamp 1674134341366 23/01/19 13:19:02 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file 23/01/19 13:19:03 WARN Utils: spark.executor.instances less than spark.dynamicAllocation.minExecutors is invalid, ignoring its setting, please update your configs. 23/01/19 13:19:03 INFO Utils: Using initial executors = 2, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances 23/01/19 13:19:03 INFO ExecutorPodsAllocator: Going to request 2 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 0, sharedSlotFromPendingPods: 2147483647. 23/01/19 13:19:03 INFO KubernetesClientUtils: Spark configuration files loaded from Some(/opt/spark/conf) : log4j.properties,metrics.properties,jmx_config.yml 23/01/19 13:19:03 INFO KubernetesClientUtils: Spark configuration files loaded from Some(/opt/spark/conf) : log4j.properties,metrics.properties,jmx_config.yml 23/01/19 13:19:03 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script 23/01/19 13:19:03 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 7079. 23/01/19 13:19:03 INFO NettyBlockTransferService: Server created on spark-25d8f985ca2ea186-driver-svc.kyuubi-jobs.svc:7079 23/01/19 13:19:03 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 23/01/19 13:19:03 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, spark-25d8f985ca2ea186-driver-svc.kyuubi-jobs.svc, 7079, None) 23/01/19 13:19:03 INFO BlockManagerMasterEndpoint: Registering block manager spark-25d8f985ca2ea186-driver-svc.kyuubi-jobs.svc:7079 with 1048.8 MiB RAM, BlockManagerId(driver, spark-25d8f985ca2ea186-driver-svc.kyuubi-jobs.svc, 7079, None) 23/01/19 13:19:03 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, spark-25d8f985ca2ea186-driver-svc.kyuubi-jobs.svc, 7079, None) 23/01/19 13:19:03 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, spark-25d8f985ca2ea186-driver-svc.kyuubi-jobs.svc, 7079, None) 23/01/19 13:19:03 INFO KubernetesClientUtils: Spark configuration files loaded from Some(/opt/spark/conf) : log4j.properties,metrics.properties,jmx_config.yml 23/01/19 13:19:03 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script 23/01/19 13:19:03 INFO SingleEventLogFileWriter: Logging events to s3a://<logs_bucket_name>.inprogress 23/01/19 13:19:04 WARN S3ABlockOutputStream: Application invoked the Syncable API against stream writing to shared/spark-logs/spark-85247ce9c71c432290615c69a192b85e.inprogress. This is unsupported 23/01/19 13:19:04 WARN Utils: spark.executor.instances less than spark.dynamicAllocation.minExecutors is invalid, ignoring its setting, please update your configs. 23/01/19 13:19:04 INFO Utils: Using initial executors = 2, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances 23/01/19 13:19:04 INFO ExecutorAllocationManager: Dynamic allocation is enabled without a shuffle service. 23/01/19 13:19:08 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.10.110.198:59146) with ID 1, ResourceProfileId 0 23/01/19 13:19:08 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.10.81.195:57268) with ID 2, ResourceProfileId 0 23/01/19 13:19:08 INFO ExecutorMonitor: New executor 1 has registered (new total is 1) 23/01/19 13:19:08 INFO ExecutorMonitor: New executor 2 has registered (new total is 2) 23/01/19 13:19:08 INFO KubernetesClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 23/01/19 13:19:08 INFO BlockManagerMasterEndpoint: Registering block manager 10.10.81.195:40511 with 413.9 MiB RAM, BlockManagerId(2, 10.10.81.195, 40511, None) 23/01/19 13:19:08 INFO BlockManagerMasterEndpoint: Registering block manager 10.10.110.198:34749 with 413.9 MiB RAM, BlockManagerId(1, 10.10.110.198, 34749, None) 23/01/19 13:19:08 INFO SharedState: Setting hive.metastore.warehouse.dir ('s3a://<bucket_name>') to the value of spark.sql.warehouse.dir. 23/01/19 13:19:08 INFO SharedState: Warehouse path is 's3a://<bucket_name>'. 23/01/19 13:19:08 WARN SQLConf: The SQL config 'spark.sql.legacy.parquet.datetimeRebaseModeInWrite' has been deprecated in Spark v3.2 and may be removed in the future. Use 'spark.sql.parquet.datetimeRebaseModeInWrite' instead. 23/01/19 13:19:08 WARN SQLConf: The SQL config 'spark.sql.legacy.parquet.datetimeRebaseModeInWrite' has been deprecated in Spark v3.2 and may be removed in the future. Use 'spark.sql.parquet.datetimeRebaseModeInWrite' instead. 23/01/19 13:19:08 WARN SQLConf: The SQL config 'spark.sql.legacy.parquet.datetimeRebaseModeInWrite' has been deprecated in Spark v3.2 and may be removed in the future. Use 'spark.sql.parquet.datetimeRebaseModeInWrite' instead. 23/01/19 13:19:11 WARN SQLConf: The SQL config 'spark.sql.legacy.parquet.datetimeRebaseModeInWrite' has been deprecated in Spark v3.2 and may be removed in the future. Use 'spark.sql.parquet.datetimeRebaseModeInWrite' instead. 23/01/19 13:19:12 INFO HiveUtils: Initializing HiveMetastoreConnection version 2.3.9 using Spark classes. 23/01/19 13:19:12 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist 23/01/19 13:19:12 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist 23/01/19 13:19:12 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.9) is s3a://<bucket_name> 23/01/19 13:19:12 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist 23/01/19 13:19:12 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist 23/01/19 13:19:12 INFO HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 23/01/19 13:19:12 INFO ObjectStore: ObjectStore, initialize called 23/01/19 13:19:12 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 23/01/19 13:19:12 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored 23/01/19 13:19:13 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist 23/01/19 13:19:13 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist 23/01/19 13:19:13 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 23/01/19 13:19:14 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is POSTGRES 23/01/19 13:19:14 INFO ObjectStore: Initialized ObjectStore 23/01/19 13:19:14 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0 23/01/19 13:19:14 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore UNKNOWN@10.10.46.26 23/01/19 13:19:14 INFO HiveMetaStore: Added admin role in metastore 23/01/19 13:19:14 INFO HiveMetaStore: Added public role in metastore 23/01/19 13:19:14 INFO HiveMetaStore: No user is added in admin role, since config is empty 23/01/19 13:19:14 INFO HiveMetaStore: 0: get_database: default 23/01/19 13:19:14 INFO audit: ugi=anonymous ip=unknown-ip-addr cmd=get_database: default 23/01/19 13:19:14 INFO HiveMetaStore: 0: get_databases: * 23/01/19 13:19:14 INFO audit: ugi=anonymous ip=unknown-ip-addr cmd=get_databases: * 23/01/19 13:19:14 INFO CodeGenerator: Code generated in 218.772709 ms 23/01/19 13:19:14 INFO CodeGenerator: Code generated in 11.306738 ms 23/01/19 13:19:14 INFO CodeGenerator: Code generated in 15.345877 ms 23/01/19 13:19:15 INFO SparkContext: Starting job: isEmpty at KyuubiSparkUtil.scala:48 23/01/19 13:19:15 INFO DAGScheduler: Got job 0 (isEmpty at KyuubiSparkUtil.scala:48) with 1 output partitions 23/01/19 13:19:15 INFO DAGScheduler: Final stage: ResultStage 0 (isEmpty at KyuubiSparkUtil.scala:48) 23/01/19 13:19:15 INFO DAGScheduler: Parents of final stage: List() 23/01/19 13:19:15 INFO DAGScheduler: Missing parents: List() 23/01/19 13:19:15 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at isEmpty at KyuubiSparkUtil.scala:48), which has no missing parents 23/01/19 13:19:15 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 6.7 KiB, free 1048.8 MiB) 23/01/19 13:19:15 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.6 KiB, free 1048.8 MiB) 23/01/19 13:19:15 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on spark-25d8f985ca2ea186-driver-svc.kyuubi-jobs.svc:7079 (size: 3.6 KiB, free: 1048.8 MiB) 23/01/19 13:19:15 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1513 23/01/19 13:19:15 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at isEmpty at KyuubiSparkUtil.scala:48) (first 15 tasks are for partitions Vector(0)) 23/01/19 13:19:15 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 0 23/01/19 13:19:15 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (10.10.81.195, executor 2, partition 0, PROCESS_LOCAL, 8496 bytes) taskResourceAssignments Map() 23/01/19 13:19:15 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.10.81.195:40511 (size: 3.6 KiB, free: 413.9 MiB) 23/01/19 13:19:16 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 997 ms on 10.10.81.195 (executor 2) (1/1) 23/01/19 13:19:16 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 23/01/19 13:19:16 INFO DAGScheduler: ResultStage 0 (isEmpty at KyuubiSparkUtil.scala:48) finished in 1.233 s 23/01/19 13:19:16 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job 23/01/19 13:19:16 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished 23/01/19 13:19:16 INFO DAGScheduler: Job 0 finished: isEmpty at KyuubiSparkUtil.scala:48, took 1.286901 s 23/01/19 13:19:16 INFO ThreadUtils: SparkSQLSessionManager-exec-pool: pool size: 100, wait queue size: 100, thread keepalive time: 600000 ms 23/01/19 13:19:16 INFO SparkSQLOperationManager: Service[SparkSQLOperationManager] is initialized. 23/01/19 13:19:16 INFO SparkSQLSessionManager: Service[SparkSQLSessionManager] is initialized. 23/01/19 13:19:16 INFO SparkSQLBackendService: Service[SparkSQLBackendService] is initialized. 23/01/19 13:19:16 INFO SparkTBinaryFrontendService: Initializing SparkTBinaryFrontend on kyuubi-connection-spark-sql-anonymous-35d34b4c-e204-485d-b559-8:37197 with [9, 999] worker threads 23/01/19 13:19:16 INFO CuratorFrameworkImpl: Starting 23/01/19 13:19:16 INFO ZooKeeper: Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT 23/01/19 13:19:16 INFO ZooKeeper: Client environment:host.name=kyuubi-connection-spark-sql-anonymous-35d34b4c-e204-485d-b559-8 23/01/19 13:19:16 INFO ZooKeeper: Client environment:java.version=11.0.15 23/01/19 13:19:16 INFO ZooKeeper: Client environment:java.vendor=Oracle Corporation 23/01/19 13:19:16 INFO ZooKeeper: Client environment:java.home=/usr/local/openjdk-11 23/01/19 13:19:16 INFO ZooKeeper: Client environment:java.class.path=/opt/spark/conf/:/opt/spark/jars/bonecp-0.8.0.RELEASE.jar:/opt/spark/jars/osgi-resource-locator-1.0.3.jar:/opt/spark/jars/json-1.8.jar:/opt/spark/jars/parquet-column-1.12.2.jar:/opt/spark/jars/jersey-client-2.34.jar:/opt/spark/jars/snakeyaml-1.30.jar:/opt/spark/jars/jodd-core-3.5.2.jar:/opt/spark/jars/httpcore-4.4.14.jar:/opt/spark/jars/jersey-container-servlet-core-2.34.jar:/opt/spark/jars/jaxb-runtime-2.3.2.jar:/opt/spark/jars/netty-transport-native-kqueue-4.1.74.Final-osx-x86_64.jar:/opt/spark/jars/jackson-module-scala_2.12-2.13.3.jar:/opt/spark/jars/scala-xml_2.12-1.2.0.jar:/opt/spark/jars/jackson-mapper-asl-1.9.13.jar:/opt/spark/jars/jackson-databind-2.13.3.jar:/opt/spark/jars/spark-launcher_2.12-3.3.0.jar:/opt/spark/jars/kubernetes-model-core-5.12.2.jar:/opt/spark/jars/scala-compiler-2.12.15.jar:/opt/spark/jars/javassist-3.25.0-GA.jar:/opt/spark/jars/spark-kvstore_2.12-3.3.0.jar:/opt/spark/jars/hive-shims-0 .23-2.3.9.jar:/opt/spark/jars/kubernetes-model-extensions-5.12.2.jar:/opt/spark/jars/spark-kubernetes_2.12-3.3.0.jar:/opt/spark/jars/commons-lang3-3.12.0.jar:/opt/spark/jars/scala-collection-compat_2.12-2.1.1.jar:/opt/spark/jars/guava-14.0.1.jar:/opt/spark/jars/istack-commons-runtime-3.0.8.jar:/opt/spark/jars/commons-compress-1.21.jar:/opt/spark/jars/metrics-jmx-4.2.7.jar:/opt/spark/jars/spark-network-common_2.12-3.3.0.jar:/opt/spark/jars/arrow-vector-7.0.0.jar:/opt/spark/jars/breeze-macros_2.12-1.2.jar:/opt/spark/jars/volcano-model-v1beta1-5.12.2.jar:/opt/spark/jars/chill_2.12-0.10.0.jar:/opt/spark/jars/velocity-1.5.jar:/opt/spark/jars/parquet-encoding-1.12.2.jar:/opt/spark/jars/log4j-1.2-api-2.17.2.jar:/opt/spark/jars/jakarta.ws.rs-api-2.1.6.jar:/opt/spark/jars/jta-1.1.jar:/opt/spark/jars/datanucleus-rdbms-4.1.19.jar:/opt/spark/jars/jersey-hk2-2.34.jar:/opt/spark/jars/algebra_2.12-2.0.1.jar:/opt/spark/jars/jakarta.annotation-api-1.3.5.jar:/opt/spark/jars/rocksdbjni-6.20.3.jar:/opt /spark/jars/kubernetes-model-certificates-5.12.2.jar:/opt/spark/jars/arrow-format-7.0.0.jar:/opt/spark/jars/okhttp-3.12.12.jar:/opt/spark/jars/commons-collections-3.2.2.jar:/opt/spark/jars/netty-transport-classes-epoll-4.1.74.Final.jar:/opt/spark/jars/commons-collections4-4.4.jar:/opt/spark/jars/curator-recipes-2.13.0.jar:/opt/spark/jars/json4s-scalap_2.12-3.7.0-M11.jar:/opt/spark/jars/metrics-graphite-4.2.7.jar:/opt/spark/jars/jackson-annotations-2.13.3.jar:/opt/spark/jars/jersey-common-2.34.jar:/opt/spark/jars/spark-catalyst_2.12-3.3.0.jar:/opt/spark/jars/volcano-client-5.12.2.jar:/opt/spark/jars/zookeeper-3.6.2.jar:/opt/spark/jars/hive-shims-2.3.9.jar:/opt/spark/jars/antlr-runtime-3.5.2.jar:/opt/spark/jars/derby-10.14.2.0.jar:/opt/spark/jars/kubernetes-model-scheduling-5.12.2.jar:/opt/spark/jars/spark-mllib-local_2.12-3.3.0.jar:/opt/spark/jars/javolution-5.5.1.jar:/opt/spark/jars/janino-3.0.16.jar:/opt/spark/jars/kubernetes-model-admissionregistration-5.12.2.jar:/opt/spark/jars/p arquet-hadoop-1.12.2.jar:/opt/spark/jars/JTransforms-3.1.jar:/opt/spark/jars/avro-mapred-1.11.0.jar:/opt/spark/jars/spark-network-shuffle_2.12-3.3.0.jar:/opt/spark/jars/kubernetes-model-rbac-5.12.2.jar:/opt/spark/jars/metrics-json-4.2.7.jar:/opt/spark/jars/log4j-core-2.17.2.jar:/opt/spark/jars/kubernetes-model-apiextensions-5.12.2.jar:/opt/spark/jars/ST4-4.0.4.jar:/opt/spark/jars/spark-streaming_2.12-3.3.0.jar:/opt/spark/jars/core-1.1.2.jar:/opt/spark/jars/hive-shims-common-2.3.9.jar:/opt/spark/jars/okio-1.14.0.jar:/opt/spark/jars/shapeless_2.12-2.3.7.jar:/opt/spark/jars/netty-tcnative-classes-2.0.48.Final.jar:/opt/spark/jars/hadoop-client-api-3.3.4.jar:/opt/spark/jars/automaton-1.11-8.jar:/opt/spark/jars/hk2-locator-2.6.1.jar:/opt/spark/jars/snappy-java-1.1.8.4.jar:/opt/spark/jars/avro-1.11.0.jar:/opt/spark/jars/hive-vector-code-gen-2.3.9.jar:/opt/spark/jars/jakarta.servlet-api-4.0.3.jar:/opt/spark/jars/paranamer-2.8.jar:/opt/spark/jars/jline-2.14.6.jar:/opt/spark/jars/arrow-memory -core-7.0.0.jar:/opt/spark/jars/spark-sql_2.12-3.3.0.jar:/opt/spark/jars/hive-shims-scheduler-2.3.9.jar:/opt/spark/jars/commons-pool-1.5.4.jar:/opt/spark/jars/activation-1.1.1.jar:/opt/spark/jars/netty-transport-4.1.74.Final.jar:/opt/spark/jars/spire-util_2.12-0.17.0.jar:/opt/spark/jars/kubernetes-model-storageclass-5.12.2.jar:/opt/spark/jars/arrow-memory-netty-7.0.0.jar:/opt/spark/jars/aopalliance-repackaged-2.6.1.jar:/opt/spark/jars/spark-hive_2.12-3.3.0.jar:/opt/spark/jars/libfb303-0.9.3.jar:/opt/spark/jars/kryo-shaded-4.0.2.jar:/opt/spark/jars/json4s-ast_2.12-3.7.0-M11.jar:/opt/spark/jars/tink-1.6.1.jar:/opt/spark/jars/arpack-2.2.1.jar:/opt/spark/jars/spark-sketch_2.12-3.3.0.jar:/opt/spark/jars/antlr4-runtime-4.8.jar:/opt/spark/jars/netty-all-4.1.74.Final.jar:/opt/spark/jars/netty-transport-native-kqueue-4.1.74.Final-osx-aarch_64.jar:/opt/spark/jars/metrics-core-4.2.7.jar:/opt/spark/jars/commons-crypto-1.1.0.jar:/opt/spark/jars/spark-unsafe_2.12-3.3.0.jar:/opt/spark/jars/objenes is-3.2.jar:/opt/spark/jars/stream-2.9.6.jar:/opt/spark/jars/jackson-dataformat-yaml-2.13.3.jar:/opt/spark/jars/jul-to-slf4j-1.7.32.jar:/opt/spark/jars/lapack-2.2.1.jar:/opt/spark/jars/zjsonpatch-0.3.0.jar:/opt/spark/jars/orc-shims-1.7.4.jar:/opt/spark/jars/spire-platform_2.12-0.17.0.jar:/opt/spark/jars/netty-common-4.1.74.Final.jar:/opt/spark/jars/spire_2.12-0.17.0.jar:/opt/spark/jars/netty-transport-native-epoll-4.1.74.Final-linux-aarch_64.jar:/opt/spark/jars/jdo-api-3.0.1.jar:/opt/spark/jars/hive-metastore-2.3.9.jar:/opt/spark/jars/scala-library-2.12.15.jar:/opt/spark/jars/jackson-core-asl-1.9.13.jar:/opt/spark/jars/netty-transport-native-unix-common-4.1.74.Final.jar:/opt/spark/jars/threeten-extra-1.5.0.jar:/opt/spark/jars/jackson-datatype-jsr310-2.13.3.jar:/opt/spark/jars/netty-resolver-4.1.74.Final.jar:/opt/spark/jars/lz4-java-1.8.0.jar:/opt/spark/jars/kubernetes-model-networking-5.12.2.jar:/opt/spark/jars/kubernetes-client-5.12.2.jar:/opt/spark/jars/jersey-container-servlet-2.3 4.jar:/opt/spark/jars/hk2-api-2.6.1.jar:/opt/spark/jars/commons-dbcp-1.4.jar:/opt/spark/jars/commons-codec-1.15.jar:/opt/spark/jars/pickle-1.2.jar:/opt/spark/jars/kubernetes-model-coordination-5.12.2.jar:/opt/spark/jars/libthrift-0.12.0.jar:/opt/spark/jars/spark-tags_2.12-3.3.0-tests.jar:/opt/spark/jars/HikariCP-2.5.1.jar:/opt/spark/jars/univocity-parsers-2.9.1.jar:/opt/spark/jars/shims-0.9.25.jar:/opt/spark/jars/stax-api-1.0.1.jar:/opt/spark/jars/netty-transport-classes-kqueue-4.1.74.Final.jar:/opt/spark/jars/log4j-api-2.17.2.jar:/opt/spark/jars/RoaringBitmap-0.9.25.jar:/opt/spark/jars/parquet-common-1.12.2.jar:/opt/spark/jars/netty-handler-4.1.74.Final.jar:/opt/spark/jars/spark-repl_2.12-3.3.0.jar:/opt/spark/jars/netty-buffer-4.1.74.Final.jar:/opt/spark/jars/spark-mllib_2.12-3.3.0.jar:/opt/spark/jars/kubernetes-model-flowcontrol-5.12.2.jar:/opt/spark/jars/oro-2.0.8.jar:/opt/spark/jars/annotations-17.0.0.jar:/opt/spark/jars/flatbuffers-java-1.12.0.jar:/opt/spark/jars/xz-1.8.jar:/op t/spark/jars/commons-compiler-3.0.16.jar:/opt/spark/jars/commons-math3-3.6.1.jar:/opt/spark/jars/zookeeper-jute-3.6.2.jar:/opt/spark/jars/hive-serde-2.3.9.jar:/opt/spark/jars/opencsv-2.3.jar:/opt/spark/jars/generex-1.0.2.jar:/opt/spark/jars/JLargeArrays-1.5.jar:/opt/spark/jars/json4s-jackson_2.12-3.7.0-M11.jar:/opt/spark/jars/audience-annotations-0.5.0.jar:/opt/spark/jars/commons-cli-1.5.0.jar:/opt/spark/jars/py4j-0.10.9.5.jar:/opt/spark/jars/hive-llap-common-2.3.9.jar:/opt/spark/jars/curator-framework-2.13.0.jar:/opt/spark/jars/cats-kernel_2.12-2.1.1.jar:/opt/spark/jars/jcl-over-slf4j-1.7.32.jar:/opt/spark/jars/httpclient-4.5.13.jar:/opt/spark/jars/netty-codec-4.1.74.Final.jar:/opt/spark/jars/commons-lang-2.6.jar:/opt/spark/jars/curator-client-2.13.0.jar:/opt/spark/jars/jakarta.xml.bind-api-2.3.2.jar:/opt/spark/jars/jsr305-3.0.0.jar:/opt/spark/jars/log4j-slf4j-impl-2.17.2.jar:/opt/spark/jars/datanucleus-core-4.1.17.jar:/opt/spark/jars/kubernetes-model-node-5.12.2.jar:/opt/spark/jar s/logging-interceptor-3.12.12.jar:/opt/spark/jars/slf4j-api-1.7.32.jar:/opt/spark/jars/commons-text-1.9.jar:/opt/spark/jars/kubernetes-model-events-5.12.2.jar:/opt/spark/jars/blas-2.2.1.jar:/opt/spark/jars/scala-parser-combinators_2.12-1.1.2.jar:/opt/spark/jars/protobuf-java-2.5.0.jar:/opt/spark/jars/zstd-jni-1.5.2-1.jar:/opt/spark/jars/netty-transport-native-epoll-4.1.74.Final-linux-x86_64.jar:/opt/spark/jars/jersey-server-2.34.jar:/opt/spark/jars/orc-core-1.7.4.jar:/opt/spark/jars/transaction-api-1.1.jar:/opt/spark/jars/hive-exec-2.3.9-core.jar:/opt/spark/jars/spire-macros_2.12-0.17.0.jar:/opt/spark/jars/hadoop-client-runtime-3.3.4.jar:/opt/spark/jars/parquet-format-structures-1.12.2.jar:/opt/spark/jars/datanucleus-api-jdo-4.2.4.jar:/opt/spark/jars/kubernetes-model-metrics-5.12.2.jar:/opt/spark/jars/aircompressor-0.21.jar:/opt/spark/jars/parquet-jackson-1.12.2.jar:/opt/spark/jars/hk2-utils-2.6.1.jar:/opt/spark/jars/kubernetes-model-batch-5.12.2.jar:/opt/spark/jars/kubernetes-model -autoscaling-5.12.2.jar:/opt/spark/jars/kubernetes-model-policy-5.12.2.jar:/opt/spark/jars/orc-mapreduce-1.7.4.jar:/opt/spark/jars/avro-ipc-1.11.0.jar:/opt/spark/jars/spark-graphx_2.12-3.3.0.jar:/opt/spark/jars/kubernetes-model-apps-5.12.2.jar:/opt/spark/jars/javax.jdo-3.2.0-m3.jar:/opt/spark/jars/spark-core_2.12-3.3.0.jar:/opt/spark/jars/jakarta.validation-api-2.0.2.jar:/opt/spark/jars/commons-io-2.11.0.jar:/opt/spark/jars/compress-lzf-1.1.jar:/opt/spark/jars/arpack_combined_all-0.1.jar:/opt/spark/jars/hive-storage-api-2.7.2.jar:/opt/spark/jars/kubernetes-model-common-5.12.2.jar:/opt/spark/jars/hive-common-2.3.9.jar:/opt/spark/jars/jackson-core-2.13.3.jar:/opt/spark/jars/joda-time-2.10.13.jar:/opt/spark/jars/gson-2.2.4.jar:/opt/spark/jars/leveldbjni-all-1.8.jar:/opt/spark/jars/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/spark/jars/chill-java-0.10.0.jar:/opt/spark/jars/breeze_2.12-1.2.jar:/opt/spark/jars/json4s-core_2.12-3.7.0-M11.jar:/opt/spark/jars/kubernetes-model- discovery-5.12.2.jar:/opt/spark/jars/spark-tags_2.12-3.3.0.jar:/opt/spark/jars/jakarta.inject-2.6.1.jar:/opt/spark/jars/minlog-1.3.0.jar:/opt/spark/jars/xbean-asm9-shaded-4.20.jar:/opt/spark/jars/ivy-2.5.0.jar:/opt/spark/jars/scala-reflect-2.12.15.jar:/opt/spark/jars/commons-logging-1.1.3.jar:/opt/spark/jars/metrics-jvm-4.2.7.jar:/opt/spark/jars/uap-scala_2.12-0.11.0.jar:/opt/spark/jars/common-config-5.3.4.jar:/opt/spark/jars/hive-service-rpc-3.1.2.jar:/opt/spark/jars/spark-avro_2.12-3.1.2.jar:/opt/spark/jars/jackson-annotations-2.9.10.jar:/opt/spark/jars/spark-sql-kafka-0-10_2.12-3.1.2.jar:/opt/spark/jars/commons-pool2-2.6.2.jar:/opt/spark/jars/avro-1.8.1.jar:/opt/spark/jars/kafka-clients-2.6.0.jar:/opt/spark/jars/jmx_prometheus_javaagent-0.16.1.jar:/opt/spark/jars/slf4j-api-1.7.30.jar:/opt/spark/jars/paranamer-2.7.jar:/opt/spark/jars/zkclient-0.10.jar:/opt/spark/jars/commons_2.12-0.1.0.jar:/opt/spark/jars/postgresql-42.2.20.jar:/opt/spark/jars/spark-token-provider-kafka-0-10_2.12- 3.1.2.jar:/opt/spark/jars/hadoop-aws-3.3.4.jar:/opt/spark/jars/lz4-java-1.7.1.jar:/opt/spark/jars/jackson-core-2.9.10.jar:/opt/spark/jars/abris_2.12-5.1.1.jar:/opt/spark/jars/jackson-databind-2.9.10.5.jar:/opt/spark/jars/snappy-java-1.1.8.2.jar:/opt/spark/jars/spotbugs-annotations-3.1.8.jar:/opt/spark/jars/zookeeper-3.4.14.jar:/opt/spark/jars/common-utils-5.3.4.jar:/opt/spark/jars/aws-java-sdk-bundle-1.12.81.jar:/opt/spark/jars/commons-compress-1.8.1.jar:/opt/spark/jars/joda-time-2.10.6.jar:/opt/spark/jars/xz-1.5.jar:/opt/spark/jars/zstd-jni-1.4.8-1.jar:/opt/spark/jars/kafka-avro-serializer-5.3.4.jar:/opt/spark/jars/netty-3.10.6.Final.jar:/opt/spark/jars/unused-1.0.0.jar:/opt/spark/jars/kafka-schema-registry-client-5.3.4.jar:/opt/spark/jars/jline-0.9.94.jar:/opt/hadoop/conf/ 23/01/19 13:19:16 INFO ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib 23/01/19 13:19:16 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp 23/01/19 13:19:16 INFO ZooKeeper: Client environment:java.compiler=<NA> 23/01/19 13:19:16 INFO ZooKeeper: Client environment:os.name=Linux 23/01/19 13:19:16 INFO ZooKeeper: Client environment:os.arch=amd64 23/01/19 13:19:16 INFO ZooKeeper: Client environment:os.version=4.15.0-192-generic 23/01/19 13:19:16 INFO ZooKeeper: Client environment:user.name=185 23/01/19 13:19:16 INFO ZooKeeper: Client environment:user.home=/opt/spark 23/01/19 13:19:16 INFO ZooKeeper: Client environment:user.dir=/opt/spark/work-dir 23/01/19 13:19:16 INFO ZooKeeper: Initiating client connection, connectString=kyuubi-zookeeper.zookeeper:2181 sessionTimeout=60000000 watcher=org.apache.kyuubi.shade.org.apache.curator.ConnectionState@fffdd40 23/01/19 13:19:16 INFO EngineServiceDiscovery: Service[EngineServiceDiscovery] is initialized. 23/01/19 13:19:16 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is initialized. 23/01/19 13:19:16 INFO SparkSQLEngine: Service[SparkSQLEngine] is initialized. 23/01/19 13:19:16 INFO ClientCnxn: Opening socket connection to server kyuubi-zookeeper.zookeeper/10.108.54.123:2181. Will not attempt to authenticate using SASL (unknown error) 23/01/19 13:19:16 INFO ClientCnxn: Socket connection established to kyuubi-zookeeper.zookeeper/10.108.54.123:2181, initiating session 23/01/19 13:19:16 INFO SparkSQLOperationManager: Service[SparkSQLOperationManager] is started. 23/01/19 13:19:16 INFO SparkSQLSessionManager: Service[SparkSQLSessionManager] is started. 23/01/19 13:19:16 INFO SparkSQLBackendService: Service[SparkSQLBackendService] is started. 23/01/19 13:19:16 INFO ClientCnxn: Session establishment complete on server kyuubi-zookeeper.zookeeper/10.108.54.123:2181, sessionid = 0x200720fd76819a6, negotiated timeout = 40000 23/01/19 13:19:16 INFO ConnectionStateManager: State change: CONNECTED 23/01/19 13:19:16 INFO ZookeeperDiscoveryClient: Zookeeper client connection state changed to: CONNECTED 23/01/19 13:19:16 INFO ZookeeperDiscoveryClient: Created a /kyuubi-de_1.6.1-incubating_CONNECTION_SPARK_SQL/anonymous/35d34b4c-e204-485d-b559-8afde030e936/serviceUri=10.10.46.26:37197;version=1.6.1-incubating;refId=35d34b4c-e204-485d-b559-8afde030e936;sequence=0000000000 on ZooKeeper for KyuubiServer uri: 10.10.46.26:37197 23/01/19 13:19:16 INFO EngineServiceDiscovery: Service[EngineServiceDiscovery] is started. 23/01/19 13:19:16 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is started. 23/01/19 13:19:16 INFO SparkSQLEngine: Service[SparkSQLEngine] is started. 23/01/19 13:19:16 INFO SparkSQLEngine: Spark application name: kyuubi_CONNECTION_SPARK_SQL_anonymous_35d34b4c-e204-485d-b559-8afde030e936 application ID: spark-85247ce9c71c432290615c69a192b85e application tags: application web UI: http://spark-25d8f985ca2ea186-driver-svc.kyuubi-jobs.svc:4040 master: k8s://https://kubernetes.default.svc:443 version: 3.3.0 driver: [cpu: 1, mem: 2g] executor: [cpu: 2, mem: 1g, maxNum: 20] Start time: Thu Jan 19 13:19:01 UTC 2023 User: anonymous (shared mode: CONNECTION) State: STARTED 23/01/19 13:19:16 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V10 23/01/19 13:19:16 INFO SparkSQLSessionManager: Opening session for anonymous@10.10.51.9 23/01/19 13:19:16 WARN SparkSessionImpl: Cannot modify the value of a Spark config: spark.driver.memory. See also 'https://spark.apache.org/docs/latest/sql-migration-guide.html#ddl-statements' 23/01/19 13:19:16 INFO HiveMetaStore: 1: get_database: global_temp 23/01/19 13:19:16 INFO audit: ugi=anonymous ip=unknown-ip-addr cmd=get_database: global_temp 23/01/19 13:19:16 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist 23/01/19 13:19:16 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist 23/01/19 13:19:16 INFO HiveMetaStore: 1: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 23/01/19 13:19:16 INFO ObjectStore: ObjectStore, initialize called 23/01/19 13:19:16 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is POSTGRES 23/01/19 13:19:16 INFO ObjectStore: Initialized ObjectStore 23/01/19 13:19:16 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException 23/01/19 13:19:16 INFO HiveMetaStore: 1: get_database: default 23/01/19 13:19:16 INFO audit: ugi=anonymous ip=unknown-ip-addr cmd=get_database: default 23/01/19 13:19:16 WARN SparkSessionImpl: Cannot modify the value of a Spark config: spark.kubernetes.executor.podNamePrefix. See also 'https://spark.apache.org/docs/latest/sql-migration-guide.html#ddl-statements' 23/01/19 13:19:16 WARN SparkSessionImpl: Cannot modify the value of a Spark config: spark.dynamicAllocation.minExecutors. See also 'https://spark.apache.org/docs/latest/sql-migration-guide.html#ddl-statements' 23/01/19 13:19:17 INFO BlockManagerInfo: Removed broadcast_0_piece0 on spark-25d8f985ca2ea186-driver-svc.kyuubi-jobs.svc:7079 in memory (size: 3.6 KiB, free: 1048.8 MiB) 23/01/19 13:19:17 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 10.10.81.195:40511 in memory (size: 3.6 KiB, free: 413.9 MiB) 23/01/19 13:19:17 INFO SparkSQLSessionManager: anonymous's session with SessionHandle [0d8be737-c71d-4870-954b-831835390f09] is opened, current opening sessions 1 23/01/19 13:19:17 INFO GetCurrentCatalog: Processing anonymous's query[54bdff01-9247-4930-9965-c1d925338f5f]: INITIALIZED_STATE -> RUNNING_STATE, statement: GetCurrentCatalog 23/01/19 13:19:17 INFO GetCurrentCatalog: Processing anonymous's query[54bdff01-9247-4930-9965-c1d925338f5f]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.004 seconds 23/01/19 13:19:17 ERROR SparkTBinaryFrontendService: Error fetching results: org.apache.kyuubi.KyuubiSQLException: OperationHandle [54bdff01-9247-4930-9965-c1d925338f5f] failed to generate operation log at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.operation.OperationManager.$anonfun$getOperationLogRowSet$2(OperationManager.scala:146) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.operation.OperationManager.getOperationLogRowSet(OperationManager.scala:146) at org.apache.kyuubi.session.AbstractSession.fetchResults(AbstractSession.scala:236) at org.apache.kyuubi.service.AbstractBackendService.fetchResults(AbstractBackendService.scala:204) at org.apache.kyuubi.service.TFrontendService.FetchResults(TFrontendService.scala:520) at org.apache.kyuubi.shade.org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) at org.apache.kyuubi.shade.org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) at org.apache.kyuubi.shade.org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.kyuubi.shade.org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.kyuubi.shade.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) 23/01/19 13:19:17 ERROR SparkTBinaryFrontendService: Error fetching results: org.apache.kyuubi.KyuubiSQLException: OperationHandle [54bdff01-9247-4930-9965-c1d925338f5f] failed to generate operation log at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.operation.OperationManager.$anonfun$getOperationLogRowSet$2(OperationManager.scala:146) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.operation.OperationManager.getOperationLogRowSet(OperationManager.scala:146) at org.apache.kyuubi.session.AbstractSession.fetchResults(AbstractSession.scala:236) at org.apache.kyuubi.service.AbstractBackendService.fetchResults(AbstractBackendService.scala:204) at org.apache.kyuubi.service.TFrontendService.FetchResults(TFrontendService.scala:520) at org.apache.kyuubi.shade.org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) at org.apache.kyuubi.shade.org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) at org.apache.kyuubi.shade.org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.kyuubi.shade.org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.kyuubi.shade.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) 23/01/19 13:19:17 INFO GetTypeInfo: Processing anonymous's query[594236bd-4d08-42be-8eaf-7e8f5fc2b831]: INITIALIZED_STATE -> RUNNING_STATE, statement: GetTypeInfo 23/01/19 13:19:17 INFO GetTypeInfo: Processing anonymous's query[594236bd-4d08-42be-8eaf-7e8f5fc2b831]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.001 seconds 23/01/19 13:19:17 INFO GetCatalogs: Processing anonymous's query[fcbef772-348c-4523-8708-00d856ba2121]: INITIALIZED_STATE -> RUNNING_STATE, statement: GetCatalogs 23/01/19 13:19:17 INFO GetCatalogs: Processing anonymous's query[fcbef772-348c-4523-8708-00d856ba2121]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.004 seconds 23/01/19 13:20:07 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [0d8be737-c71d-4870-954b-831835390f09] 23/01/19 13:20:07 INFO SparkSQLSessionManager: SessionHandle [0d8be737-c71d-4870-954b-831835390f09] is closed, current opening sessions 0 23/01/19 13:20:07 INFO SparkSQLSessionManager: Session stopped due to shared level is Connection. 23/01/19 13:20:07 INFO SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping. 23/01/19 13:20:07 INFO SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping. 23/01/19 13:20:07 WARN ZookeeperDiscoveryClient: This Kyuubi instance 10.10.46.26:37197 is now de-registered from ZooKeeper. The server will be shut down after the last client session completes. 23/01/19 13:20:07 INFO EngineServiceDiscovery: Clean up discovery service due to this is connection share level. 23/01/19 13:20:07 INFO CuratorFrameworkImpl: backgroundOperationsLoop exiting 23/01/19 13:20:07 INFO ZooKeeper: Session: 0x200720fd76819a6 closed 23/01/19 13:20:07 INFO ClientCnxn: EventThread shut down for session: 0x200720fd76819a6 23/01/19 13:20:07 INFO EngineServiceDiscovery: Service[EngineServiceDiscovery] is stopped. 23/01/19 13:20:07 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is stopped. 23/01/19 13:20:07 INFO SparkTBinaryFrontendService: SparkTBinaryFrontend has stopped 23/01/19 13:20:07 INFO SparkSQLEngine: Service: [SparkSQLBackendService] is stopping. 23/01/19 13:20:07 INFO SparkSQLBackendService: Service: [SparkSQLSessionManager] is stopping. 23/01/19 13:20:07 INFO SparkSQLSessionManager: Service: [SparkSQLOperationManager] is stopping. 23/01/19 13:20:07 INFO SparkSQLOperationManager: Service[SparkSQLOperationManager] is stopped. 23/01/19 13:20:07 INFO SparkSQLSessionManager: Service[SparkSQLSessionManager] is stopped. 23/01/19 13:20:07 INFO SparkSQLBackendService: Service[SparkSQLBackendService] is stopped. 23/01/19 13:20:07 INFO SparkSQLEngine: Service[SparkSQLEngine] is stopped. 23/01/19 13:20:07 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [0d8be737-c71d-4870-954b-831835390f09] 23/01/19 13:20:07 INFO SparkUI: Stopped Spark web UI at http://spark-25d8f985ca2ea186-driver-svc.kyuubi-jobs.svc:4040 23/01/19 13:20:07 INFO KubernetesClusterSchedulerBackend: Shutting down all executors 23/01/19 13:20:07 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down 23/01/19 13:20:07 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed. 23/01/19 13:20:08 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 23/01/19 13:20:08 INFO MemoryStore: MemoryStore cleared 23/01/19 13:20:08 INFO BlockManager: BlockManager stopped 23/01/19 13:20:08 INFO BlockManagerMaster: BlockManagerMaster stopped 23/01/19 13:20:08 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 23/01/19 13:20:08 INFO SparkContext: Successfully stopped SparkContext 23/01/19 13:20:08 INFO ShutdownHookManager: Shutdown hook called 23/01/19 13:20:08 INFO ShutdownHookManager: Deleting directory /var/data/spark-250920f8-2aa9-4458-99de-9aa60e944508/spark-3f250941-abb3-43cb-85dd-f42cbd079f4e 23/01/19 13:20:08 INFO ShutdownHookManager: Deleting directory /tmp/spark-2c6ca710-f0ef-4024-b373-e3d24d67b819 23/01/19 13:20:08 INFO ShutdownHookManager: Deleting directory /tmp/spark-aff3b193-2214-4183-b374-38e30d7a5b28 23/01/19 13:20:08 INFO ShutdownHookManager: Deleting directory /tmp/spark-d5fd5670-e8c9-465a-b657-dbf1ab4b83bd 23/01/19 13:20:08 INFO ShutdownHookManager: Deleting directory /tmp/spark-9c76c8ef-8bf3-4436-8cde-a9cf8a428ec8 23/01/19 13:20:08 INFO MetricsSystemImpl: Stopping s3a-file-system metrics system... 23/01/19 13:20:08 INFO MetricsSystemImpl: s3a-file-system metrics system stopped. 23/01/19 13:20:08 INFO MetricsSystemImpl: s3a-file-system metrics system shutdown complete. Stream closed EOF for kyuubi-jobs/kyuubi-connection-spark-sql-anonymous-35d34b4c-e204-485d-b559-8afde030e936-3d6f5585ca2e9c90-driver (kyuubi-spark-driver) ``` ### Kyuubi Server Configurations ```yaml # Kyuubi configurations https://kyuubi.apache.org/docs/latest/deployment/settings.html kyuubi.authentication=NONE kyuubi.engine.share.level=CONNECTION kyuubi.frontend.bind.host={{ .Values.server.bind.host }} kyuubi.frontend.bind.port={{ .Values.server.bind.port }} kyuubi.metrics.reporters=PROMETHEUS # Zookeeper required for kyuubi HA kyuubi.ha.enabled=true kyuubi.ha.addresses=kyuubi-zookeeper.zookeeper:2181 # Required for k8s CLUSTER mode kyuubi.frontend.connection.url.use.hostname=false # Timeouts kyuubi.backend.engine.exec.pool.keepalive.time=PT10M kyuubi.backend.server.exec.pool.keepalive.time=PT10M kyuubi.batch.application.check.interval=PT10M kyuubi.engine.user.isolated.spark.session.idle.interval=PT10M kyuubi.frontend.thrift.worker.keepalive.time=PT10M kyuubi.ha.zookeeper.session.timeout=60000000 kyuubi.zookeeper.embedded.min.session.timeout=600000 kyuubi.zookeeper.embedded.max.session.timeout=6000000 kyuubi.zookeeper.embedded.tick.time=30000 kyuubi.session.engine.alive.timeout=PT10M kyuubi.session.engine.idle.timeout=PT10M kyuubi.session.engine.check.interval=PT10M kyuubi.session.engine.alive.probe.interval=PT10M kyuubi.session.engine.login.timeout=PT10M kyuubi.session.check.interval=PT10M kyuubi.session.idle.timeout=PT10M kyuubi.session.engine.startup.waitCompletion=true # Spark configurations spark.master=k8s://https://kubernetes.default.svc:443 spark.submit.deployMode=cluster spark.kubernetes.container.image=harbor.dwh.runit.cc/de-image-spark/spark:v3.0.65 spark.kubernetes.namespace={{ .Values.sparkNamespace }} spark.kubernetes.authenticate.driver.serviceAccountName={{ .Release.Name }}-kyuubi spark.kubernetes.authenticate.serviceAccountName={{ .Release.Name }}-kyuubi # Cert & token required to connect to k8s api spark.kubernetes.authenticate.caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt spark.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token # Required for k8s CLUSTER mode, to transfer kyuubi sqlengine jar to driver pod spark.kubernetes.file.upload.path=s3a://<bucket_name> spark.dynamicAllocation.enabled=true spark.dynamicAllocation.minExecutors=2 spark.dynamicAllocation.maxExecutors=20 spark.dynamicAllocation.initialExecutors=2 spark.dynamicAllocation.shuffleTracking.enabled=true spark.dynamicAllocation.executorAllocationRatio=1 spark.kubernetes.driver.podNamePrefix=kyuubi-{{ .Release.Name }} spark.kubernetes.executor.podNamePrefix=kyuubi-{{ .Release.Name }} kyuubi.ha.zookeeper.namespace=kyuubi-{{ .Release.Name }} spark.driver.memory=4g # Timeouts spark.dynamicAllocation.executorIdleTimeout=600s # S3 spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem spark.hadoop.fs.s3a.connection.ssl.enabled=true spark.hadoop.fs.s3a.fast.upload=true spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp spark.hadoop.fs.s3a.endpoint=https://<endpoint> spark.hadoop.fs.s3a.downgrade.syncable.exceptions=true spark.hadoop.fs.s3a.change.detection.mode=warn # Logs spark.eventLog.dir=s3a://<logs_bucket_name> spark.eventLog.enabled=true spark.eventLog.rotation.enabled=true spark.eventLog.rotation.interval=3600 spark.eventLog.rotation.minFileSize=100m spark.eventLog.rotation.maxFilesToRetain=2 # Spark SQL spark.sql.adaptive.enabled=true spark.sql.broadcastTimeout=30000 spark.sql.legacy.parquet.datetimeRebaseModeInWrite=LEGACY spark.sql.parquet.mergeSchema=true spark.sql.sources.partitionOverwriteMode=dynamic spark.sql.warehouse.dir={{ .Values.spark_sql_warehouse_dir }} # path to spark pod template spark.kubernetes.driver.podTemplateFile=/opt/kyuubi/conf/pod_template.yml spark.kubernetes.executor.podTemplateFile=/opt/kyuubi/conf/pod_template.yml ``` ### Kyuubi Engine Configurations _No response_ ### Additional context _No response_ ### Are you willing to submit PR? - [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix. - [X] No. I cannot submit a PR at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@kyuubi.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: notifications-unsubscr...@kyuubi.apache.org For additional commands, e-mail: notifications-h...@kyuubi.apache.org
