chengwei977 opened a new issue, #3478: URL: https://github.com/apache/incubator-kyuubi/issues/3478
### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the bug kyuubi runs in CDH6.3.2 environment, I compiled Spark3.1.2 on CDH6.3.2 version, last week my hue client connected to kyuubi 1.5.2-incubating was still normal, but since I upgraded to 1.6 and started hue client It's abnormal. A point worth explaining: After I upgraded kyuubi to 1.6, the configuration still uses the configuration of version 1.5.2 ### Affects Version(s) 1.6.0-incubating ### Kyuubi Server Log Output ```logtalk [root@con01 bin]# ./kyuubi Warn: Not find kyuubi environment file /opt/apache-kyuubi-1.6.0-incubating-bin/conf/kyuubi-env.sh, using default ones... Starting org.apache.kyuubi.server.KyuubiServer, logging to /opt/apache-kyuubi-1.6.0-incubating-bin/logs/kyuubi-root-org.apache.kyuubi.server.KyuubiServer-con01.xuncetech.com.out Welcome to __ __ __ /` \/` \ /` \ __ \ \ \/ / __ __ __ __ __ __\ \ \____ /\_\ \ \ , < /` \/` \/` \/` \/` \/` \\ \ \__` \/` \ \ \ \ \`\\ \ \_\ \\ \_\ \\ \_\ \\ \ \L\ \ \ \ \ \_\ \_ \/`____ \ \____/ \ \____`\ \_,__/ \ \_\ \/_/\/_/ `/___/> \/___/ \/___/ \/___/ \/_/ /\___/ \/__\/ [root@con01 bin]# tail -30f ../logs/kyuubi-root-org.apache.kyuubi.server.KyuubiServer-con01.xuncetech.com.out 2022-09-12 11:26:50.080 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2022-09-12 11:26:50.081 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2022-09-12 11:26:50.082 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux 2022-09-12 11:26:50.083 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64 2022-09-12 11:26:50.083 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.10.0-957.el7.x86_64 2022-09-12 11:26:50.083 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=root 2022-09-12 11:26:50.083 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/root 2022-09-12 11:26:50.083 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/opt/apache-kyuubi-1.6.0-incubating-bin/bin 2022-09-12 11:26:50.084 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=con01.xuncetech.com,con02.xuncetech.com,con03.xuncetech.com sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@6ea94d6a 2022-09-12 11:26:50.101 INFO org.apache.kyuubi.ha.client.KyuubiServiceDiscovery: Service[KyuubiServiceDiscovery] is initialized. 2022-09-12 11:26:50.103 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Service[KyuubiTBinaryFrontend] is initialized. 2022-09-12 11:26:50.104 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server con02.xuncetech.com/192.168.0.106:2181. Will not attempt to authenticate using SASL (unknown error) 2022-09-12 11:26:50.105 INFO org.apache.kyuubi.server.KyuubiServer: Service[KyuubiServer] is initialized. 2022-09-12 11:26:50.106 INFO org.apache.kyuubi.server.KinitAuxiliaryService: Service[KinitAuxiliaryService] is started. 2022-09-12 11:26:50.107 INFO org.apache.kyuubi.metrics.JsonReporterService: Service[JsonReporterService] is started. 2022-09-12 11:26:50.108 INFO org.apache.kyuubi.metrics.MetricsSystem: Service[MetricsSystem] is started. 2022-09-12 11:26:50.111 INFO org.apache.kyuubi.engine.KyuubiApplicationManager: Service[KyuubiApplicationManager] is started. 2022-09-12 11:26:50.111 INFO org.apache.kyuubi.credentials.HadoopCredentialsManager: Service[HadoopCredentialsManager] is started. 2022-09-12 11:26:50.112 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to con02.xuncetech.com/192.168.0.106:2181, initiating session 2022-09-12 11:26:50.112 INFO org.apache.kyuubi.operation.KyuubiOperationManager: Service[KyuubiOperationManager] is started. 2022-09-12 11:26:50.112 INFO org.apache.kyuubi.session.KyuubiSessionManager: Service[KyuubiSessionManager] is started. 2022-09-12 11:26:50.112 INFO org.apache.kyuubi.server.KyuubiServer: Service[KyuubiBackendService] is started. 2022-09-12 11:26:50.113 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Starting and exposing JDBC connection at: jdbc:hive2://0.0.0.0:10009/ 2022-09-12 11:26:50.120 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server con02.xuncetech.com/192.168.0.106:2181, sessionid = 0xff832fa31ff30026, negotiated timeout = 60000 2022-09-12 11:26:50.127 INFO org.apache.curator.framework.state.ConnectionStateManager: State change: CONNECTED 2022-09-12 11:26:50.129 INFO org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient: Zookeeper client connection state changed to: CONNECTED 2022-09-12 11:26:50.155 INFO org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient: Created a /kyuubi/serviceUri=0.0.0.0:10009;version=1.6.0-incubating;sequence=0000000010 on ZooKeeper for KyuubiServer uri: 0.0.0.0:10009 2022-09-12 11:26:50.156 INFO org.apache.kyuubi.ha.client.KyuubiServiceDiscovery: Service[KyuubiServiceDiscovery] is started. 2022-09-12 11:26:50.156 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Service[KyuubiTBinaryFrontend] is started. 2022-09-12 11:26:50.156 INFO org.apache.kyuubi.server.KyuubiServer: Service[KyuubiServer] is started. 2022-09-12 11:27:18.854 ERROR org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Error getting tables: org.apache.kyuubi.KyuubiSQLException: Invalid SessionHandle [5f6f1df5-dd2d-4359-86b0-7579345a056e] at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) ~[kyuubi-common_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.session.SessionManager.$anonfun$getSession$1(SessionManager.scala:150) ~[kyuubi-common_2.12-1.6.0-incubating.jar:1.6.0-incubating] at scala.Option.getOrElse(Option.scala:189) ~[scala-library-2.12.15.jar:?] at org.apache.kyuubi.session.SessionManager.getSession(SessionManager.scala:150) ~[kyuubi-common_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.service.AbstractBackendService.getTables(AbstractBackendService.scala:93) ~[kyuubi-common_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.server.KyuubiServer$$anon$1.org$apache$kyuubi$server$BackendServiceMetric$$super$getTables(KyuubiServer.scala:111) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.server.BackendServiceMetric.$anonfun$getTables$1(BackendServiceMetric.scala:94) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.metrics.MetricsSystem$.timerTracing(MetricsSystem.scala:111) ~[kyuubi-metrics_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.server.BackendServiceMetric.getTables(BackendServiceMetric.scala:94) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.server.BackendServiceMetric.getTables$(BackendServiceMetric.scala:87) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.server.KyuubiServer$$anon$1.getTables(KyuubiServer.scala:111) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.service.TFrontendService.GetTables(TFrontendService.scala:316) ~[kyuubi-common_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetTables.getResult(TCLIService.java:1637) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetTables.getResult(TCLIService.java:1622) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) ~[kyuubi-common_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[libthrift-0.9.3.jar:0.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212] 2022-09-12 11:27:18.886 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7 2022-09-12 11:27:18.895 INFO org.apache.kyuubi.session.KyuubiSessionManager: Opening session for [email protected] 2022-09-12 11:27:18.902 WARN org.apache.kyuubi.config.KyuubiConf: The Kyuubi config 'kyuubi.frontend.bind.port' has been deprecated in Kyuubi v1.4.0 and may be removed in the future. Use kyuubi.frontend.thrift.binary.bind.port instead 2022-09-12 11:27:18.902 WARN org.apache.kyuubi.config.KyuubiConf: The Kyuubi config 'kyuubi.ha.zookeeper.quorum' has been deprecated in Kyuubi v1.6.0 and may be removed in the future. Use kyuubi.ha.addresses instead 2022-09-12 11:27:18.902 WARN org.apache.kyuubi.config.KyuubiConf: The Kyuubi config 'kyuubi.ha.zookeeper.namespace' has been deprecated in Kyuubi v1.6.0 and may be removed in the future. Use kyuubi.ha.namespace instead 2022-09-12 11:27:18.904 WARN org.apache.kyuubi.config.KyuubiConf: The Kyuubi config 'kyuubi.frontend.bind.port' has been deprecated in Kyuubi v1.4.0 and may be removed in the future. Use kyuubi.frontend.thrift.binary.bind.port instead 2022-09-12 11:27:18.924 INFO org.apache.kyuubi.operation.log.OperationLog: Creating operation log file /opt/apache-kyuubi-1.6.0-incubating-bin/work/server_operation_logs/ee9185f2-5a31-4cc4-b1f5-1f542a88a7cd/fc78f617-c9cd-4a1c-932c-c4eb6d85d2ff 2022-09-12 11:27:18.927 INFO org.apache.kyuubi.session.KyuubiSessionManager: hive's session with SessionHandle [ee9185f2-5a31-4cc4-b1f5-1f542a88a7cd] is opened, current opening sessions 1 2022-09-12 11:27:18.927 INFO org.apache.kyuubi.operation.LaunchEngine: Processing hive's query[fc78f617-c9cd-4a1c-932c-c4eb6d85d2ff]: PENDING_STATE -> RUNNING_STATE, statement: LaunchEngine 2022-09-12 11:27:18.929 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: Starting 2022-09-12 11:27:18.930 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=con01.xuncetech.com,con02.xuncetech.com,con03.xuncetech.com sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@13408767 2022-09-12 11:27:18.932 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server con03.xuncetech.com/192.168.0.107:2181. Will not attempt to authenticate using SASL (unknown error) 2022-09-12 11:27:18.935 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to con03.xuncetech.com/192.168.0.107:2181, initiating session 2022-09-12 11:27:18.937 INFO org.apache.kyuubi.session.KyuubiSessionImpl: [hive:192.168.0.106] SessionHandle [ee9185f2-5a31-4cc4-b1f5-1f542a88a7cd] - Starting to wait the launch engine operation finished 2022-09-12 11:27:18.945 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server con03.xuncetech.com/192.168.0.107:2181, sessionid = 0xff832fa31fcf0024, negotiated timeout = 60000 2022-09-12 11:27:18.945 INFO org.apache.curator.framework.state.ConnectionStateManager: State change: CONNECTED 2022-09-12 11:27:18.988 INFO org.apache.kyuubi.engine.EngineRef: Launching engine: /opt/apache-kyuubi-1.6.0-incubating-bin/bin/bin/spark-submit \ --class org.apache.kyuubi.engine.spark.SparkSQLEngine \ --conf spark.kyuubi.ha.zookeeper.auth.keytab=/usr/local/kafka/zookeeper.keytab \ --conf spark.yarn.jars=hdfs://con01.xuncetech.com:8020/user/spark3_2_0/*.jar \ --conf spark.kyuubi.session.engine.idle.timeout=PT5M \ --conf spark.dynamicAllocation.sustainedSchedulerBacklogTimeout=5s \ --conf spark.executor.heartbeatInterval=30s \ --conf spark.driver.maxResultSize=4g \ --conf spark.kyuubi.ha.zookeeper.quorum=con01.xuncetech.com,con02.xuncetech.com,con03.xuncetech.com \ --conf spark.shuffle.service.enabled=true \ --conf spark.hadoop.fs.hdfs.impl.disable.cache=true \ --conf spark.kyuubi.engine.submit.time=1662953238967 \ --conf spark.kyuubi.ha.zookeeper.client.port=2181 \ --conf spark.app.name=kyuubi_USER_SPARK_SQL_hive_default_ee9185f2-5a31-4cc4-b1f5-1f542a88a7cd \ --conf spark.dynamicAllocation.schedulerBacklogTimeout=1s \ --conf spark.driver.memory=4g \ --conf spark.kyuubi.ha.zookeeper.auth.principal=zookeeper/[email protected] \ --conf spark.kyuubi.ha.engine.ref.id=ee9185f2-5a31-4cc4-b1f5-1f542a88a7cd \ --conf spark.dynamicAllocation.maxExecutors=20 \ --conf spark.files=/usr/local/kafka/zookeeper.keytab \ --conf spark.kyuubi.ha.zookeeper.auth.type=KERBEROS \ --conf spark.submit.deployMode=client \ --conf spark.hive.server2.proxy.user=hive \ --conf spark.master=yarn \ --conf spark.shuffle.useOldFetchProtocol=true \ --conf spark.yarn.tags=KYUUBI,ee9185f2-5a31-4cc4-b1f5-1f542a88a7cd \ --conf spark.kyuubi.engine.share.level=USER \ --conf spark.kyuubi.ha.zookeeper.namespace=kyuubi \ --conf spark.dynamicAllocation.enabled=true \ --conf spark.kyuubi.client.ipAddress=192.168.0.106 \ --conf spark.kyuubi.ha.enabled=true \ --conf spark.dynamicAllocation.minExecutors=1 \ --conf spark.dynamicAllocation.initialExecutors=1 \ --conf spark.dynamicAllocation.executorIdleTimeout=60s \ --conf spark.kyuubi.ha.namespace=/kyuubi_1.6.0-incubating_USER_SPARK_SQL/hive/default \ --proxy-user hive /opt/apache-kyuubi-1.6.0-incubating-bin/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.6.0-incubating.jar 2022-09-12 11:27:18.996 INFO org.apache.kyuubi.engine.ProcBuilder: Logging to /opt/apache-kyuubi-1.6.0-incubating-bin/work/hive/kyuubi-spark-sql-engine.log.0 2022-09-12 11:27:19.002 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting 2022-09-12 11:27:19.005 INFO org.apache.zookeeper.ZooKeeper: Session: 0xff832fa31fcf0024 closed 2022-09-12 11:27:19.005 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0xff832fa31fcf0024 2022-09-12 11:27:19.012 INFO org.apache.kyuubi.operation.LaunchEngine: Processing hive's query[fc78f617-c9cd-4a1c-932c-c4eb6d85d2ff]: RUNNING_STATE -> ERROR_STATE, time taken: 0.084 seconds 2022-09-12 11:27:19.013 ERROR org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Error getting tables: java.util.concurrent.ExecutionException: org.apache.kyuubi.KyuubiSQLException: Error operating LaunchEngine: java.io.IOException: Cannot run program "/opt/apache-kyuubi-1.6.0-incubating-bin/bin/bin/spark-submit" (in directory "/opt/apache-kyuubi-1.6.0-incubating-bin/work/hive"): error=2, 没有那个文件或目录 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) at org.apache.kyuubi.engine.ProcBuilder.start(ProcBuilder.scala:198) at org.apache.kyuubi.engine.ProcBuilder.start$(ProcBuilder.scala:197) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.start(SparkProcessBuilder.scala:36) at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:194) at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.tryWithLock(ZookeeperDiscoveryClient.scala:173) at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:160) at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:165) at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:238) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:238) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:119) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:112) at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36) at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:112) at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:49) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: error=2, 没有那个文件或目录 at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:247) at java.lang.ProcessImpl.start(ProcessImpl.java:134) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ... 20 more at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_212] at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_212] at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$waitForEngineLaunched$1(KyuubiSessionImpl.scala:162) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$waitForEngineLaunched$1$adapted(KyuubiSessionImpl.scala:158) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at scala.Option.foreach(Option.scala:407) ~[scala-library-2.12.15.jar:?] at org.apache.kyuubi.session.KyuubiSessionImpl.waitForEngineLaunched(KyuubiSessionImpl.scala:158) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.session.KyuubiSessionImpl.runOperation(KyuubiSessionImpl.scala:148) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.session.AbstractSession.getTables(AbstractSession.scala:160) ~[kyuubi-common_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.service.AbstractBackendService.getTables(AbstractBackendService.scala:94) ~[kyuubi-common_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.server.KyuubiServer$$anon$1.org$apache$kyuubi$server$BackendServiceMetric$$super$getTables(KyuubiServer.scala:111) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.server.BackendServiceMetric.$anonfun$getTables$1(BackendServiceMetric.scala:94) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.metrics.MetricsSystem$.timerTracing(MetricsSystem.scala:111) ~[kyuubi-metrics_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.server.BackendServiceMetric.getTables(BackendServiceMetric.scala:94) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.server.BackendServiceMetric.getTables$(BackendServiceMetric.scala:87) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.server.KyuubiServer$$anon$1.getTables(KyuubiServer.scala:111) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.service.TFrontendService.GetTables(TFrontendService.scala:316) ~[kyuubi-common_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetTables.getResult(TCLIService.java:1637) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetTables.getResult(TCLIService.java:1622) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) ~[kyuubi-common_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[libthrift-0.9.3.jar:0.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212] Caused by: org.apache.kyuubi.KyuubiSQLException: Error operating LaunchEngine: java.io.IOException: Cannot run program "/opt/apache-kyuubi-1.6.0-incubating-bin/bin/bin/spark-submit" (in directory "/opt/apache-kyuubi-1.6.0-incubating-bin/work/hive"): error=2, 没有那个文件或目录 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) at org.apache.kyuubi.engine.ProcBuilder.start(ProcBuilder.scala:198) at org.apache.kyuubi.engine.ProcBuilder.start$(ProcBuilder.scala:197) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.start(SparkProcessBuilder.scala:36) at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:194) at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.tryWithLock(ZookeeperDiscoveryClient.scala:173) at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:160) at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:165) at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:238) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:238) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:119) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:112) at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36) at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:112) at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:49) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: error=2, 没有那个文件或目录 at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:247) at java.lang.ProcessImpl.start(ProcessImpl.java:134) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ... 20 more at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) ~[kyuubi-common_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.operation.KyuubiOperation$$anonfun$onError$1.applyOrElse(KyuubiOperation.scala:75) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.operation.KyuubiOperation$$anonfun$onError$1.applyOrElse(KyuubiOperation.scala:56) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38) ~[scala-library-2.12.15.jar:?] at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:51) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_212] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212] ... 3 more Caused by: java.io.IOException: Cannot run program "/opt/apache-kyuubi-1.6.0-incubating-bin/bin/bin/spark-submit" (in directory "/opt/apache-kyuubi-1.6.0-incubating-bin/work/hive"): error=2, 没有那个文件或目录 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[?:1.8.0_212] at org.apache.kyuubi.engine.ProcBuilder.start(ProcBuilder.scala:198) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.engine.ProcBuilder.start$(ProcBuilder.scala:197) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.engine.spark.SparkProcessBuilder.start(SparkProcessBuilder.scala:36) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:194) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.tryWithLock(ZookeeperDiscoveryClient.scala:173) ~[kyuubi-ha_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:160) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:165) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:238) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at scala.Option.getOrElse(Option.scala:189) ~[scala-library-2.12.15.jar:?] at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:238) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:119) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:112) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36) ~[kyuubi-ha_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:112) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:49) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_212] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212] ... 3 more Caused by: java.io.IOException: error=2, 没有那个文件或目录 at java.lang.UNIXProcess.forkAndExec(Native Method) ~[?:1.8.0_212] at java.lang.UNIXProcess.<init>(UNIXProcess.java:247) ~[?:1.8.0_212] at java.lang.ProcessImpl.start(ProcessImpl.java:134) ~[?:1.8.0_212] at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[?:1.8.0_212] at org.apache.kyuubi.engine.ProcBuilder.start(ProcBuilder.scala:198) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.engine.ProcBuilder.start$(ProcBuilder.scala:197) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.engine.spark.SparkProcessBuilder.start(SparkProcessBuilder.scala:36) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:194) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.tryWithLock(ZookeeperDiscoveryClient.scala:173) ~[kyuubi-ha_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:160) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:165) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:238) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at scala.Option.getOrElse(Option.scala:189) ~[scala-library-2.12.15.jar:?] at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:238) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:119) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:112) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36) ~[kyuubi-ha_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:112) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:49) ~[kyuubi-server_2.12-1.6.0-incubating.jar:1.6.0-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_212] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212] ... 3 more ``` ### Kyuubi Engine Log Output _No response_ ### Kyuubi Server Configurations ```yaml # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ## Kyuubi Configurations # # kyuubi.authentication NONE # kyuubi.frontend.bind.host localhost # kyuubi.frontend.bind.port 10009 # # Details in https://kyuubi.apache.org/docs/latest/deployment/settings.html kyuubi.frontend.bind.port=10009 kyuubi.frontend.bind.host=0.0.0.0 kyuubi.engine.share.level=USER kyuubi.session.engine.idle.timeout=PT5M # 启动的spark引擎以yarn-cluster模式跑 spark.master=yarn spark.submit.deployMode=client spark.driver.memory=4g spark.hadoop.fs.hdfs.impl.disable.cache=true spark.executor.heartbeatInterval=30s spark.yarn.jars=hdfs://con01.xuncetech.com:8020/user/spark3_2_0/*.jar spark.shuffle.useOldFetchProtocol=true spark.shuffle.service.enabled=true spark.dynamicAllocation.enabled=true spark.dynamicAllocation.minExecutors=1 spark.dynamicAllocation.maxExecutors=20 spark.dynamicAllocation.initialExecutors=1 spark.dynamicAllocation.schedulerBacklogTimeout=1s spark.dynamicAllocation.executorIdleTimeout=60s spark.dynamicAllocation.sustainedSchedulerBacklogTimeout=5s spark.driver.maxResultSize=4g kyuubi.ha.enabled=true kyuubi.ha.zookeeper.auth.type=KERBEROS kyuubi.ha.zookeeper.auth.keytab=/usr/local/kafka/zookeeper.keytab kyuubi.ha.zookeeper.auth.principal=zookeeper/[email protected] kyuubi.ha.zookeeper.namespace=kyuubi kyuubi.ha.zookeeper.quorum=con01.xuncetech.com,con02.xuncetech.com,con03.xuncetech.com kyuubi.ha.zookeeper.client.port=2181 #kyuubi.authentication=KERBEROS kyuubi.authentication=NONE kyuubi.kinit.keytab=/usr/local/kafka/xunce.keytab kyuubi.kinit.principal=hive/[email protected] ``` ### Kyuubi Engine Configurations _No response_ ### Additional context _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
