531651225 opened a new issue, #4735: URL: https://github.com/apache/kyuubi/issues/4735
### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the bug We are running sql on spark sql engine connected to Kyuubi server. failing with KyuubiSQLException: Error while cleaning up the engine resources This issue is reproducible every time, Running multiple different SQL statements both can cause this issue ### Affects Version(s) 1.7.1 ### Kyuubi Server Log Output ```logtalk 2023-04-19 14:46:09.333 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing airflow's query[79890581-cc15-4f94-959a-dcc04253b6af]: PENDING_STATE -> RUNNING_STATE, statement: with flow_raw as ( select t1.* ,coalesce(t2.parent_page_guid,t1.page_guid_raw) page_guid from (select dt ,device_id ,uid ,app_code ,page_guid page_guid_raw ,page_name ,event_time ,index_in_launch ,dimession2 ,dimession3 .... 2023-04-19 14:46:14.347 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:46:19.366 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:46:24.368 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:46:29.370 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:46:34.372 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:46:39.373 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:46:44.375 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:46:49.384 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:46:54.386 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:46:59.388 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:47:04.389 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:47:09.390 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:47:14.392 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:47:19.393 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:47:24.394 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:47:29.396 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:47:34.397 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:47:39.398 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:47:44.400 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:47:49.500 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:50:02.512 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:50:06.437 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:50:07.515 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:50:12.517 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:50:12.816 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:50:17.521 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:50:18.060 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:50:22.523 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:50:23.734 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:50:27.525 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:50:28.756 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:50:32.526 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:50:35.295 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:50:37.527 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:50:41.616 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:50:42.529 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:50:46.783 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:50:47.531 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:50:52.532 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:50:52.800 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:50:57.534 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:50:58.045 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:51:02.535 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:51:04.771 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:51:07.537 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:51:12.538 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:51:12.547 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:51:17.540 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:51:19.802 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:51:22.542 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in RUNNING_STATE 2023-04-19 14:51:26.259 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43] in FINISHED_STATE 2023-04-19 14:51:26.259 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing mfw_hadoop's query[c4f5f8f6-fe97-40aa-929a-2c0df9c18b43]: RUNNING_STATE -> FINISHED_STATE, time taken: 108.769 seconds 2023-04-19 14:51:26.333 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:51:26.374 INFO org.apache.kyuubi.client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:27 21 A9 92 E5 81 42 B2 95 9D 3D B0 3A F3 7C 3C, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side 2023-04-19 14:51:26.392 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [703e7147-50cb-4505-bfea-23cce353100b] 2023-04-19 14:51:26.392 INFO org.apache.kyuubi.session.KyuubiSessionManager: mfw_hadoop's session with SessionHandle [703e7147-50cb-4505-bfea-23cce353100b] is closed, current opening sessions 2 2023-04-19 14:51:26.449 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [703e7147-50cb-4505-bfea-23cce353100b] 2023-04-19 14:51:33.081 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:52:00.919 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:52:56.701 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:54:50.247 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 14:59:21.497 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 15:01:53.693 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[79890581-cc15-4f94-959a-dcc04253b6af] in RUNNING_STATE 2023-04-19 15:03:05.092 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing airflow's query[79890581-cc15-4f94-959a-dcc04253b6af]: RUNNING_STATE -> ERROR_STATE, time taken: 1015.758 seconds 2023-04-19 15:03:05.613 WARN org.apache.kyuubi.operation.ExecuteStatement: Error closing THandleIdentifier(guid:E2 D7 DE 4E D9 84 4A 28 B6 E8 9E 95 63 7C 66 90, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38): null org.apache.thrift.transport.TTransportException: null at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseOperation(TCLIService.java:521) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:508) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$closeOperation$1(KyuubiSyncThriftClient.scala:386) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$withLockAcquiredAsyncRequest$2(KyuubiSyncThriftClient.scala:142) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_71] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_71] 2023-04-19 15:03:05.677 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [1115d786-85d8-4099-816b-c180dabb4454] 2023-04-19 15:03:05.677 INFO org.apache.kyuubi.session.KyuubiSessionManager: airflow's session with SessionHandle [1115d786-85d8-4099-816b-c180dabb4454] is closed, current opening sessions 1 2023-04-19 15:03:05.678 WARN org.apache.thrift.transport.TIOStreamTransport: Error closing output stream. java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:116) ~[?:1.8.0_71] at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[?:1.8.0_71] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_71] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_71] at java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[?:1.8.0_71] at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.transport.TSocket.close(TSocket.java:235) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.transport.TSaslTransport.close(TSaslTransport.java:402) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.transport.TSaslClientTransport.close(TSaslClientTransport.java:37) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$closeSession$5(KyuubiSyncThriftClient.scala:225) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$closeSession$5$adapted(KyuubiSyncThriftClient.scala:224) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at scala.collection.immutable.List.foreach(List.scala:431) ~[scala-library-2.12.17.jar:?] at org.apache.kyuubi.client.KyuubiSyncThriftClient.closeSession(KyuubiSyncThriftClient.scala:224) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:252) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.session.SessionManager.closeSession(SessionManager.scala:131) ~[kyuubi-common_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.session.KyuubiSessionManager.closeSession(KyuubiSessionManager.scala:121) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.service.AbstractBackendService.closeSession(AbstractBackendService.scala:49) ~[kyuubi-common_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.server.KyuubiServer$$anon$1.org$apache$kyuubi$server$BackendServiceMetric$$super$closeSession(KyuubiServer.scala:147) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.server.BackendServiceMetric.$anonfun$closeSession$1(BackendServiceMetric.scala:43) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.17.jar:?] at org.apache.kyuubi.metrics.MetricsSystem$.timerTracing(MetricsSystem.scala:111) ~[kyuubi-metrics_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.server.BackendServiceMetric.closeSession(BackendServiceMetric.scala:43) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.server.BackendServiceMetric.closeSession$(BackendServiceMetric.scala:41) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.server.KyuubiServer$$anon$1.closeSession(KyuubiServer.scala:147) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.service.TFrontendService.CloseSession(TFrontendService.scala:209) ~[kyuubi-common_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1517) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1502) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) ~[kyuubi-common_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[libthrift-0.9.3.jar:0.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_71] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_71] 2023-04-19 15:03:05.679 ERROR org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Error closing session: org.apache.kyuubi.KyuubiSQLException: Error while cleaning up the engine resources at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) ~[kyuubi-common_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.client.KyuubiSyncThriftClient.closeSession(KyuubiSyncThriftClient.scala:210) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:252) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.session.SessionManager.closeSession(SessionManager.scala:131) ~[kyuubi-common_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.session.KyuubiSessionManager.closeSession(KyuubiSessionManager.scala:121) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.service.AbstractBackendService.closeSession(AbstractBackendService.scala:49) ~[kyuubi-common_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.server.KyuubiServer$$anon$1.org$apache$kyuubi$server$BackendServiceMetric$$super$closeSession(KyuubiServer.scala:147) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.server.BackendServiceMetric.$anonfun$closeSession$1(BackendServiceMetric.scala:43) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.17.jar:?] at org.apache.kyuubi.metrics.MetricsSystem$.timerTracing(MetricsSystem.scala:111) ~[kyuubi-metrics_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.server.BackendServiceMetric.closeSession(BackendServiceMetric.scala:43) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.server.BackendServiceMetric.closeSession$(BackendServiceMetric.scala:41) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.server.KyuubiServer$$anon$1.closeSession(KyuubiServer.scala:147) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.service.TFrontendService.CloseSession(TFrontendService.scala:209) ~[kyuubi-common_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1517) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1502) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) ~[kyuubi-common_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[libthrift-0.9.3.jar:0.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_71] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_71] Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.transport.TSaslTransport.flush(TSaslTransport.java:501) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.transport.TSaslClientTransport.flush(TSaslClientTransport.java:37) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.hive.service.rpc.thrift.TCLIService$Client.send_CloseSession(TCLIService.java:193) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseSession(TCLIService.java:185) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$closeSession$1(KyuubiSyncThriftClient.scala:205) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$withLockAcquiredAsyncRequest$2(KyuubiSyncThriftClient.scala:142) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_71] ... 3 more Caused by: java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_71] at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[?:1.8.0_71] at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[?:1.8.0_71] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_71] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_71] at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.transport.TSaslTransport.flush(TSaslTransport.java:501) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.transport.TSaslClientTransport.flush(TSaslClientTransport.java:37) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.hive.service.rpc.thrift.TCLIService$Client.send_CloseSession(TCLIService.java:193) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseSession(TCLIService.java:185) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$closeSession$1(KyuubiSyncThriftClient.scala:205) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$withLockAcquiredAsyncRequest$2(KyuubiSyncThriftClient.scala:142) ~[kyuubi-server_2.12-1.7.1-SNAPSHOT.jar:1.7.1-SNAPSHOT] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_71] ... 3 more ``` ### Kyuubi Engine Log Output ```logtalk 23/04/19 15:02:54 INFO Client: Application report for application_1679388567772_362090 (state: RUNNING) 23/04/19 15:02:55 INFO Client: Application report for application_1679388567772_362090 (state: RUNNING) 23/04/19 15:02:56 INFO Client: Application report for application_1679388567772_362090 (state: RUNNING) 23/04/19 15:02:57 INFO Client: Application report for application_1679388567772_362090 (state: RUNNING) 23/04/19 15:02:58 INFO Client: Application report for application_1679388567772_362090 (state: RUNNING) 23/04/19 15:02:59 INFO Client: Application report for application_1679388567772_362090 (state: RUNNING) 23/04/19 15:03:00 INFO Client: Application report for application_1679388567772_362090 (state: RUNNING) 23/04/19 15:03:01 INFO Client: Application report for application_1679388567772_362090 (state: RUNNING) 23/04/19 15:03:02 INFO Client: Application report for application_1679388567772_362090 (state: RUNNING) 23/04/19 15:03:03 INFO Client: Application report for application_1679388567772_362090 (state: RUNNING) 23/04/19 15:03:04 INFO Client: Application report for application_1679388567772_362090 (state: RUNNING) 23/04/19 15:03:05 INFO Client: Application report for application_1679388567772_362090 (state: ACCEPTED) 23/04/19 15:03:05 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.dm start time: 1681886737777 final status: UNDEFINED tracking URL: http://192.168.7.169:9021/proxy/application_1679388567772_362090/ user: airflow 23/04/19 15:03:06 INFO Client: Application report for application_1679388567772_362090 (state: ACCEPTED) 23/04/19 15:03:07 INFO Client: Application report for application_1679388567772_362090 (state: ACCEPTED) 23/04/19 15:03:08 INFO Client: Application report for application_1679388567772_362090 (state: ACCEPTED) 23/04/19 15:03:09 INFO Client: Application report for application_1679388567772_362090 (state: ACCEPTED) 23/04/19 15:03:10 INFO Client: Application report for application_1679388567772_362090 (state: ACCEPTED) 23/04/19 15:03:11 INFO Client: Application report for application_1679388567772_362090 (state: ACCEPTED) 23/04/19 15:03:12 INFO Client: Application report for application_1679388567772_362090 (state: ACCEPTED) 23/04/19 15:03:13 INFO Client: Application report for application_1679388567772_362090 (state: ACCEPTED) 23/04/19 15:03:14 INFO Client: Application report for application_1679388567772_362090 (state: ACCEPTED) 23/04/19 15:03:15 INFO Client: Application report for application_1679388567772_362090 (state: ACCEPTED) 23/04/19 15:03:16 INFO Client: Application report for application_1679388567772_362090 (state: ACCEPTED) 23/04/19 15:03:17 INFO Client: Application report for application_1679388567772_362090 (state: ACCEPTED) 23/04/19 15:03:18 INFO Client: Application report for application_1679388567772_362090 (state: ACCEPTED) 23/04/19 15:03:19 INFO Client: Application report for application_1679388567772_362090 (state: ACCEPTED) 23/04/19 15:03:20 INFO Client: Application report for application_1679388567772_362090 (state: FAILED) 23/04/19 15:03:20 INFO Client: client token: N/A diagnostics: Application application_1679388567772_362090 failed 4 times due to AM Container for appattempt_1679388567772_362090_000004 exited with exitCode: 13 For more detailed output, check application tracking page:http://192.168.7.169:9021/proxy/application_1679388567772_362090/Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_e25_1679388567772_362090_04_000001 Exit code: 13 Stack trace: ExitCodeException exitCode=13: at org.apache.hadoop.util.Shell.runCommand(Shell.java:604) at org.apache.hadoop.util.Shell.run(Shell.java:507) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 13 Failing this attempt. Failing the application. ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.dm start time: 1681886737777 final status: FAILED tracking URL: http://node007169:9041/cluster/app/application_1679388567772_362090 user: airflow 23/04/19 15:03:20 ERROR Client: Application diagnostics message: Application application_1679388567772_362090 failed 4 times due to AM Container for appattempt_1679388567772_362090_000004 exited with exitCode: 13 For more detailed output, check application tracking page:http://192.168.7.169:9021/proxy/application_1679388567772_362090/Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_e25_1679388567772_362090_04_000001 Exit code: 13 Stack trace: ExitCodeException exitCode=13: at org.apache.hadoop.util.Shell.runCommand(Shell.java:604) at org.apache.hadoop.util.Shell.run(Shell.java:507) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 13 Failing this attempt. Failing the application. 23/04/19 15:03:20 WARN UserGroupInformation: PriviledgedActionException as:airflow (auth:PROXY) via mfw_hadoop (auth:SIMPLE) cause:org.apache.spark.SparkException: Application application_1679388567772_362090 finished with failed status Exception in thread "main" java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1937) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: org.apache.spark.SparkException: Application application_1679388567772_362090 finished with failed status at org.apache.spark.deploy.yarn.Client.run(Client.scala:1265) at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1657) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951) at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165) at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) ... 6 more 23/04/19 15:03:20 INFO ShutdownHookManager: Shutdown hook called 23/04/19 15:03:20 INFO ShutdownHookManager: Deleting directory /tmp/spark-83205067-1460-4aad-9333-a26dc29b3cf8 23/04/19 15:03:20 INFO ShutdownHookManager: Deleting directory /tmp/spark-37aef1ec-4069-4af8-bc30-4d9eec5c78ae ``` ### Kyuubi Server Configurations ```yaml # spark引擎共享级别，user，即同一用户共享一个引擎， kyuubi.engine.share.level=CONNECTION #kyuubi.session.engine.idle.timeout=PT10H # # # 启用HA，指定ZK地址 kyuubi.ha.enabled true kyuubi.ha.zookeeper.quorum xxx:2181 kyuubi.ha.zookeeper.client.port 2181 kyuubi.ha.zookeeper.namespace kyuubi_prod_1 ``` ### Kyuubi Engine Configurations ```yaml spark.master=yarn spark.submit.deployMode=cluster ___airflow___.spark.dynamicAllocation.enabled=true ##false if prefer shuffle tracking than ESS ___airflow___.spark.shuffle.service.enabled=true ___airflow___.spark.dynamicAllocation.initialExecutors=10 ___airflow___.spark.dynamicAllocation.minExecutors=10 ___airflow___.spark.dynamicAllocation.maxExecutors=500 ___airflow___.spark.dynamicAllocation.executorAllocationRatio=0.5 ___airflow___.spark.dynamicAllocation.executorIdleTimeout=60s ___airflow___.spark.dynamicAllocation.cachedExecutorIdleTimeout=30min ### true if prefer shuffle tracking than ESS ___airflow___.spark.dynamicAllocation.shuffleTracking.enabled=false ___airflow___.spark.dynamicAllocation.shuffleTracking.timeout=30min ___airflow___.spark.dynamicAllocation.schedulerBacklogTimeout=1s ___airflow___.spark.dynamicAllocation.sustainedSchedulerBacklogTimeout=1s ___airflow___.spark.cleaner.periodicGC.interval=5min ___airflow___.spark.sql.adaptive.enabled=true ___airflow___.spark.sql.adaptive.forceApply=false ___airflow___.spark.sql.adaptive.logLevel=info ___airflow___.spark.sql.adaptive.advisoryPartitionSizeInBytes=256m ___airflow___.spark.sql.adaptive.coalescePartitions.enabled=true ___airflow___.spark.sql.adaptive.coalescePartitions.minPartitionNum=1 ___airflow___.spark.sql.adaptive.coalescePartitions.initialPartitionNum=8192 ___airflow___.spark.sql.adaptive.fetchShuffleBlocksInBatch=true ___airflow___.spark.sql.adaptive.localShuffleReader.enabled=true ___airflow___.spark.sql.adaptive.skewJoin.enabled=true ___airflow___.spark.sql.adaptive.skewJoin.skewedPartitionFactor=5 ___airflow___.spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes=400m ___airflow___.spark.sql.adaptive.nonEmptyPartitionRatioForBroadcastJoin=0.2 ___airflow___.spark.sql.adaptive.optimizer.excludedRules ___airflow___.spark.sql.autoBroadcastJoinThreshold=-1 ___airflow___.spark.sql.extensions=org.apache.kyuubi.sql.KyuubiSparkSQLExtension ___airflow___.spark.sql.optimizer.insertRepartitionBeforeWrite.enabled=true ``` ### Additional context _No response_ ### Are you willing to submit PR? - [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix. - [X] No. I cannot submit a PR at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
