zhanghe-git opened a new issue, #6594: URL: https://github.com/apache/kyuubi/issues/6594
### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the bug I encountered an issue when querying an ICEBERG table using a self-built Kyuubi (version 1.9.1) with Spark. The query involves retrieving a single row of data that is 100MB in size. While the query executes successfully on the Spark side, the Kyuubi side reports a MaxMessageSize reached error. I attempted to resolve this by adjusting the kyuubi.frontend.thrift.max.message.size parameter, but the error persists. It seems that some max size limit is being reached in Kyuubi. How can this be adjusted? Additional information: When using Alibaba Cloud's EMR with its built-in Kyuubi (version 1.7.1), the query returns successfully. (My kyuubi-defaults configuration is consistent with Alibaba Cloud's setup.)  ### Affects Version(s) 1.9.1 ### Kyuubi Server Log Output ```logtalk 2024-08-07 16:29:53.048 INFO KyuubiSessionManager-exec-pool: Thread-178 org.apache.kyuubi.operation.ExecuteStatement: Processing he.zhang's query[05f5c252-9420-43b2-81d1-f9b7596f343b]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.113 seconds 2024-08-07 16:29:53.059 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.operation.ExecuteStatement: Processing he.zhang's query[05f5c252-9420-43b2-81d1-f9b7596f343b]: FINISHED_STATE -> CLOSED_STATE, time taken: 0.124 seconds 2024-08-07 16:29:53.080 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:05 F5 C2 52 94 20 43 B2 81 D1 F9 B7 59 6F 34 3B, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side 2024-08-07 16:29:53.090 INFO KyuubiSessionManager-exec-pool: Thread-179 org.apache.kyuubi.operation.ExecuteStatement: Processing he.zhang's query[4619304f-68e8-41d6-94e6-64b8d70913c5]: PENDING_STATE -> RUNNING_STATE, statement: select * from db8822.zlc_test_person_003 where id='ee4c83fa-4e8a-4a12-b3ce-b055a2ab595b' 2024-08-07 16:29:58.094 INFO KyuubiSessionManager-exec-pool: Thread-179 org.apache.kyuubi.operation.ExecuteStatement: Query[4619304f-68e8-41d6-94e6-64b8d70913c5] in RUNNING_STATE 2024-08-07 16:30:02.205 INFO KyuubiRestFrontendService-182 org.apache.kyuubi.server.http.authentication.AuthenticationAuditLogger: user=anonymous(auth:BASIC) ip=10.69.11.32 proxyIp=null method=GET uri=/api/v1/sessions/count params=null protocol=HTTP/1.1 status=200 2024-08-07 16:30:03.097 INFO KyuubiSessionManager-exec-pool: Thread-179 org.apache.kyuubi.operation.ExecuteStatement: Query[4619304f-68e8-41d6-94e6-64b8d70913c5] in RUNNING_STATE 2024-08-07 16:30:03.692 INFO KyuubiSessionManager-exec-pool: Thread-179 org.apache.kyuubi.operation.ExecuteStatement: Query[4619304f-68e8-41d6-94e6-64b8d70913c5] in FINISHED_STATE 2024-08-07 16:30:03.692 INFO KyuubiSessionManager-exec-pool: Thread-179 org.apache.kyuubi.operation.ExecuteStatement: Processing he.zhang's query[4619304f-68e8-41d6-94e6-64b8d70913c5]: RUNNING_STATE -> FINISHED_STATE, time taken: 10.602 seconds 2024-08-07 16:30:04.141 ERROR KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Error fetching results: org.apache.kyuubi.shaded.thrift.transport.TTransportException: MaxMessageSize reached at org.apache.kyuubi.shaded.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96) at org.apache.kyuubi.shaded.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97) at org.apache.kyuubi.shaded.thrift.transport.TSaslTransport.read(TSaslTransport.java:390) at org.apache.kyuubi.shaded.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39) at org.apache.kyuubi.shaded.thrift.transport.TTransport.readAll(TTransport.java:109) at org.apache.kyuubi.shaded.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:434) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TBinaryColumn$TBinaryColumnStandardScheme.read(TBinaryColumn.java:431) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TBinaryColumn$TBinaryColumnStandardScheme.read(TBinaryColumn.java:411) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TBinaryColumn.read(TBinaryColumn.java:345) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TColumn.standardSchemeReadValue(TColumn.java:317) at org.apache.kyuubi.shaded.thrift.TUnion$TUnionStandardScheme.read(TUnion.java:228) at org.apache.kyuubi.shaded.thrift.TUnion$TUnionStandardScheme.read(TUnion.java:217) at org.apache.kyuubi.shaded.thrift.TUnion.read(TUnion.java:138) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TRowSet$TRowSetStandardScheme.read(TRowSet.java:718) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TRowSet$TRowSetStandardScheme.read(TRowSet.java:670) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TRowSet.read(TRowSet.java:580) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TFetchResultsResp$TFetchResultsRespStandardScheme.read(TFetchResultsResp.java:495) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TFetchResultsResp$TFetchResultsRespStandardScheme.read(TFetchResultsResp.java:463) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TFetchResultsResp.read(TFetchResultsResp.java:385) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$FetchResults_result$FetchResults_resultStandardScheme.read(TCLIService.java:16912) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$FetchResults_result$FetchResults_resultStandardScheme.read(TCLIService.java:16897) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$FetchResults_result.read(TCLIService.java:16844) at org.apache.kyuubi.shaded.thrift.TServiceClient.receiveBase(TServiceClient.java:88) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.recv_FetchResults(TCLIService.java:548) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.FetchResults(TCLIService.java:535) at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$fetchResults$1(KyuubiSyncThriftClient.scala:437) at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$withLockAcquiredAsyncRequest$2(KyuubiSyncThriftClient.scala:167) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-08-07 16:30:04.146 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.operation.ExecuteStatement: Processing he.zhang's query[4619304f-68e8-41d6-94e6-64b8d70913c5]: FINISHED_STATE -> CLOSED_STATE, time taken: 11.056 seconds 2024-08-07 16:30:04.147 WARN KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.operation.ExecuteStatement: Error closing THandleIdentifier(guid:46 19 30 4F 68 E8 41 D6 94 E6 64 B8 D7 09 13 C5, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38): MaxMessageSize reached org.apache.kyuubi.shaded.thrift.transport.TTransportException: MaxMessageSize reached at org.apache.kyuubi.shaded.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96) at org.apache.kyuubi.shaded.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97) at org.apache.kyuubi.shaded.thrift.transport.TSaslTransport.read(TSaslTransport.java:390) at org.apache.kyuubi.shaded.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39) at org.apache.kyuubi.shaded.thrift.transport.TTransport.readAll(TTransport.java:109) at org.apache.kyuubi.shaded.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464) at org.apache.kyuubi.shaded.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:362) at org.apache.kyuubi.shaded.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:245) at org.apache.kyuubi.shaded.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.recv_CloseOperation(TCLIService.java:502) at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:489) at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$closeOperation$1(KyuubiSyncThriftClient.scala:413) at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$withLockAcquiredAsyncRequest$2(KyuubiSyncThriftClient.scala:167) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-08-07 16:30:04.469 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Session [SessionHandle [b921325e-0066-4cde-8c9e-8ca43275c2d0]] disconnected without closing properly, close it now 2024-08-07 16:30:04.470 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.session.KyuubiSessionManager: he.zhang's KyuubiSessionImpl with SessionHandle [b921325e-0066-4cde-8c9e-8ca43275c2d0] is closed, current opening sessions 0 2024-08-07 16:30:04.474 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.operation.LaunchEngine: Processing he.zhang's query[9f58a672-629e-4b98-8c3a-f1fb756fd1ca]: FINISHED_STATE -> CLOSED_STATE, time taken: 43.801 seconds 2024-08-07 16:30:04.516 ERROR KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Failed closing session org.apache.kyuubi.KyuubiSQLException: Error while cleaning up the engine resources at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.client.KyuubiSyncThriftClient.closeSession(KyuubiSyncThriftClient.scala:237) at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:281) at org.apache.kyuubi.session.SessionManager.closeSession(SessionManager.scala:134) at org.apache.kyuubi.session.KyuubiSessionManager.closeSession(KyuubiSessionManager.scala:129) ``` ### Kyuubi Engine Log Output _No response_ ### Kyuubi Server Configurations _No response_ ### Kyuubi Engine Configurations _No response_ ### Additional context _No response_ ### Are you willing to submit PR? - [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix. - [X] No. I cannot submit a PR at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@kyuubi.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: notifications-unsubscr...@kyuubi.apache.org For additional commands, e-mail: notifications-h...@kyuubi.apache.org
