pravin1406 commented on issue #4591: URL: https://github.com/apache/kyuubi/issues/4591#issuecomment-2115683310
@lordk911 @yaooqinn We are also getting the exact issue in our prod deployment. We also see this happening in 1 of the kyuubi server only out of 4. On checking the report.json metric file inside that server we get_operation_status having mean of 2600 millisecond. have attached metric report for this. [report-4.json](https://github.com/apache/kyuubi/files/15338212/report-4.json) basically this operation was taking loads of time and seems to get stuck a lot. This could be something related to high user load i guess. We have seen this a few times now. We investigate , but find only restarting as the solution mostly. I was thinking of decreasing the kyuubi.operation.idle.timeout to a very small value maybe few minute. Will that have any wrong effects on my queries? Some of our queries to run for hours. Below is the kyuubi server stacktrace which we see has high cpu time and elapsed time. I could not co relate this guid with a string operation handle id, as i did not know how to easily convert and seemed like to much effort. ` "async-request-executor-TSessionHandle(sessionId:THandleIdentifier(guid:66 CD F8 DE F2 CC 44 72 86 96 8F E7 97 DC EB 00, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38)): Thread-54226" #54226 daemon prio=5 os_prio=0 cpu=290468489.96ms elapsed=707467.31s tid=0x00007f3fe8018000 nid=0x7c807 runnable [0x00007f422b7fa000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0([email protected]/Native Method) at java.net.SocketInputStream.socketRead([email protected]/Unknown Source) at java.net.SocketInputStream.read([email protected]/Unknown Source) at java.net.SocketInputStream.read([email protected]/Unknown Source) at java.io.BufferedInputStream.fill([email protected]/Unknown Source) at java.io.BufferedInputStream.read1([email protected]/Unknown Source) at java.io.BufferedInputStream.read([email protected]/Unknown Source) - locked <0x0000000083be8ad0> (a java.io.BufferedInputStream) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376) at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453) at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435) at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_GetOperationStatus(TCLIService.java:475) at org.apache.hive.service.rpc.thrift.TCLIService$Client.GetOperationStatus(TCLIService.java:462) at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$getOperationStatus$1(KyuubiSyncThriftClient.scala:369) at org.apache.kyuubi.client.KyuubiSyncThriftClient$$Lambda$943/0x0000000840a79040.apply(Unknown Source) at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$withLockAcquiredAsyncRequest$2(KyuubiSyncThriftClient.scala:142) at org.apache.kyuubi.client.KyuubiSyncThriftClient$$Lambda$938/0x0000000840a6f040.call(Unknown Source) at java.util.concurrent.FutureTask.run([email protected]/Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run([email protected]/Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown Source) at java.lang.Thread.run([email protected]/Unknown Source)` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
