qiuliang988 opened a new issue #1034:
URL: https://github.com/apache/incubator-kyuubi/issues/1034
### 1. Describe the bug
The lock order for close operationLog is OperationLog -> RootLogger
```
"FrontendServiceHandler-Pool: Thread-588":
at org.apache.log4j.Category.callAppenders(Category.java:204)
- waiting to lock <0x000000015fc4b7e8> (a
org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:576)
at org.apache.kyuubi.Logging.error(Logging.scala:78)
at org.apache.kyuubi.Logging.error$(Logging.scala:76)
at
org.apache.kyuubi.operation.log.OperationLog.error(OperationLog.scala:86)
at
org.apache.kyuubi.operation.log.OperationLog.close(OperationLog.scala:137)
- locked <0x00000002c3818180> (a
org.apache.kyuubi.operation.log.OperationLog)
at
org.apache.kyuubi.engine.spark.operation.SparkOperation.$anonfun$close$1(SparkOperation.scala:142)
at
org.apache.kyuubi.engine.spark.operation.SparkOperation.$anonfun$close$1$adapted(SparkOperation.scala:142)
at
org.apache.kyuubi.engine.spark.operation.SparkOperation$$Lambda$2491/880482099.apply(Unknown
Source)
at scala.Option.foreach(Option.scala:407)
at
org.apache.kyuubi.engine.spark.operation.SparkOperation.close(SparkOperation.scala:142)
at
org.apache.kyuubi.operation.OperationManager.closeOperation(OperationManager.scala:103)
at
org.apache.kyuubi.session.AbstractSession.$anonfun$closeOperation$1(AbstractSession.scala:197)
at
org.apache.kyuubi.session.AbstractSession$$Lambda$2490/481951832.apply$mcZ$sp(Unknown
Source)
at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
at
org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:78)
at
org.apache.kyuubi.session.AbstractSession.closeOperation(AbstractSession.scala:196)
at
org.apache.kyuubi.service.AbstractBackendService.closeOperation(AbstractBackendService.scala:154)
at
org.apache.kyuubi.service.FrontendService.CloseOperation(FrontendService.scala:447)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1797)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1782)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
and the lock order for log4j is RootLogger -> LogDivertAppender ->
OperationLog
```
"SparkSQLSessionManager-exec-pool: Thread-80018":
at
org.apache.kyuubi.operation.log.OperationLog.write(OperationLog.scala:96)
- waiting to lock <0x00000002c3818180> (a
org.apache.kyuubi.operation.log.OperationLog)
at
org.apache.kyuubi.operation.log.LogDivertAppender.subAppend(LogDivertAppender.scala:55)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
- locked <0x0000000161df65b0> (a
org.apache.kyuubi.operation.log.LogDivertAppender)
at
org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
- locked <0x000000015fc4b7e8> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:305)
at org.apache.spark.internal.Logging.logInfo(Logging.scala:57)
at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56)
at
org.apache.spark.scheduler.DAGScheduler.logInfo(DAGScheduler.scala:114)
at
org.apache.spark.scheduler.DAGScheduler.cancelJobGroup(DAGScheduler.scala:988)
at org.apache.spark.SparkContext.cancelJobGroup(SparkContext.scala:2519)
at
org.apache.kyuubi.engine.spark.operation.SparkOperation$$anonfun$onError$1.applyOrElse(SparkOperation.scala:96)
at
org.apache.kyuubi.engine.spark.operation.SparkOperation$$anonfun$onError$1.applyOrElse(SparkOperation.scala:92)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
at
org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:137)
at
org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$Lambda$2447/570798066.apply$mcV$sp(Unknown
Source)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at
org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:219)
at
org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:86)
at
org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$2$$anon$3.run(ExecuteStatement.scala:172)
at
org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$2$$anon$3.run(ExecuteStatement.scala:169)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at
org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$2.run(ExecuteStatement.scala:169)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
The engine may be deadlocked.
### 2. Environments
#### 2.1 Versions
- Kyuubi
- [ ] 1.3.0-SNAPSHOT
- Spark
- [ ] 3.1.1
#### 2.2 Kyuubi Configurations
#### 2.3 Spark Configurations
### 3. Steps to reproduce
#### 3.1 Operations
#### 3.2 Expected behavior
<!-- A clear and concise description of what you expected to happen. -->
#### 3.3 Actual behavior
### 4. Additional context
#### 4.1 Log of the server
#### 4.2 Log of the corresponding engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
