[
https://issues.apache.org/jira/browse/IOTDB-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584267#comment-17584267
]
Song Ziyang commented on IOTDB-4218:
------------------------------------
问题1 JSTACK:
2022-08-24 22:07:25
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.311-b11 mixed mode):
"Attach Listener" #207 daemon prio=9 os_prio=0 tid=0x00007f1e08002800
nid=0x1cc0a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"logback-8" #150 daemon prio=5 os_prio=0 tid=0x00007f1d50435800 nid=0x192bc
waiting on condition [0x00007f1e4a08d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c002a048> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"logback-7" #143 daemon prio=5 os_prio=0 tid=0x00007f1d5c216000 nid=0x1924d
waiting on condition [0x00007f1e4a893000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c002a048> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"logback-6" #138 daemon prio=5 os_prio=0 tid=0x00007f1dc0001800 nid=0x191e3
waiting on condition [0x00007f1e512d9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c002a048> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"logback-5" #136 daemon prio=5 os_prio=0 tid=0x00007f1de4006000 nid=0x191ad
waiting on condition [0x00007f1e4aa95000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c002a048> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"logback-4" #130 daemon prio=5 os_prio=0 tid=0x00007f1d5c215000 nid=0x19170
waiting on condition [0x00007f1e534f7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c002a048> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"pool-17-IoTDB-InternalServiceRPC-Client-3" #125 prio=5 os_prio=0
tid=0x00007f1d6c008000 nid=0x1914e runnable [0x00007f1e4b29d000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x000000041a65f868> (a java.io.BufferedInputStream)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:176)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at
org.apache.iotdb.rpc.TElasticFramedTransport.readFrame(TElasticFramedTransport.java:112)
at
org.apache.iotdb.rpc.TElasticFramedTransport.read(TElasticFramedTransport.java:107)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:463)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:361)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:244)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x0000000410f904f0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"pool-25-IoTDB-FlushTask-Submit-Pool-1" #121 prio=5 os_prio=0
tid=0x000055e13ecdd800 nid=0x1914a waiting on condition [0x00007f1e4b8f7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c87e34f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"logback-3" #93 daemon prio=5 os_prio=0 tid=0x00007f1de4003000 nid=0x19122
waiting on condition [0x00007f1e514db000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c002a048> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"pool-23-IoTDB-instance-management-1" #57 prio=5 os_prio=0
tid=0x00007f1d4401f000 nid=0x190f7 runnable [0x00007f1e53dfe000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c077d130> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"pool-22-IoTDB-Compaction_Schedule-root.test.g0_0-1-1" #49 prio=5 os_prio=0
tid=0x00007f1d680a3800 nid=0x190ef waiting on condition [0x00007f1e593d0000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c077a010> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"logback-2" #48 daemon prio=5 os_prio=0 tid=0x00007f1de4001800 nid=0x190e2
waiting on condition [0x00007f1e58bca000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c002a048> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"pool-17-IoTDB-InternalServiceRPC-Client-1" #46 prio=5 os_prio=0
tid=0x00007f1d6c001800 nid=0x19049 runnable [0x00007f1e58dcc000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000003c09523d8> (a java.io.BufferedInputStream)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:176)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at
org.apache.iotdb.rpc.TElasticFramedTransport.readFrame(TElasticFramedTransport.java:112)
at
org.apache.iotdb.rpc.TElasticFramedTransport.read(TElasticFramedTransport.java:107)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:463)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:361)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:244)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00000003c07be688> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"DestroyJavaVM" #45 prio=5 os_prio=0 tid=0x00007f1e7c018000 nid=0x19011 waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Sentinel-Thread" #36 prio=5 os_prio=0 tid=0x00007f1e7ce03000 nid=0x1903d in
Object.wait() [0x00007f1e5a0e8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000003c0019720> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L1PriorityQueue)
at java.lang.Object.wait(Object.java:502)
at
org.apache.iotdb.db.mpp.execution.schedule.queue.IndexedBlockingQueue.poll(IndexedBlockingQueue.java:66)
- locked <0x00000003c0019720> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L1PriorityQueue)
at
org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:56)
Locked ownable synchronizers:
- None
"Worker-Thread-7" #35 prio=5 os_prio=0 tid=0x00007f1e7ce01800 nid=0x1903c in
Object.wait() [0x00007f1e5a1e9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at java.lang.Object.wait(Object.java:502)
at
org.apache.iotdb.db.mpp.execution.schedule.queue.IndexedBlockingQueue.poll(IndexedBlockingQueue.java:66)
- locked <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at
org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:56)
Locked ownable synchronizers:
- None
"Worker-Thread-6" #34 prio=5 os_prio=0 tid=0x00007f1e7cdff800 nid=0x1903b in
Object.wait() [0x00007f1e5a2ea000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at java.lang.Object.wait(Object.java:502)
at
org.apache.iotdb.db.mpp.execution.schedule.queue.IndexedBlockingQueue.poll(IndexedBlockingQueue.java:66)
- locked <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at
org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:56)
Locked ownable synchronizers:
- None
"Worker-Thread-5" #33 prio=5 os_prio=0 tid=0x00007f1e7cdfe000 nid=0x1903a in
Object.wait() [0x00007f1e5a3eb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at java.lang.Object.wait(Object.java:502)
at
org.apache.iotdb.db.mpp.execution.schedule.queue.IndexedBlockingQueue.poll(IndexedBlockingQueue.java:66)
- locked <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at
org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:56)
Locked ownable synchronizers:
- None
"Worker-Thread-4" #32 prio=5 os_prio=0 tid=0x00007f1e7cdfc000 nid=0x19039 in
Object.wait() [0x00007f1e5a4ec000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at java.lang.Object.wait(Object.java:502)
at
org.apache.iotdb.db.mpp.execution.schedule.queue.IndexedBlockingQueue.poll(IndexedBlockingQueue.java:66)
- locked <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at
org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:56)
Locked ownable synchronizers:
- None
"Worker-Thread-3" #31 prio=5 os_prio=0 tid=0x00007f1e7cdfa800 nid=0x19038 in
Object.wait() [0x00007f1e5a5ed000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at java.lang.Object.wait(Object.java:502)
at
org.apache.iotdb.db.mpp.execution.schedule.queue.IndexedBlockingQueue.poll(IndexedBlockingQueue.java:66)
- locked <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at
org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:56)
Locked ownable synchronizers:
- None
"Worker-Thread-2" #30 prio=5 os_prio=0 tid=0x00007f1e7cdf8800 nid=0x19037 in
Object.wait() [0x00007f1e5a6ee000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at java.lang.Object.wait(Object.java:502)
at
org.apache.iotdb.db.mpp.execution.schedule.queue.IndexedBlockingQueue.poll(IndexedBlockingQueue.java:66)
- locked <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at
org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:56)
Locked ownable synchronizers:
- None
"Worker-Thread-1" #29 prio=5 os_prio=0 tid=0x00007f1e7cdf7800 nid=0x19036 in
Object.wait() [0x00007f1e5a7ef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at java.lang.Object.wait(Object.java:502)
at
org.apache.iotdb.db.mpp.execution.schedule.queue.IndexedBlockingQueue.poll(IndexedBlockingQueue.java:66)
- locked <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at
org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:56)
Locked ownable synchronizers:
- None
"Worker-Thread-0" #28 prio=5 os_prio=0 tid=0x00007f1e7cdf5800 nid=0x19035 in
Object.wait() [0x00007f1e5a8f0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at java.lang.Object.wait(Object.java:502)
at
org.apache.iotdb.db.mpp.execution.schedule.queue.IndexedBlockingQueue.poll(IndexedBlockingQueue.java:66)
- locked <0x00000003c00198c8> (a
org.apache.iotdb.db.mpp.execution.schedule.queue.L2PriorityQueue)
at
org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:56)
Locked ownable synchronizers:
- None
"pool-10-IoTDB-Timed-Flush-Unseq-Memtable-1" #26 prio=5 os_prio=0
tid=0x00007f1e7cdb7800 nid=0x19033 waiting on condition [0x00007f1e5abf3000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c0021858> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"pool-2-IoTDB-timedQuerySqlCount-1" #13 prio=5 os_prio=0 tid=0x00007f1e7ccb1000
nid=0x19026 waiting on condition [0x00007f1e5b8fe000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c0022100> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"logback-1" #10 daemon prio=5 os_prio=0 tid=0x00007f1e7ca1f800 nid=0x19024
waiting on condition [0x00007f1e609fd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c002a048> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f1e7c0e5800 nid=0x19022
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f1e7c0da800
nid=0x19021 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f1e7c0d8800
nid=0x19020 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f1e7c0d6800
nid=0x1901f waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f1e7c0d3800
nid=0x1901e waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f1e7c0d0000
nid=0x1901d runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f1e7c09c800 nid=0x1901c in
Object.wait() [0x00007f1e61bfa000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x00000003c0133d50> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f1e7c098000
nid=0x1901b in Object.wait() [0x00007f1e61cfb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000003c0022580> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=0 tid=0x00007f1e7c08e800 nid=0x1901a runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f1e7c02d000 nid=0x19012
runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f1e7c02f000 nid=0x19013
runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f1e7c030800 nid=0x19014
runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f1e7c032800 nid=0x19015
runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f1e7c034000 nid=0x19016
runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f1e7c036000 nid=0x19017
runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f1e7c037800 nid=0x19018
runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f1e7c039800 nid=0x19019
runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f1e7c0ea800 nid=0x19023 waiting
on condition
JNI global references: 1462
> [ remove datanode ] The new peer does not continue to synchronize, the leader
> does not accept new writes
> --------------------------------------------------------------------------------------------------------
>
> Key: IOTDB-4218
> URL: https://issues.apache.org/jira/browse/IOTDB-4218
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
> Affects Versions: 0.14.0-preview1
> Reporter: 刘珍
> Assignee: Song Ziyang
> Priority: Major
> Labels: pull-request-available
> Attachments: bm_config.properties, image-2022-08-24-11-05-21-029.png,
> image-2022-08-24-11-06-50-633.png, image-2022-08-24-11-07-43-638.png,
> ip1_log_all.log, ip3_log_all.log, ip4_log_all.log, ip5_log_all.log,
> screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> 测试版本：宋子阳提供的依赖他本地的ratis包版本，datanode的代码的commitID是
> 3585d6b2316b0efcac3ff35fdafd7806185f0285
> 3副本3C5D , 1 dataregion，缩容1个follower（ip1）：
> 问题1：{color:#DE350B}ip1 的data已经被删除，但是datanode进程不退出{color}
> 缩容节点ip1的error 日志
> !screenshot-1.png!
> 问题2：{color:#DE350B}leader（ip4）有报错，且不继续接受新的写入（缩容期间bm写入并不停止）{color}
> 2022-08-24 10:12:07,280 [pool-16-IoTDB-Region-Migrate-Pool-1] ERROR
> o.a.i.d.s.RegionMigrateService$AddRegionPeerTask:294 - add new peer
> TEndPoint(ip:192.168.130.5, port:40010) for region DataRegion[1] failed,
> resp: ConsensusGenericResponse{success=false}
> exception=org.apache.iotdb.consensus.exception.RatisRequestFailedException:
> Ratis request failed
> 问题3：接问题2的{color:#DE350B}leader报错add new peer failed，但是show regions ，却看到new
> peer是add成功的{color}。
> !image-2022-08-24-11-07-43-638.png!
> 问题4：{color:#DE350B}new peer ip5不同步{color}：
> !image-2022-08-24-11-05-21-029.png!
> leader ip4的raft log（10:22后没有新的写入，此时bm并没停止）
> !image-2022-08-24-11-06-50-633.png!
> 复现流程：
> 1. 192.168.130.1 / 2/3/4/5
> 3,4,5 机器 16核32GB，
> 1，2 机器 8核32GB
> bm在ip2
> 数据库配置参数
> ConfigNode
> MAX_HEAP_SIZE="4G"
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> schema_replication_factor=3
> data_replication_factor=3
> DataNode配置参数
> MAX_HEAP_SIZE="16G"
> max_waiting_time_when_insert_blocked=3600000
> query_timeout_threshold=36000000
> partition_interval=86400
> 2. 启动benchmark，配置文件见附件
> regions信息：
> !screenshot-2.png!
> 3. 缩容follower ip1 (ip1 有snapshot）
> 4. 问题现象见问题描述
> ip1 ，ip4，ip5的日志见附件
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
