LangYuanzh commented on issue #11375:
URL: https://github.com/apache/iotdb/issues/11375#issuecomment-1786441482
>
三个节点的配置和硬件是否一致，数据分布是否均匀（每个设备的写入频率？），是否有一些轮询操作，是否使用一些其他功能（pipe?视图？）。可以在三个节点上打个火焰图看看线程池在忙啥
三个节点硬件一致均为8C16G。没有轮询，视图等特殊功能。目前停掉了写入和查询。
节点1 和节点 3上WAL文件会正常清理掉。但是节点2上wal日志一直未清理。
修改了配置文件中的iot_consensus_throttle_threshold_in_byte参数，但是看日志未生效。
另外日志里看到了一些compaction内存不足的INFO日志，不确定是否有影响。
2023-10-31 04:45:47,615 [pool-33-IoTDB-Compaction-Worker-5] INFO
o.a.i.d.e.c.e.t.CrossSpaceCompactionTask:387 - No enough memory for current
compaction task root.pre_trc-26-2817 task seq files are [file is
/data1/iotdb/apache-iotdb-1.2.0-all-bin/data/datanode/data/sequence/root.pre_trc/26/2817/1697441628722-1-2-1.tsfile,
status: COMPACTION_CANDIDATE, file is
/data1/iotdb/apache-iotdb-1.2.0-all-bin/data/datanode/data/sequence/root.pre_trc/26/2817/1697442224603-63-5-0.tsfile,
status: COMPACTION_CANDIDATE] , unseq files are [file is
/data1/iotdb/apache-iotdb-1.2.0-all-bin/data/datanode/data/unsequence/root.pre_trc/26/2817/1697934461705-1686-9-0.tsfile,
status: COMPACTION_CANDIDATE]
org.apache.iotdb.db.engine.compaction.execute.exception.CompactionMemoryNotEnoughException:
Required memory cost 7444181649 bytes is greater than the total memory budget
for compaction 3514064151 bytes
at
org.apache.iotdb.db.rescon.SystemInfo.addCompactionMemoryCost(SystemInfo.java:223)
at
org.apache.iotdb.db.engine.compaction.execute.task.CrossSpaceCompactionTask.checkValidAndSetMerging(CrossSpaceCompactionTask.java:380)
at
org.apache.iotdb.db.engine.compaction.schedule.CompactionWorker.run(CompactionWorker.java:58)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-10-31 04:45:50,201
[AsyncDataNodeIoTConsensusServiceClientPool-selector-94] INFO
o.a.i.c.i.c.AsyncIoTConsensusServiceClient:112 - Unexpected exception occurs in
AsyncConfigNodeIServiceClient{TEndPoint(ip:iotdb-aesc-trc0001, port:10760)} :
java.lang.IllegalStateException: Client has an error!
at
org.apache.thrift.async.TAsyncClient.checkReady(TAsyncClient.java:83)
at
org.apache.iotdb.consensus.iot.client.AsyncIoTConsensusServiceClient.isReady(AsyncIoTConsensusServiceClient.java:109)
at
org.apache.iotdb.consensus.iot.client.AsyncIoTConsensusServiceClient$Factory.validateObject(AsyncIoTConsensusServiceClient.java:152)
at
org.apache.iotdb.consensus.iot.client.AsyncIoTConsensusServiceClient$Factory.validateObject(AsyncIoTConsensusServiceClient.java:122)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:1470)
at
org.apache.iotdb.commons.client.ClientManager.lambda$returnClient$0(ClientManager.java:70)
at java.base/java.util.Optional.ifPresent(Optional.java:178)
at
org.apache.iotdb.commons.client.ClientManager.returnClient(ClientManager.java:67)
at
org.apache.iotdb.consensus.iot.client.AsyncIoTConsensusServiceClient.returnSelf(AsyncIoTConsensusServiceClient.java:99)
at
org.apache.iotdb.consensus.iot.client.AsyncIoTConsensusServiceClient.onError(AsyncIoTConsensusServiceClient.java:74)
at
org.apache.thrift.async.TAsyncMethodCall.onError(TAsyncMethodCall.java:215)
at
org.apache.thrift.async.TAsyncMethodCall.transition(TAsyncMethodCall.java:210)
at
org.apache.thrift.async.TAsyncClientManager$SelectThread.transitionMethods(TAsyncClientManager.java:143)
at
org.apache.thrift.async.TAsyncClientManager$SelectThread.run(TAsyncClientManager.java:113)
Caused by: org.apache.thrift.transport.TTransportException: Read call frame
size failed
at
org.apache.thrift.async.TAsyncMethodCall.doReadingResponseSize(TAsyncMethodCall.java:246)
at
org.apache.thrift.async.TAsyncMethodCall.transition(TAsyncMethodCall.java:198)
... 2 common frames omitted
2023-10-31 04:45:50,201
[AsyncDataNodeIoTConsensusServiceClientPool-selector-94] WARN
o.a.i.c.i.c.DispatchLogHandler:81 - Can not send Batch{startIndex=3549,
endIndex=3549, size=1, serializedSize=214780866} to peer for
Peer{groupId=DataRegion[26], endpoint=TEndPoint(ip:iotdb-aesc-trc0001,
port:10760), nodeId=1} times 333 because {}
org.apache.thrift.transport.TTransportException: Read call frame size failed
at
org.apache.thrift.async.TAsyncMethodCall.doReadingResponseSize(TAsyncMethodCall.java:246)
at
org.apache.thrift.async.TAsyncMethodCall.transition(TAsyncMethodCall.java:198)
at
org.apache.thrift.async.TAsyncClientManager$SelectThread.transitionMethods(TAsyncClientManager.java:143)
at
org.apache.thrift.async.TAsyncClientManager$SelectThread.run(TAsyncClientManager.java:113)
2023-10-31 04:45:58,225 [pool-6-IoTDB-WAL-Delete-1] WARN
o.a.i.db.wal.WALManager:180 - WAL disk usage 53712680440 is larger than the
iot_consensus_throttle_threshold_in_byte 50949672960, please check your write
load, iot consensus and the pipe module. It's better to allocate more disk for
WAL.
2023-10-31 04:46:00,061
[AsyncDataNodeIoTConsensusServiceClientPool-selector-94] INFO
o.a.i.c.i.c.AsyncIoTConsensusServiceClient:112 - Unexpected exception occurs in
AsyncConfigNodeIServiceClient{TEndPoint(ip:iotdb-aesc-trc0001, port:10760)} :
java.lang.IllegalStateException: Client has an error!
at
org.apache.thrift.async.TAsyncClient.checkReady(TAsyncClient.java:83)
at
org.apache.iotdb.consensus.iot.client.AsyncIoTConsensusServiceClient.isReady(AsyncIoTConsensusServiceClient.java:109)
at
org.apache.iotdb.consensus.iot.client.AsyncIoTConsensusServiceClient$Factory.validateObject(AsyncIoTConsensusServiceClient.java:152)
at
org.apache.iotdb.consensus.iot.client.AsyncIoTConsensusServiceClient$Factory.validateObject(AsyncIoTConsensusServiceClient.java:122)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:1470)
at
org.apache.iotdb.commons.client.ClientManager.lambda$returnClient$0(ClientManager.java:70)
at java.base/java.util.Optional.ifPresent(Optional.java:178)
at
org.apache.iotdb.commons.client.ClientManager.returnClient(ClientManager.java:67)
at
org.apache.iotdb.consensus.iot.client.AsyncIoTConsensusServiceClient.returnSelf(AsyncIoTConsensusServiceClient.java:99)
at
org.apache.iotdb.consensus.iot.client.AsyncIoTConsensusServiceClient.onError(AsyncIoTConsensusServiceClient.java:74)
at
org.apache.thrift.async.TAsyncMethodCall.onError(TAsyncMethodCall.java:215)
at
org.apache.thrift.async.TAsyncMethodCall.transition(TAsyncMethodCall.java:210)
at
org.apache.thrift.async.TAsyncClientManager$SelectThread.transitionMethods(TAsyncClientManager.java:143)
at
org.apache.thrift.async.TAsyncClientManager$SelectThread.run(TAsyncClientManager.java:113)
Caused by: org.apache.thrift.transport.TTransportException: Read call frame
size failed
at
org.apache.thrift.async.TAsyncMethodCall.doReadingResponseSize(TAsyncMethodCall.java:246)
at
org.apache.thrift.async.TAsyncMethodCall.transition(TAsyncMethodCall.java:198)
... 2 common frames omitted
2023-10-31 04:46:00,062
[AsyncDataNodeIoTConsensusServiceClientPool-selector-94] WARN
o.a.i.c.i.c.DispatchLogHandler:81 - Can not send Batch{startIndex=3550,
endIndex=3550, size=1, serializedSize=214781055} to peer for
Peer{groupId=DataRegion[26], endpoint=TEndPoint(ip:iotdb-aesc-trc0001,
port:10760), nodeId=1} times 324 because {}
org.apache.thrift.transport.TTransportException: Read call frame size failed
at
org.apache.thrift.async.TAsyncMethodCall.doReadingResponseSize(TAsyncMethodCall.java:246)
at
org.apache.thrift.async.TAsyncMethodCall.transition(TAsyncMethodCall.java:198)
at
org.apache.thrift.async.TAsyncClientManager$SelectThread.transitionMethods(TAsyncClientManager.java:143)
at
org.apache.thrift.async.TAsyncClientManager$SelectThread.run(TAsyncClientManager.java:113)
2023-10-31 04:46:18,229 [pool-6-IoTDB-WAL-Delete-1] WARN
o.a.i.db.wal.WALManager:180 - WAL disk usage 53712680440 is larger than the
iot_consensus_throttle_threshold_in_byte 50949672960, please check your write
load, iot consensus and the pipe module. It's better to allocate more disk for
WAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
