shuwenwei commented on issue #11375: URL: https://github.com/apache/iotdb/issues/11375#issuecomment-1791824819
> > Hi, I noticed that you are writing a request size of more than 200M, which is actually too large. We suggest reducing the batch size to reduce the memory pressure on the server. > > In addition, in version 1.2.0, IoTConsensus was unable to synchronize requests larger than 100M in size, which could cause individual nodes to pile up the wal log after receiving a large request, up to 50GB. This issue was [fixed](https://github.com/apache/iotdb/pull/11145) in version 1.2.2. > > It is recommended to upgrade to 1.2.2, reduce the batch size appropriately and try again > > Thansk for your advice. I have upgrade to 1.2.2 and WAL problem seems solved. But I have seen some **INFO** log about compaction memory both in 1.2.0 and 1.2.2, I have change the memory ratio of storage engine and it seems can not be avoid. It's this any affect？ `2023-11-01 06:09:29,157 [pool-44-IoTDB-Compaction-Worker-8] INFO o.a.i.d.s.d.c.e.t.CrossSpaceCompactionTask:397 - No enough memory for current compaction task root.pre_trc-27-2922 task seq files are [file is /data1/iotdb/apache-iotdb-1.2.2-all-bin/data/datanode/data/sequence/root.pre_trc/27/2922/1697717961585-1-2-4.tsfile, status: COMPACTION_CANDIDATE] , unseq files are [file is /data1/iotdb/apache-iotdb-1.2.2-all-bin/data/datanode/data/unsequence/root.pre_trc/27/2922/1698751814360-23-1-0.tsfile, status: COMPACTION_CANDIDATE] org.apache.iotdb.db.storageengine.dataregion.compaction.execute.exception.CompactionMemoryNotEnoughException: Required memory cost 1115215382 bytes is greater than the total memory budget for com paction 823216046 bytes at org.apache.iotdb.db.storageengine.rescon.memory.SystemInfo.addCompactionMemoryCost(SystemInfo.java:223) at org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.CrossSpaceCompactionTask.checkValidAndSetMerging(CrossSpaceCompactionTask.java:389) at org.apache.iotdb.db.storageengine.dataregion.compaction.schedule.CompactionWorker.run(CompactionWorker.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)` Do you use aligned series for storage？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
