[ 
https://issues.apache.org/jira/browse/IOTDB-4700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627060#comment-17627060
 ] 

刘珍 commented on IOTDB-4700:
---------------------------

 !screenshot-1.png! 

> Flush performance is slow
> -------------------------
>
>                 Key: IOTDB-4700
>                 URL: https://issues.apache.org/jira/browse/IOTDB-4700
>             Project: Apache IoTDB
>          Issue Type: Improvement
>          Components: mpp-cluster
>            Reporter: 刘珍
>            Assignee: Haiming Zhu
>            Priority: Major
>         Attachments: image-2022-10-20-17-12-15-915.png, 
> image-2022-10-20-17-21-01-580.png, image-2022-10-20-17-26-48-200.png, 
> image-2022-10-20-17-27-46-815.png, more_metadata.conf, screenshot-1.png
>
>
> m_1019_f2ffb49
> 3rep,3C3D
> {color:#DE350B}stop-datanode.sh , flushing metadata is slow :
> At first , there is only one flush thread (pool-4-IoTDB-Flush-1 )
> There are three Dataregions  , so three flush threads should execute in 
> parallel{color}
>  !image-2022-10-20-17-12-15-915.png! 
> *{color:#DE350B}grep "time consumption of flushing metadata" log_all.log 
> {color}*
> 2022-10-20 16:11:31,753 [pool-4-IoTDB-Flush-1] INFO  
> o.a.i.d.e.s.TsFileProcessor:1440 - Storage group root.test.g_0-5 close the 
> file 
> /data/mpp_test/m_1019_f2ffb49/datanode/sbin/../data/data/sequence/root.test.g_0/5/0/1666252685941-1-0-0.tsfile,
>  TsFile size is 997716505, time consumption of flushing metadata is 154677ms 
> 2022-10-20 16:17:06,898 [pool-4-IoTDB-Flush-2] INFO  
> o.a.i.d.e.s.TsFileProcessor:1440 - Storage group root.test.g_2-3 close the 
> file 
> /data/mpp_test/m_1019_f2ffb49/datanode/sbin/../data/data/sequence/root.test.g_2/3/0/1666252682557-1-0-0.tsfile,
>  TsFile size is 944837569, time consumption of flushing metadata is 267027ms 
> 2022-10-20 16:17:21,113 [pool-4-IoTDB-Flush-3] INFO  
> o.a.i.d.e.s.TsFileProcessor:1440 - Storage group root.test.g_1-4 close the 
> file 
> /data/mpp_test/m_1019_f2ffb49/datanode/sbin/../data/data/sequence/root.test.g_1/4/0/1666252674037-1-0-0.tsfile,
>  TsFile size is 997716510, time consumption of flushing metadata is 272974ms 
> {color:#DE350B}*grep "flushing a memtable has finished" 
> logs/log_all.log*{color}
> 2022-10-20 16:08:51,254 [pool-4-IoTDB-Flush-1] INFO  
> o.a.i.d.e.f.MemTableFlushTask:187 - Storage group root.test.g_0-5 memtable 
> PrimitiveMemTable{planIndex=[9223372036854775807,-9223372036854775808]} 
> flushing a memtable has finished! Time consumption: 53988ms 
> 2022-10-20 16:12:30,663 [pool-4-IoTDB-Flush-2] INFO  
> o.a.i.d.e.f.MemTableFlushTask:187 - Storage group root.test.g_2-3 memtable 
> PrimitiveMemTable{planIndex=[9223372036854775807,-9223372036854775808]} 
> flushing a memtable has finished! Time consumption: 58898ms 
> 2022-10-20 16:12:38,571 [pool-4-IoTDB-Flush-3] INFO  
> o.a.i.d.e.f.MemTableFlushTask:187 - Storage group root.test.g_1-4 memtable 
> PrimitiveMemTable{planIndex=[9223372036854775807,-9223372036854775808]} 
> flushing a memtable has finished! Time consumption: 66798ms 
> 1. 192.168.10.72/73/74       48CPU 384GB
> ConfigNode
> MAX_HEAP_SIZE="8G"
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
> schema_replication_factor=3
> data_replication_factor=3
> DataNode
> MAX_HEAP_SIZE="256G"
> MAX_DIRECT_MEMORY_SIZE="32G"
> max_connection_for_internal_service=300
> enable_timed_flush_seq_memtable=true
> seq_memtable_flush_interval_in_ms=600000
> seq_memtable_flush_check_interval_in_ms=300000
> enable_timed_flush_unseq_memtable=true
> seq_memtable_flush_interval_in_ms=600000
> seq_memtable_flush_check_interval_in_ms=300000
> max_waiting_time_when_insert_blocked=3600000
> query_timeout_threshold=3600000
> 2. bm
> See the attachment
>  !image-2022-10-20-17-21-01-580.png! 
> 3. benchmark write completed, ip74 datanode synchronization completed
> 4. stop ip74 datanode
> Execute  stop-datanode.sh
> before stop datanode:
>  !image-2022-10-20-17-26-48-200.png! 
> after stop datanode(take snapshot):
>  !image-2022-10-20-17-27-46-815.png! 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to