choubenson opened a new pull request, #11692: URL: https://github.com/apache/iotdb/pull/11692
Add `enable_separate_data` param to switch whether to enable data separation or not. The default is true, which means the data is separated into sequence and unsequence data dir. If it is false, all data will be written to the unsequence data dir. Our system maintains two data structures in the `HashLastFlushTimeMap` class, one is `partitionLatestFlushedTimeForEachDevice` and the other is `globalLatestFlushedTimeForEachDevice`. The former is used to separate seq and unseq data in each partition, and the latter is used for last cache queries. 1. When the `enable_separate_data` parameter is false, `partitionLatestFlushedTimeForEachDevice` is no longer maintained in the memory. There is no need to judge the order of data, and the data is written directly to the unseq data directory. In the writing scenario of massive devices, it can save a lot of memory and greatly improve the writing performance, avoiding disk IO to obtain `deviceLastFlushTime` after resource degrades, which consumes a lot of CPU. 2. When the `enable_last_cache` parameter is false, `globalLatestFlushedTimeForEachDevice` is no longer maintained in the memory, which can save a lot of memory in the scenario of writing massive devices. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
