Hi,

I met a problem that iotdb will write small chunk data when lack of memtable 
num, this causes the system to query hot data more slowly.


So I create a new type of file -- vm file, and use it to do hot compaction in 
flush processor. With this, we can flexiblily controll the size of each chunk. 
The configuration and usage changes will be described as follows:
* add a new parameter enable_vm in iotdb-engine.properties: indicates 
whether to use virtual memory
* use parameter 
avg_series_point_number_threshold in iotdb-engine.properties: 
indicates the minimum average number of chunk data points after hot 
compaction
* add a new parameter max_vm_num in iotdb-engine.properties: 
indicates that a TsFileProcessor has at most the number of virtual memory 
files
* add a new parameter max_merge_chunk_num_in_tsfile 
in iotdb-engine.properties: indicates the vm files max merge times
* the suffix of the vm file is'.vm', and the naming convention is 
{tsfile_name}-{level}-{timestamp}.vm


And there are many detail changes like:
* set virtual memory file list List<List<TsFileResource&gt;&gt; 
vmTsFileResources for each TsFileProcessor, add 
List<List<RestorableTsFileIOWriter&gt;&gt; vmWriters for management
* in the recover process, the recovery of the vm file is newly added, and the 
corresponding TsFileProcessor is injected after the recovery

The compaction strategy is now writen like LeveledCompactionStrategy 
in&nbsp;Cassandra, and it can be&nbsp;optimized later.&nbsp;


I put the detail zh-doc in the attachment.

Thanks,
--
Lingzhe Zhang
School of Software, Tsinghua University

??????
???????? ????????

Reply via email to