Re: add vm(hot compaction) in tsfile processor

[445073309]

Hi,

max_vm_num means that the most number of vm files relation to a tsfile's level.
for example, we set max_vm_num=5 and we flush 11 times, then the compaction 
procedure can be described as below:
* when we flush 5(max_vm_num) times, the current level will do compaction 
to the next level

* when we flush all 11 times, the compaction procedure is

* if we close the tsfile, the whole compaction procedure will be


And we set default max_vm_num=5 in current version, if user do not know 
which value is suitable, just use the default value is enough to make chunk 
larger.
Best,
-----------------------------------
Lingzhe Zhang
School of Software, Tsinghua University

??????
???????? ????????




------------------ ???????? ------------------
??????:                                                                         
                                               "dev"                            
                                                        <saint...@gmail.com&gt;;
????????:&nbsp;2020??7??20??(??????) ????3:10
??????:&nbsp;"dev"<dev@iotdb.apache.org&gt;;

????:&nbsp;Re: add vm(hot compaction) in tsfile processor



Hi Lingzhe,

&gt;max_vm_num: indicates that a TsFileProcessor has at most the number of
virtual memory files

what does this mean? and how do I know what value is suitable? (For
example, if I set it as 1, is there any impact?)

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

&nbsp;??????
???????? ????????


445073309 <sure...@foxmail.com&gt; ??2020??7??20?????? ????12:42??????

&gt; Hi,
&gt;
&gt;
&gt; I met a problem that iotdb will write small chunk data when lack of
&gt; memtable num, this causes the system to query hot data more slowly.
&gt;
&gt;
&gt; So I create a new type of file -- vm file, and use it to do hot compaction
&gt; in flush processor. With this, we can flexiblily controll the size of each
&gt; chunk. The configuration and usage changes will be described as follows:
&gt; * add a new parameter enable_vm in iotdb-engine.properties: indicates
&gt; whether to use virtual memory
&gt; * use parameter avg_series_point_number_threshold in 
iotdb-engine.properties:
&gt; indicates the minimum average number of chunk data points after hot
&gt; compaction
&gt; * add a new parameter max_vm_num in iotdb-engine.properties:
&gt; indicates that a TsFileProcessor has at most the number of virtual memory
&gt; files
&gt; * add a new parameter max_merge_chunk_num_in_tsfile in 
iotdb-engine.properties:
&gt; indicates the vm files max merge times
&gt; * the suffix of the vm file is'.vm', and the naming convention is
&gt; {tsfile_name}-{level}-{timestamp}.vm
&gt;
&gt; And there are many detail changes like:
&gt; * set virtual memory file list List<List<TsFileResource&gt;&gt;
&gt; vmTsFileResources for each TsFileProcessor, add
&gt; List<List<RestorableTsFileIOWriter&gt;&gt; vmWriters for management
&gt; * in the recover process, the recovery of the vm file is newly added, and
&gt; the corresponding TsFileProcessor is injected after the recovery
&gt;
&gt; The compaction strategy is now writen like LeveledCompactionStrategy in 
Cassandra,
&gt; and it can be optimized later.
&gt;
&gt; I put the detail zh-doc in the attachment.
&gt;
&gt; Thanks,
&gt; --
&gt; Lingzhe Zhang
&gt; School of Software, Tsinghua University
&gt;
&gt; ??????
&gt; ???????? ????????
&gt;