Hi,
Param seq_memtable_flush_check_interval_in_ms and
unseq_memtable_flush_check_interval_in_ms aim to make timed flush memtable
configuration more flexible. If users don't have this requirement, i will merge
both params into one param memtable_flush_check_interval_in_ms.
Thanks,
??????????????????????????????????
Haiming Zhu
------------------ ???????? ------------------
??????:
"dev"
<[email protected]>;
????????: 2021??8??19??(??????) ????11:05
??????: "dev-iotdb"<[email protected]>;
????: Re: Timed flush memtable and timed close TsFileProcessor
Hi,
I think the param number 3 and number 6 can be merged into one parameter.
The checking interval can be the same value for both sequence and unsequence
memtables.
I don??t think any users will try to set different value for them.
How do you think?
Thanks,
Haonan Hou
> On Aug 19, 2021, at 6:04 PM, Jialin Qiao <[email protected]> wrote:
>
> Hi,
>
>> I am confused whether to open timed flush seq/unseq memtable and timed
> close TsFileProcessor by default, because they can save memory in some user
> cases while they just waste cpu in other user cases.
>
> It could be enabled by default, both seq and unseq.
> The 12 hours for memtable flush interval and close file interval is fine.
> If it really kept 12 hours in memory, the CPU load should be very low, no
> need to worry.
>
> Thanks,
> ??????????????????????????????????
> Jialin Qiao
> School of Software, Tsinghua University
>
> ??????
> ???????? ????????
>
>
> Haiming Zhu <[email protected]> ??2021??8??19?????? ????4:37??????
>
>> Hi,
>>
>>
>> I have already implemented both issues [1][2].
>>
>>
>> Timed flush memtable [3] is used to flush inactive memtables
periodically
>> according to their created time. For example, if we set
>> seq_memtable_flush_interval_in_ms to 1000, memtable A, created at time
>> 2000, can be flushed after time 3000.
>>
>>
>> Timed close TsFileProcessor [4] is used to close inactive
TsFileProcessors
>> periodically according to how long they don't process write
operations. For
>> example, if we set close_tsfile_interval_after_flushing_in_ms to 1000,
>> TsFileProcessor A, whose last workMemtable was flush at time 2000 and
>> current workMemtable is null (i.e., no more wirite operations after
last
>> flush), can be closed after time 3000.
>>
>>
>> But, i am confused whether to open timed flush seq/unseq memtable and
>> timed close TsFileProcessor by default, because they can save memory in
>> some user cases while they just waste cpu in other user cases.
>>
>>
>> Look forward to your reply.
>>
>>
>> [1]&nbsp;https://github.com/apache/iotdb/pull/3776
>> [2]&nbsp;https://github.com/apache/iotdb/pull/3777
>>
>>
[3]&nbsp;https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496
>>
[4]&nbsp;https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569
>>
>>
>> Thanks,
>> ??????????????????????????????????
>> Haiming Zhu
>>
>>
>>
>>
>> ------------------&nbsp;????????&nbsp;------------------
>> ??????:
>>
"dev"
>>
>> <[email protected]&gt;;
>> ????????:&nbsp;2021??8??16??(??????) ????5:18
>> ??????:&nbsp;"IoTDB????????"<[email protected]&gt;;
>>
>> ????:&nbsp;Timed flush memtable and timed close TsFileProcessor
>>
>>
>>
>> Hi,
>>
>>
>> If we open a TsFileProcessor and then never write to it, this
>> TsFileProcessor will become a zombie because nothing can trigger a
flush or
>> close operation. So, i am implementing timed flush memtable [1] and
timed
>> close TsFileProcessor [2]. Timed flush memtable is used to flush
inactive
>> memtables periodically according to their created time. Timed close
>> TsFileProcessor is used to close inactive
>> TsFileProcessors&amp;nbsp;periodically according to how long they
don't
>> process write operations.
>>
>>
>> I add 9 params to control above two functions, and these 9 params all
can
>> be hot loaded by 'load configuration' command.
>>
>>
>> 1. enable_timed_flush_seq_memtable: whether to timed flush sequence
>> tsfiles' memtables.
>> 2. seq_memtable_flush_interval_in_ms: when a sequence memTable's
created
>> time is older than current time minus this, the memtable is flushed to
disk.
>> 3. seq_memtable_flush_check_interval_in_ms: the interval to check
whether
>> the sequence&amp;nbsp;memtables need flushing.
>> 4. enable_timed_flush_unseq_memtable: whether to timed flush unsequence
>> tsfiles' memtables.
>> 5. unseq_memtable_flush_interval_in_ms: when an unsequence memTable's
>> created time is older than current time minus this, the memtable is
flushed
>> to disk.
>> 6. unseq_memtable_flush_check_interval_in_ms: the interval to check
>> whether the unsequence&amp;nbsp;memtables need flushing.
>> 7. enable_timed_close_tsfile: whether to timed close TsfileProcessors.
>> 8.&amp;nbsp;close_tsfile_interval_after_flushing_in_ms: when
>> TsfileProcessor's last working memtable flush time is older than
current
>> time minus this and its working memtable is null, this TsfileProcessor
will
>> be closed.
>> 9.&amp;nbsp;close_tsfile_check_interval_in_ms: the interval to
check
>> whether the TsfileProcessors&amp;nbsp;need closing.
>>
>>
>> What do you think?
>>
>>
>> [1]&amp;nbsp;
>> https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496
>> [2] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569
>>
>>
>> Thanks,
>> ??????????????????????????????????
>> Haiming Zhu
>
