Hi, I think the param number 3 and number 6 can be merged into one parameter. The checking interval can be the same value for both sequence and unsequence memtables. I don’t think any users will try to set different value for them.
How do you think? Thanks, Haonan Hou > On Aug 19, 2021, at 6:04 PM, Jialin Qiao <[email protected]> wrote: > > Hi, > >> I am confused whether to open timed flush seq/unseq memtable and timed > close TsFileProcessor by default, because they can save memory in some user > cases while they just waste cpu in other user cases. > > It could be enabled by default, both seq and unseq. > The 12 hours for memtable flush interval and close file interval is fine. > If it really kept 12 hours in memory, the CPU load should be very low, no > need to worry. > > Thanks, > ————————————————— > Jialin Qiao > School of Software, Tsinghua University > > 乔嘉林 > 清华大学 软件学院 > > > Haiming Zhu <[email protected]> 于2021年8月19日周四 下午4:37写道： > >> Hi, >> >> >> I have already implemented both issues [1][2]. >> >> >> Timed flush memtable [3] is used to flush inactive memtables periodically >> according to their created time. For example, if we set >> seq_memtable_flush_interval_in_ms to 1000, memtable A, created at time >> 2000, can be flushed after time 3000. >> >> >> Timed close TsFileProcessor [4] is used to close inactive TsFileProcessors >> periodically according to how long they don't process write operations. For >> example, if we set close_tsfile_interval_after_flushing_in_ms to 1000, >> TsFileProcessor A, whose last workMemtable was flush at time 2000 and >> current workMemtable is null (i.e., no more wirite operations after last >> flush), can be closed after time 3000. >> >> >> But, i am confused whether to open timed flush seq/unseq memtable and >> timed close TsFileProcessor by default, because they can save memory in >> some user cases while they just waste cpu in other user cases. >> >> >> Look forward to your reply. >> >> >> [1] https://github.com/apache/iotdb/pull/3776 >> [2] https://github.com/apache/iotdb/pull/3777 >> >> [3] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 >> [4] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 >> >> >> Thanks, >> ————————————————— >> Haiming Zhu >> >> >> >> >> ------------------ 原始邮件 ------------------ >> 发件人: >> "dev" >> >> <[email protected]>; >> 发送时间: 2021年8月16日(星期一) 下午5:18 >> 收件人: "IoTDB邮件列表"<[email protected]>; >> >> 主题: Timed flush memtable and timed close TsFileProcessor >> >> >> >> Hi, >> >> >> If we open a TsFileProcessor and then never write to it, this >> TsFileProcessor will become a zombie because nothing can trigger a flush or >> close operation. So, i am implementing timed flush memtable [1] and timed >> close TsFileProcessor [2]. Timed flush memtable is used to flush inactive >> memtables periodically according to their created time. Timed close >> TsFileProcessor is used to close inactive >> TsFileProcessors&nbsp;periodically according to how long they don't >> process write operations. >> >> >> I add 9 params to control above two functions, and these 9 params all can >> be hot loaded by 'load configuration' command. >> >> >> 1. enable_timed_flush_seq_memtable: whether to timed flush sequence >> tsfiles' memtables. >> 2. seq_memtable_flush_interval_in_ms: when a sequence memTable's created >> time is older than current time minus this, the memtable is flushed to disk. >> 3. seq_memtable_flush_check_interval_in_ms: the interval to check whether >> the sequence&nbsp;memtables need flushing. >> 4. enable_timed_flush_unseq_memtable: whether to timed flush unsequence >> tsfiles' memtables. >> 5. unseq_memtable_flush_interval_in_ms: when an unsequence memTable's >> created time is older than current time minus this, the memtable is flushed >> to disk. >> 6. unseq_memtable_flush_check_interval_in_ms: the interval to check >> whether the unsequence&nbsp;memtables need flushing. >> 7. enable_timed_close_tsfile: whether to timed close TsfileProcessors. >> 8.&nbsp;close_tsfile_interval_after_flushing_in_ms: when >> TsfileProcessor's last working memtable flush time is older than current >> time minus this and its working memtable is null, this TsfileProcessor will >> be closed. >> 9.&nbsp;close_tsfile_check_interval_in_ms: the interval to check >> whether the TsfileProcessors&nbsp;need closing. >> >> >> What do you think? >> >> >> [1]&nbsp; >> https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 >> [2] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 >> >> >> Thanks, >> ————————————————— >> Haiming Zhu >
