Hi, > I am confused whether to open timed flush seq/unseq memtable and timed close TsFileProcessor by default, because they can save memory in some user cases while they just waste cpu in other user cases.
It could be enabled by default, both seq and unseq. The 12 hours for memtable flush interval and close file interval is fine. If it really kept 12 hours in memory, the CPU load should be very low, no need to worry. Thanks, ————————————————— Jialin Qiao School of Software, Tsinghua University 乔嘉林 清华大学 软件学院 Haiming Zhu <[email protected]> 于2021年8月19日周四 下午4:37写道： > Hi, > > > I have already implemented both issues [1][2]. > > > Timed flush memtable [3] is used to flush inactive memtables periodically > according to their created time. For example, if we set > seq_memtable_flush_interval_in_ms to 1000, memtable A, created at time > 2000, can be flushed after time 3000. > > > Timed close TsFileProcessor [4] is used to close inactive TsFileProcessors > periodically according to how long they don't process write operations. For > example, if we set close_tsfile_interval_after_flushing_in_ms to 1000, > TsFileProcessor A, whose last workMemtable was flush at time 2000 and > current workMemtable is null (i.e., no more wirite operations after last > flush), can be closed after time 3000. > > > But, i am confused whether to open timed flush seq/unseq memtable and > timed close TsFileProcessor by default, because they can save memory in > some user cases while they just waste cpu in other user cases. > > > Look forward to your reply. > > > [1] https://github.com/apache/iotdb/pull/3776 > [2] https://github.com/apache/iotdb/pull/3777 > > [3] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 > [4] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 > > > Thanks, > ————————————————— > Haiming Zhu > > > > > ------------------ 原始邮件 ------------------ > 发件人: > "dev" > > <[email protected]>; > 发送时间: 2021年8月16日(星期一) 下午5:18 > 收件人: "IoTDB邮件列表"<[email protected]>; > > 主题: Timed flush memtable and timed close TsFileProcessor > > > > Hi, > > > If we open a TsFileProcessor and then never write to it, this > TsFileProcessor will become a zombie because nothing can trigger a flush or > close operation. So, i am implementing timed flush memtable [1] and timed > close TsFileProcessor [2]. Timed flush memtable is used to flush inactive > memtables periodically according to their created time. Timed close > TsFileProcessor is used to close inactive > TsFileProcessors&nbsp;periodically according to how long they don't > process write operations. > > > I add 9 params to control above two functions, and these 9 params all can > be hot loaded by 'load configuration' command. > > > 1. enable_timed_flush_seq_memtable: whether to timed flush sequence > tsfiles' memtables. > 2. seq_memtable_flush_interval_in_ms: when a sequence memTable's created > time is older than current time minus this, the memtable is flushed to disk. > 3. seq_memtable_flush_check_interval_in_ms: the interval to check whether > the sequence&nbsp;memtables need flushing. > 4. enable_timed_flush_unseq_memtable: whether to timed flush unsequence > tsfiles' memtables. > 5. unseq_memtable_flush_interval_in_ms: when an unsequence memTable's > created time is older than current time minus this, the memtable is flushed > to disk. > 6. unseq_memtable_flush_check_interval_in_ms: the interval to check > whether the unsequence&nbsp;memtables need flushing. > 7. enable_timed_close_tsfile: whether to timed close TsfileProcessors. > 8.&nbsp;close_tsfile_interval_after_flushing_in_ms: when > TsfileProcessor's last working memtable flush time is older than current > time minus this and its working memtable is null, this TsfileProcessor will > be closed. > 9.&nbsp;close_tsfile_check_interval_in_ms: the interval to check > whether the TsfileProcessors&nbsp;need closing. > > > What do you think? > > > [1]&nbsp; > https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 > [2] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 > > > Thanks, > ————————————————— > Haiming Zhu
