Hi， Recently I'm working on adjusting parameters dynamically according to different workloads. I have the following ideas：
There are three dynamically adjustable parameters in the IoTD: the number of MemTables, the size of per MemTable and the threshold size of TsFile. 1. the number of MemTables. This parameter represents the size of the MemTable available in the MemTable pool, which is closely related to the number of storage groups. In my design, this parameter is set to twice the number of storage groups plus a constant for backup. The reason why the ratio is set twice is that when the system is running stably, the speed of the flush operation is faster than that of data writing, so one is used for the Flush process and the other is used for data writing. Otherwise, the system should limit the speed of data writing to maintain stability. 2. the size of per MemTable. This parameter determines the threshold value for the MemTable in memory to be flushed into disk. When the system load increases, the parameter should be set smaller so that the data in memory can be flushed into disk as soon as possible. 3. the threshold size of TsFile. This parameter determines the speed of the tsfile seal, and then determines the maximum size of metadata information maintained in memory. When the system load increases, the parameter should be smaller to seal the file as soon as possible, release the memory occupied by the corresponding metadata information as soon as possible. When the system adds or deletes storage groups, adds or deletes timeseries, it will evaluate the load of the system and adjust the above three parameters dynamically. When the system load is too large, it will reject the corresponding operation. At this time, the system needs to increase the memory to meet the demand. Best Regards, ————————————————— Tianan Li School of Software, Tsinghua University > -----原始邮件----- > 发件人: "Jialin Qiao" <qj...@mails.tsinghua.edu.cn> > 发送时间: 2019-07-08 21:27:18 (星期一) > 收件人: dev@iotdb.apache.org > 抄送: > 主题: Re: Re: Start the release train > > > Hi, > > Though the current version is stable, it introduces some new parameters > (e.g., the number of memtables in the pool) which users should set carefully. > > I think we can make these parameters dynamically adapting the workload. It > does not need to change many codes (but need to design carefully). > > If it is no hurry, I'd like to finish this function before we start the > release process. > > Best, > -- > Jialin Qiao > School of Software, Tsinghua University > > 乔嘉林 > 清华大学 软件学院 > > > -----原始邮件----- > > 发件人: "Julian Feinauer" <j.feina...@pragmaticminds.de> > > 发送时间: 2019-07-08 21:21:42 (星期一) > > 收件人: "dev@iotdb.apache.org" <dev@iotdb.apache.org> > > 抄送: > > 主题: Re: Start the release train > > > > Just a short notice we should call it `develop` not `dev`. > > At least my opiniton : ) > > > > Am 08.07.19, 15:19 schrieb "Xiangdong Huang" <saint...@gmail.com>: > > > > Hi all, > > > > Does anyone has opinions for putting the releasing process at the top of > > the agendas? > > > > If there is no more opinions, I will start the first step: open a vote > > for > > moving the `master` branch into the `dev`. > > > > Best, > > ----------------------------------- > > Xiangdong Huang > > School of Software, Tsinghua University > > > > 黄向东 > > 清华大学 软件学院 > > > > > > >
