Hi jialin, I submit some modifications for:
* add the overflow data folder location setting in the iotdb-engine.properties; * let Directories.java to manage the above folder. If you need to refactor the overflow when you solving the long tail issue, you can apply the patch from [1] first to simplify your work. [1] https://issues.apache.org/jira/secure/attachment/12972547/overflow-folder.patch Best, ----------------------------------- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Xiangdong Huang <[email protected]> 于2019年6月22日周六 下午3:19写道： > If you change the process like this, i.e., there are more than one > unsealed TsFiles for each storage group, then you have to modify the WAL > module.. Because current WAL module only recognizes the last unsealed > TsFile.. > > By the way, "sealed" is better than "closed", I think.. A sealed file > means the file which has the magic string at the head and the tail. > > Best, > ----------------------------------- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > Jialin Qiao <[email protected]> 于2019年6月22日周六 下午2:54写道： > >> >> Hi, I am solving the long-tail latency problem. >> >> There are some cases (blocking points) that blocking the insertion. For a >> better understanding of this problem, I first introduce the writing process >> of IoTDB: >> >> IoTDB maintains several independent engines (storage group) that supports >> read and write. In the following, we focus on one engine. A engine >> maintains several closed data files and one unclosed data file that >> receives appended data. In memory, there is only one working memtable (m1) >> that receives writes. There is also another memtable (m2) that will take >> place m1 when m1 is full and being flushed. >> >> When a data item is inserted: >> >> (1)We insert it into the working memtable. >> (2)We check the size of the memtable. If it reaches a threshold, we >> submit a flush task “after the previous flush task is finished” and switch >> the two memtables. >> (3)We check the size of the unclosed file. If it reaches a threshold, we >> close it “after the previous flush task is finished”. >> >> In the above steps, all the "after the previous flush task is finished" >> will block the insertion process. One solution is to make all flush and >> close task asynchronous. Some questions need to carefully considered: >> >> (1) Many memtables may be flushed concurrently to an unclosed file. How >> to guarantee the order of serialization? >> (2) Once a close task is submitted, a new unclosed file will be created >> and receives appended data. So there will exists many unclosed files. How >> the query and compaction process will be impacted? >> >> Thanks, >> >> Jialin Qiao >> School of Software, Tsinghua University >> >> 乔嘉林 >> 清华大学 软件学院 >> >> > -----原始邮件----- >> > 发件人: "Xiangdong Huang" <[email protected]> >> > 发送时间: 2019-06-04 23:08:34 (星期二) >> > 收件人: [email protected], "江天" <[email protected]> >> > 抄送: >> > 主题: Re: [jira] [Created] (IOTDB-112) Avoid long tail insertion which is >> caused by synchronized close-bufferwrite >> > >> > I attached the histogram of the latency in the JIRA. >> > >> > The x-axis is the latency while the y-axis is the cumulative >> distribution. >> > We can see that about 30% insertion can be finished in 20ms, and 60% >> > insertion can be finished in 40ms even though the IoTDB instance is >> serving >> > for a heavy workload... So, eliminating the long tail insertion can make >> > the average latency far better. >> > >> > If someone is working on the refactor_overflow or refactor_bufferwrite, >> > please pay attention to the code branch for this issue. >> > >> > Best, >> > >> > ----------------------------------- >> > Xiangdong Huang >> > School of Software, Tsinghua University >> > >> > 黄向东 >> > 清华大学 软件学院 >> > >> > >> > xiangdong Huang (JIRA) <[email protected]> 于2019年6月4日周二 下午11:00写道： >> > >> > > xiangdong Huang created IOTDB-112: >> > > ------------------------------------- >> > > >> > > Summary: Avoid long tail insertion which is caused by >> > > synchronized close-bufferwrite >> > > Key: IOTDB-112 >> > > URL: https://issues.apache.org/jira/browse/IOTDB-112 >> > > Project: Apache IoTDB >> > > Issue Type: Improvement >> > > Reporter: xiangdong Huang >> > > >> > > >> > > In our test, IoTDB has a good insertion performance, and the average >> > > latency can be ~200 ms in a given workload and hardware. >> > > >> > > However, when we draw the histogram of the latency, we find that 97.5% >> > > latencies are less than 200 ms, while 2.7% latencies are greater. The >> > > result shows that there are some long tail latency. >> > > >> > > Then we find that some insertion latencies are about 30 seconds... >> (but >> > > the ratio is less than 0.5%). Indeed, for each connection, a long tail >> > > insertion appears per 1 or 2 minutes.... >> > > >> > > By reading source codes, I think it is because that in the insertion >> > > function, >> > > >> > > `private void insertBufferWrite(FileNodeProcessor fileNodeProcessor, >> long >> > > timestamp, >> > > boolean isMonitor, TSRecord tsRecord, String deviceId)`, >> > > >> > > if the corresponding TsFile is too large, the function is blocked >> until >> > > the memtable is flushed on disk and the TsFile is sealed (we call it >> as >> > > closing a TsFile). The latencies of the long tail insertions are very >> close >> > > to the time cost of flushing and sealing a TsFile. >> > > >> > > So, if we set the closing function using the async mode, we can avoid >> the >> > > long tail insertion. >> > > >> > > However, there are some side effects we have to fix: >> > > # At the same time, if a new insertion comes, then a new memtable >> should >> > > be assigned, and a new unsealed TsFile is created; >> > > # That means that there are more than 1 unsealed TsFiles if the >> system is >> > > crashed before the closing function is finished. So, we have to >> modify the >> > > startup process to recover these files. >> > > >> > > Is there any other side effect that I have to pay attention to? >> > > >> > > >> > > >> > > -- >> > > This message was sent by Atlassian JIRA >> > > (v7.6.3#76005) >> > > >> >
