xiangdong Huang created IOTDB-112:
-------------------------------------
Summary: Avoid long tail insertion which is caused by synchronized
close-bufferwrite
Key: IOTDB-112
URL: https://issues.apache.org/jira/browse/IOTDB-112
Project: Apache IoTDB
Issue Type: Improvement
Reporter: xiangdong Huang
In our test, IoTDB has a good insertion performance, and the average latency
can be ~200 ms in a given workload and hardware.
However, when we draw the histogram of the latency, we find that 97.5%
latencies are less than 200 ms, while 2.7% latencies are greater. The result
shows that there are some long tail latency.
Then we find that some insertion latencies are about 30 seconds... (but the
ratio is less than 0.5%). Indeed, for each connection, a long tail insertion
appears per 1 or 2 minutes....
By reading source codes, I think it is because that in the insertion function,
`private void insertBufferWrite(FileNodeProcessor fileNodeProcessor, long
timestamp,
boolean isMonitor, TSRecord tsRecord, String deviceId)`,
if the corresponding TsFile is too large, the function is blocked until the
memtable is flushed on disk and the TsFile is sealed (we call it as closing a
TsFile). The latencies of the long tail insertions are very close to the time
cost of flushing and sealing a TsFile.
So, if we set the closing function using the async mode, we can avoid the long
tail insertion.
However, there are some side effects we have to fix:
# At the same time, if a new insertion comes, then a new memtable should be
assigned, and a new unsealed TsFile is created;
# That means that there are more than 1 unsealed TsFiles if the system is
crashed before the closing function is finished. So, we have to modify the
startup process to recover these files.
Is there any other side effect that I have to pay attention to?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
