WangChao created IOTDB-586:
------------------------------
Summary: [Distributed] Eliminate some timeseries schemas When
memory is limited
Key: IOTDB-586
URL: https://issues.apache.org/jira/browse/IOTDB-586
Project: Apache IoTDB
Issue Type: Improvement
Components: Core/Engine
Reporter: WangChao
Some businesses have 400 million time series. It will be very memory consuming
to stored all of them. We need to eliminate some unusual time series schemas.
The possible way is,
1. The time sequence pulled from the remote is stored in a new MTREE. The
retrieval can retrieve two mtrees at the same time, and only this remote MTREE
will be eliminated
2. All schemas exist in one MTREE. Scan the whole MTREE to find the remote
schema for elimination
There are also two ways to elimination:
1. Direct discard
2. Save to local file
When does it need to be eliminated?
1. Limit the number of time series of an MTREE, exceeding that of elimination
2. Limit memory usage of MTREE (for example, 500MB), which will be eliminated
if exceeded
I tend to maintain one MTREE, discard it directly, and limit the number of time
series of MTREE.
The advantages are:
1. Under normal circumstances, it takes less time to read metadata from the
remote memory than from the disk randomly. If it is loaded from the disk, it
means that the memory is full, and it needs to eliminate other schemas when
reading from the disk, which is troublesome
2. Query performance may be poor when maintain two MTree
3. It is easy to maintain the number of time series
Please leave your opinions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
