[
https://issues.apache.org/jira/browse/IOTDB-586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082375#comment-17082375
]
Xiangdong Huang commented on IOTDB-586:
---------------------------------------
Hi,
Let's re-describe the above idea.
Given a node and a schema, let's split the shame into two parts, be responded
and not be responded, or managed or not managed, or local time series and
remote time series. I will use LOCAL and REMOTE to represent the two above.
Firstly, only save the LOCAL time series on disk, and if possible, we can save
REMOTE time series on a cache file.
For LOCAL time series, we need to load all of them in memory. It is because we
need to write data that belong to these time series locally.
For REMOTE time series, we do not need to load all of them in memory. It is
because we do not write the data locally. Actually, Knowing their storage
groups is enough.
If we put LOCAL and REMOTE time series into one MTree in memory, we just need
to change the serialize method to skip REMOTE time series;
If we put LOCAL and REMOTE time series into two MTree in memory, it is also
easy to be implemented, just serialize one while ignore the other.
Now let's discuss another question, which has little relation with the cluster
mode, but may be important for single node mode: How can we do if there are too
many LOCAL time series?
IMO, there are two choices:
1. we require the memory MUST be large enough to handle all time series schema
in memory. Otherwise, we throw exception to shutdown or reject more time series
registration.
2. use some cache strategy. It is because if there are billions of time series
and the user does not want to enlarge its memory hardware, maybe the concurrent
write load is not heavy. Or, many time series are not active. Then, we can load
a part of time series in memory and exchange them when new time series are
written. (But, do not use Linkedhashmap, it jitters heavily).
> [Distributed] Eliminate some timeseries schemas When memory is limited
> ----------------------------------------------------------------------
>
> Key: IOTDB-586
> URL: https://issues.apache.org/jira/browse/IOTDB-586
> Project: Apache IoTDB
> Issue Type: Improvement
> Components: Core/Cluster
> Reporter: WangChao
> Priority: Minor
>
> Some businesses have 400 million time series. It will be very memory
> consuming to stored all of them. We need to eliminate some unusual time
> series schemas.
>
> The possible way is,
> 1. The time sequence pulled from the remote is stored in a new MTREE. The
> retrieval can retrieve two mtrees at the same time, and only this remote
> MTREE will be eliminated
> 2. All schemas exist in one MTREE. Scan the whole MTREE to find the remote
> schema for elimination
>
> There are also two ways to elimination:
> 1. Direct discard
> 2. Save to local file
>
> When does it need to be eliminated?
> 1. Limit the number of time series of an MTREE, exceeding that of elimination
> 2. Limit memory usage of MTREE (for example, 500MB), which will be eliminated
> if exceeded
>
> I tend to maintain one MTREE, discard it directly, and limit the number of
> time series of MTREE.
> The advantages are:
> 1. Under normal circumstances, it takes less time to read metadata from the
> remote memory than from the disk randomly. If it is loaded from the disk, it
> means that the memory is full, and it needs to eliminate other schemas when
> reading from the disk, which is troublesome
> 2. Query performance may be poor when maintain two MTree
> 3. It is easy to maintain the number of time series
>
> Please leave your opinions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
