[
https://issues.apache.org/jira/browse/IOTDB-586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083037#comment-17083037
]
Tian Jiang commented on IOTDB-586:
----------------------------------
The proposed idea is overall acceptable. But I have some questions:
Currently, we pull remote schemas NOT TO respond to metadata queries but to
respond to requests like insertion (we need the encoding and data type of a
timeseries). When performing a metadata query, we only need to queries several
nodes and combine the results, so there is no need to store such timeseries
locally and we will not get their schema (only timeseries names) either.
Therefore, I doubt if it is necessary to still use a tree structure. As we only
need the data types and encodings of the timeseries for writes, I think the
simplest way might be two maps: Map<String, TsDataType> and Map<String,
Encoding> (or even simpler, just one map, Map<String, MeasurementSchema>).
Needless to say, the key is the full name of the timeseries. Using structures
like a map, it would be much easier for you to implement advanced cache
strategies like LRU, round-robin.
> [Distributed] Eliminate some timeseries schemas When memory is limited
> ----------------------------------------------------------------------
>
> Key: IOTDB-586
> URL: https://issues.apache.org/jira/browse/IOTDB-586
> Project: Apache IoTDB
> Issue Type: Improvement
> Components: Core/Cluster
> Reporter: WangChao
> Priority: Minor
>
> Some businesses have 400 million time series. It will be very memory
> consuming to stored all of them. We need to eliminate some unusual time
> series schemas.
>
> The possible way is,
> 1. The time sequence pulled from the remote is stored in a new MTREE. The
> retrieval can retrieve two mtrees at the same time, and only this remote
> MTREE will be eliminated
> 2. All schemas exist in one MTREE. Scan the whole MTREE to find the remote
> schema for elimination
>
> There are also two ways to elimination:
> 1. Direct discard
> 2. Save to local file
>
> When does it need to be eliminated?
> 1. Limit the number of time series of an MTREE, exceeding that of elimination
> 2. Limit memory usage of MTREE (for example, 500MB), which will be eliminated
> if exceeded
>
> I tend to maintain one MTREE, discard it directly, and limit the number of
> time series of MTREE.
> The advantages are:
> 1. Under normal circumstances, it takes less time to read metadata from the
> remote memory than from the disk randomly. If it is loaded from the disk, it
> means that the memory is full, and it needs to eliminate other schemas when
> reading from the disk, which is troublesome
> 2. Query performance may be poor when maintain two MTree
> 3. It is easy to maintain the number of time series
>
> Please leave your opinions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
