[
https://issues.apache.org/jira/browse/IOTDB-87?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jialin Qiao closed IOTDB-87.
----------------------------
Fix Version/s: 0.10.0-SNAPSHOT
Resolution: Fixed
> Improve Oveflow File Reader to save memory
> ------------------------------------------
>
> Key: IOTDB-87
> URL: https://issues.apache.org/jira/browse/IOTDB-87
> Project: Apache IoTDB
> Issue Type: Improvement
> Reporter: xiangdong Huang
> Priority: Major
> Labels: discussion, memory-control
> Fix For: 0.10.0-SNAPSHOT
>
>
> Hi, after reading source code of
> `SeriesReaderFactory.createUnSeqMergeReader`, I think the implement may take
> too much memory, and there is an un-useful read stream being opened.
> In this function, when you call `chunkLoader.getChunk(chunkMetaData)`, a
> complete Chunk (with its raw data) will be loaded into the memory.
> I think (but I am not sure whether it is good), we can only store
> ChunkMetaData in memory (and use a concise format in memory, which means just
> leaving useful info) and read at most one page for each Chunk ( if a Chunk's
> time is less than other chunks, we do not to read it), it will be fine..
> Suppose a page size is 64K, and in the worst case, each Chunk only contains
> one page, then if we have 1TB overflow data, there will be 4 million
> ChunkMetadata, and if we keep them in memory concisely (seems just the start
> time and the offset in the file are needed), the memory space will be less
> than 4*(8+8+8)MB=96MB. (In this case, the total memory cost of the first
> page of each chunk is about 64K*4M=256GB.... so we can not keep all of them
> in memory...).
> Does anyone have good idea? ( [~kakayu] )
> By the way, in the class of `EngineChunkReader`, the file input stream
> (TsFileSequenceReader) is meaningless, so we do not need to keep it as a
> field.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
