Hi all: Flushing to disk may double the memory cost: A storage group maintains a list of ChunkGroups in memory and will be flushed to disk when its occupied memory exceeding the threshold (128MB by default). In the current implementation, when starting to flush data, a ChunkGroup is encoded in memory and thereby a new byte array is kept in memory. Until all ChunkGroups have been encoded in memory, their corresponding byte arrays can be released together. Since the byte array has a comparable size with original data (0.5× to 1×), the above strategy may double the memory in the worst case. Solution: It is needed to redesign the flush strategy. In TsFile, a Page is the minimal flush unit, where a ChunkGroup contains several Chunks and a Chunk contains several pages. Once a page is encoded into a byte array, we can flush the byte array to disk and then release it. In this case, the extra memory is a page size (64KB by default) at most. This modification involves a list of cascading change, including metadata format and writing process. Memory Control Strategy: It is needed to redesign the memory Control Strategy. For example, assigning 60% memory to the writing process and 30% memory to the querying process. The writing memory includes the memory table and the flush process. As an Insert coming, if its required memory exceeds TotalMem * 0.6 - MemTableUsage - FlushUsage, the Insert will be rejected. Is the memory statistics accuracy? In current codes, the memory usage of a TSRecord Java Object, corresponding to an Insert SQL, is calculated by summating its DataPoints. e.g., "insert into root.a.b.c(timestamp,v1, v2) values(1L, true, 1.2f)", its usage is 8 + 1 + 4=13, which ignores the size of object head and others. It is needed to redesign the memory statistics accuracy carefully. Is there still the memory leak? As shown in the log of the last crash due to the out of memory exception, we find out the actual JVM memory is 18G, whereas our memory statistic module only counts 8G. Besides the inaccuracy mentioned in Q3, we doubt there are still memory leak or other potential problems. We will continue to debug it.
— 顺颂时祺 康荣 清华大学软件学院 — Best Regards, Rong Kang School of Software, Tsinghua University
