morningman commented on issue #2016: [Proposal] Limit the memory usage of Compaction URL: https://github.com/apache/incubator-doris/issues/2016#issuecomment-547930873 After the research, most of the memory overhead was concentrated on opening the segments(RowBlock) and creating the Readers(StreamBuffer). And this part of the memory overhead is strongly related to the number of columns of data and the column type. And we can't accurately estimate the size of a row of data in memory at this time, so we can't estimate the memory required for a compaction. Compaction within a rowset is a way to reduce the memory, but the root cause still exists, which is: How many segments should we selected for a compaction by given certain amount of memory. And this requires the statistic of size of a row of data in memory. I will leave this Issue opened and try to resolve it sometimes later.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
