morningman opened a new issue #1777: Limit the mem usage when loading with HLL column URL: https://github.com/apache/incubator-doris/issues/1777 When loading tables containing HLL columns, the aggregation function of HLL columns should have arena for memory allocation. Otherwise, the size of memtable can not be controlled, resulting in memtable being too large. But one problem is that HLL columns need to use 16KB storage, resulting in 100 MB of default memtable quickly reaching the threshold and brushing the disk. So one batch of load can produce a lot of segments. So first we limit memory to ensure that the system will not be OOM, and then we need to optimize the memory usage of HLL.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org For additional commands, e-mail: dev-h...@doris.apache.org
