morningman opened a new issue #1995: Limit the memory consumption of Broker Scan Node URL: https://github.com/apache/incubator-doris/issues/1995 A broker scan node will start one or more broker scanner thread to read the data, generate row batches and push them to row batch queue. And the upper caller will get row batch from this queue. Currently, we only limit the number of batches in queue, which is 1024. And if a row is very large, eg, 4K, and there are 1024 rows in a batch, so the max memory consumption of a batch queue is 4GB(4K * 1024 * 1024). And if there are many broker scan node on one Backend, it may lead to system OOM. So this memory consumption need to be limited.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
