WingsGo opened a new issue #3966: URL: https://github.com/apache/incubator-doris/issues/3966
In production enviorment, the data loading is 7*24 with lots of txns, but we found that the TransactionMgr costs too much memeory, which will leads OOM.If we want to improve the concurrent of stream load, we must optimize the memory usage in order to support frequent data load.There are some way such as clean up useless data after txn finished #3130, reduce the interval of clean txn, use ZGC to recycle memory in a short time....But It is not enough, I found FE will OOM accidental cause by TransactionMgr usage. There are two way to reduce the memory usage, one is change the data struct in DatabaseTransactionMgr to save memeory, the other is scattered the Txn to other follower FE to share the memory usage, I will invetigate the feasibility of two solutions, any advice or solution is welcome. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
