ibobak commented on PR #2723: URL: https://github.com/apache/hadoop/pull/2723#issuecomment-1294738611
Colleagues, I've taken the source code from this commit https://github.com/apache/hadoop/pull/2723/commits/47f05930c2f5c576a6c25238c187bdf3409b8f23 made a jar of it, plugged it into my Spark cluster, launched a huge job with many transformations and actions, and found that there is a serious memory leak: executors consume RAM more and more (no matter that there is a limitation of 20GB, they consumed 40GB). I've made my own version of Brotli codec (also based on brotli4j) by looking at how Snappy and others are made, and it works with no memory leaks. Soon I'll post my PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
