kaijianding edited a comment on issue #6699: release mmap immediately after merge indexes URL: https://github.com/apache/incubator-druid/pull/6699#issuecomment-444748322 @clintropolis here is how things happen. 1. We run realtime jobs on Yarn, and yarn tracks the mem usage including heap, direct memory and native memory which in druid case is mmap. 2. yarn will kill process whose mem usage is much more than it applyed. 3. after druid merges hydrant segments into merged segment, the hydrant segments are mmaped and and the mmap usage is added about the total file size of the hydrant segments which usually hundreds or thousands MB, AND the added mmap is not released due to the QueryableIndex cann't be closed because the index is still needed by queries. 4. when handoff is slow, the mmap usage will grow to a huge number when the merge process happens many many times. finally Yarn will kill this druid realtime jvm. 5. also, there are many side-effect objects added to heap after merge success, finally OOM happens. Direct memory is not a problem here, direct memory is released after each merge success/fail.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
