kaijianding commented on a change in pull request #6699: release mmap
immediately after merge indexes
URL: https://github.com/apache/incubator-druid/pull/6699#discussion_r239891065
##########
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java
##########
@@ -716,8 +717,20 @@ private DataSegment mergeAndPush(final SegmentIdentifier
identifier, final Sink
closer.register(segmentAndCloseable.rhs);
}
- mergedFile = indexMerger.mergeQueryableIndex(
- indexes,
Review comment:
Will address these comments.
The mechanism is like this:
1. mmap usage will increase after indexes merged. mmap can only be released
when close() called on indexes.
2. currently close() on the indexes are not called until abandonSegment()
which can be delayed for a very long time if handoff is slow. Before handoff
succeeds, the mmap will keep increasing.
3. to avoid this increasement, I create new QueryableIndexes object and
close them after each merge success/fail, then the increased mmap during the
merge process is released.
4. I think this magic works because even a flle is mmaped multiple times,
the mmap usage is calculated separately each mmap action, and can be un-mmaped
separately each un-mmap action. In this case, close() on QueryableIndex is the
un-mmap action.
Wish I explain the mechanism clearly. Though this PR is very simple, it
indeed help on the mmap usage.
There is an alternative solution to recycle mmap: load the mergedFile as
QueryableIndex and swap all hydrants' small QueryableIndexes with this single
big QueryableIndex to the Sink. and close all hydrants' small QueryableIndexes
and delete all hydrant segments to release mmap (the mmap usage is already
increased after merge process done)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
