kaijianding commented on a change in pull request #6699: release mmap
immediately after merge indexes
URL: https://github.com/apache/incubator-druid/pull/6699#discussion_r239678427
##########
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java
##########
@@ -716,8 +717,20 @@ private DataSegment mergeAndPush(final SegmentIdentifier
identifier, final Sink
closer.register(segmentAndCloseable.rhs);
}
- mergedFile = indexMerger.mergeQueryableIndex(
- indexes,
Review comment:
The QueryableIndexes are always new objects loaded from file, and closed
when IndexMerger.mergeSegmentFiles() is done. This behavior frees the mmap
usage increased during the merge process.
Currently we use indexMerger.mergeQueryableIndex(indexes) in
RealtimePlumber, the indexes are not closed to explicitly release the increased
mmap because these indexes are still used by queries. Then the mmap usage
always increases until abandonSegment() is called which only happens when
handoff succeeds.
If handoff is slow or coordinator is not working properly, the mmap usage
will keep increasing, this is a problem.
This PR separates the QueryableIndex used by query and used by merge
process, then we can close the QueryableIndexes used by merge process to
release mmap, and leave the QueryableIndexes used by query untouched.
This PR is verified in my product environment, it indeed controls the mmap
usage
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
