kaijianding commented on a change in pull request #6699: release mmap
immediately after merge indexes
URL: https://github.com/apache/incubator-druid/pull/6699#discussion_r240867548
##########
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java
##########
@@ -716,8 +717,20 @@ private DataSegment mergeAndPush(final SegmentIdentifier
identifier, final Sink
closer.register(segmentAndCloseable.rhs);
}
- mergedFile = indexMerger.mergeQueryableIndex(
- indexes,
Review comment:
Basically correct except for the last part.
The merged segment is not loaded as index after merge process done, so there
is no extra mmap used after the merge process is finished, therefore no
footprint left.
And our YARN environment is not maintained by me, so it's not my call to
make yarn configuration changes, and I think this behavior is good for resource
isolation.
BUT IF we do load the merged segment as index and replace the hydrant
segments in the sink, because the new merged segment is not read thus no mmap
used just after the replacement, the hydrant segments definitely use more mmap
since there maybe already be queried.
And another benefit of the replacement is that we can reduce the the file
number to reduce the inode number. In my case, I'm working one a new scenario,
I have lots of hydrant segment and better to reduce inode number after merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
