clintropolis commented on a change in pull request #6699: release mmap
immediately after merge indexes
URL: https://github.com/apache/incubator-druid/pull/6699#discussion_r240990232
##########
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java
##########
@@ -716,8 +717,20 @@ private DataSegment mergeAndPush(final SegmentIdentifier
identifier, final Sink
closer.register(segmentAndCloseable.rhs);
}
- mergedFile = indexMerger.mergeQueryableIndex(
- indexes,
Review comment:
>Basically correct except for the last part.
The merged segment is not loaded as index after merge process done, so there
is no extra mmap used after the merge process is finished, therefore no
footprint left.
Which part is not correct? Sorry if there was confusion, the hypothetical
about what would happen if loading the merged segment after merge was in
response to an earlier suggestion, I didn't mean to imply that it is currently
loaded. But the hydrant segments that are not closed are loaded still, and
querying them after merge will still cause physical memory to be occupied with
the page cache.
My main point was that even with this strategy, and regardless of whether or
not we keep using the intermediary hydrant segments or swap to the merged
segment, any queries _after_ merge will still require physical memory for the
cache of reading data out of those memory mapped segments. If the complete set
of data is queried (all rows/all columns) then its maximum footprint in terms
of physical memory from mmap is the size on disk, if the OS has the free space
to give it. In the case of continuing to serve the intermediary segments (as in
this PR) this is potentially the exact same footprint as the merge required in
the first place, so YARN could still kill it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
