kaijianding commented on a change in pull request #6699: release mmap
immediately after merge indexes
URL: https://github.com/apache/incubator-druid/pull/6699#discussion_r241253765
##########
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java
##########
@@ -716,8 +717,20 @@ private DataSegment mergeAndPush(final SegmentIdentifier
identifier, final Sink
closer.register(segmentAndCloseable.rhs);
}
- mergedFile = indexMerger.mergeQueryableIndex(
- indexes,
Review comment:
We are standing at the same page now, your understanding is totally correct.
The worst case, the complete set of data is queried (all rows/all columns),
the exact same footprint is as the merge required in the first place.
But usually(as I noticed in my production environment), very little mmap is
increased after queries, many columns are ingested but not queried at all. This
part of mmap is not a problem. And if we do the final swap after merge, this
part of mmap will decrease because of the new merged segment has no mmap
footprint. In most cases, user is like to query the latest data, when handoff
is slow, it's better to do the final swap to reduce this part of mmap for the
earlier hydrant segments.
Back to this PR, it releases mmap after merge is done, as a result, the mmap
usage is under control, it won't grow to a huge number when handoff is slow to
let the process be killed by yarn.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
