[
https://issues.apache.org/jira/browse/OAK-4966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15608700#comment-15608700
]
Alex Parvulescu commented on OAK-4966:
--------------------------------------
bq. My general concern here are all those estimates for memory consumption of
nodes, templates and strings. They may hold for one repository but might be
completely off for another one.
this only applies for the strings cache. templates and _especially_ the node
cache has a very simple structure based on recordids and computing a max weight
based on configured size will not change from one repo to another.
bq. Slight changes in the code might greatly affect how much memory e.g. a node
needs.
agreed, but I don't see the relation to the node deduplication cache. as long
as recordids don't change this won't change, and full nodes don't come into
play.
bq. Also checking memory up front doesn't account for the case where a memory
intensive operation is started on an instance where gc is already running.
yes, this was already stated on the provided patch. the previous impl didn't do
this either and as far as I know if did ok. if this has become a goal, I agree
we can also add a polling component that can stop compaction if things get too
tight in terms of resources.
bq. What about a simpler (I think so) approach where we check the available
memory say once a second during compaction.
I would augment the existing code with such a check. the current heap needed
calculation hints at a {{3GB}} heap with default values. if the instance
doesn't have that, it doesn't even make sense to start compaction.
besides using the cache weight for compaction or not, I think there is some
value in estimating how big a cache can get based on existing data and
improving on it without getting too crazy about perfection. simply stating a
_number of items_ in a cache has not relation to how much heap it can consume
and if we're not able to estimate then how would anyone else be able to? so I
would actually keep the cache weight changes even if for display value only (as
jmx).
> Re-introduce a blocker for compaction based on available heap
> -------------------------------------------------------------
>
> Key: OAK-4966
> URL: https://issues.apache.org/jira/browse/OAK-4966
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
> Reporter: Alex Parvulescu
> Assignee: Alex Parvulescu
> Fix For: 1.6, 1.5.13
>
> Attachments: OAK-4966.patch
>
>
> As seen in a local test, running compaction on a tight heap can lead to
> OOMEs. There used to be a best effort barrier against this situation 'not
> enough heap for compaction', but we removed it with the compaction maps.
> I think it makes sense to add it again based on the max size of some of the
> caches: segment cache {{256MB}} by default [0] and some writer caches which
> can go up to {{2GB}} all combined [1] and probably others I missed.
> [0]
> https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java#L48
> [1]
> https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/WriterCacheManager.java#L50
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
