[
https://issues.apache.org/jira/browse/OAK-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13582526#comment-13582526
]
Jukka Zitting commented on OAK-643:
-----------------------------------
bq. Jukka is probably refering to the fact that ...
Indeed. The code iterates through all the child nodes, which at these sizes end
up quickly filling up the 10k slots of the cache in KernelNodeStore. Many of
those entries are fairly large, since each save() call in the code results in a
new revision and thus also a new version of the flat parent node, each of which
contains a map of up to 10k child name->path mappings. With an average size of
7-8kB per cached KernelNodeState instance, we get to the mentioned 74MB.
As mentioned earlier, I think the KernelNodeState caching, at least as
currently implemented is in many ways troublesome. My suggestion is to do
something like what I'm now drafting in the SegmentMK (where optimizing memory
use is one of the key ideas), but I'd love to see also alternative ideas (or
better yet, code).
> Very high memory usage with 6000 child nodes
> --------------------------------------------
>
> Key: OAK-643
> URL: https://issues.apache.org/jira/browse/OAK-643
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core
> Reporter: Thomas Mueller
> Priority: Minor
>
> The following test case gets slower and slower the more child nodes are added
> to a node, until (I think) it eventually runs out of memory. I have analyzed
> a heap dump, and the problem seems to be the cache in oak-core:
> {code}
> @Test
> public void testManyChildren() throws RepositoryException {
> Session session = getAdminSession();
> Node root = session.getRootNode().addNode("testRoot");
> session.save();
> int count = 100;
> for (int j = 0; j < 10; j++) {
> Node test = root.
> addNode("test" + j, "nt:folder");
> session.save();
> long time = System.currentTimeMillis();
> for (int i = 0; i < count; i++) {
> test.addNode("child" + i, "nt:folder");
> if ((i % 100) == 0) {
> session.save();
> }
> }
> System.out.println(
> count + " nodes in " +
> (System.currentTimeMillis() - time) + " ms");
> count *= 2;
> }
> }
> {code}
> Output when using the MicroKernelImpl (persisted to disk):
> {code}
> 100 nodes in 757 ms
> 200 nodes in 538 ms
> 400 nodes in 957 ms
> 800 nodes in 1630 ms
> 1600 nodes in 3245 ms
> 3200 nodes in 5389 ms
> 6400 nodes in 47581 ms
> 12800 nodes in 377216 ms
> java.lang.OutOfMemoryError: Java heap space
> {code}
> Problem Suspect 1:
> 74 MB (57.72%) in com.google.common.cache.LocalCache$Segment[]
> in a thread, see below
> Problem Suspect 2:
> 39 MB (30.76%) in org.apache.jackrabbit.mk.store.DefaultRevisionStore
> (this might be the regular node cache)
> Problem Suspect 3:
> 13 MB (10.59%) in org.h2.store.PageStore
> (16 MB is the default cache size for the H2 database,
> so this isn't a leak)
> The thread for suspect 1 is:
> {code}
> at
> java.util.LinkedHashMap.createEntry(ILjava/lang/Object;Ljava/lang/Object;I)V
> (LinkedHashMap.java:424)
> at
> java.util.LinkedHashMap.addEntry(ILjava/lang/Object;Ljava/lang/Object;I)V
> (LinkedHashMap.java:406)
> at
> java.util.HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
> (HashMap.java:385)
> at org.apache.jackrabbit.oak.kernel.KernelNodeState.init()V
> (KernelNodeState.java:142)
> at
> org.apache.jackrabbit.oak.kernel.KernelNodeState.equals(Ljava/lang/Object;)Z
> (KernelNodeState.java:310)
> at
> org.apache.jackrabbit.oak.spi.state.AbstractNodeState.compareAgainstBaseState(Lorg/apache/jackrabbit/oak/spi/state/NodeState;Lorg/apache/jackrabbit/oak/spi/state/NodeStateDiff;)V
> (AbstractNodeState.java:149)
> at
> org.apache.jackrabbit.oak.kernel.KernelNodeState.compareAgainstBaseState(Lorg/apache/jackrabbit/oak/spi/state/NodeState;Lorg/apache/jackrabbit/oak/spi/state/NodeStateDiff;)V
> (KernelNodeState.java:286)
> at
> org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.merge(Lorg/apache/jackrabbit/oak/spi/state/NodeState;Lorg/apache/jackrabbit/oak/spi/state/NodeState;Lorg/apache/jackrabbit/oak/spi/state/NodeBuilder;Lorg/apache/jackrabbit/oak/spi/commit/ConflictHandler;)Lorg/apache/jackrabbit/oak/spi/state/NodeState;
> (MergingNodeStateDiff.java:69)
> at
> org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.childNodeChanged(Ljava/lang/String;Lorg/apache/jackrabbit/oak/spi/state/NodeState;Lorg/apache/jackrabbit/oak/spi/state/NodeState;)V
> (MergingNodeStateDiff.java:90)
> at
> org.apache.jackrabbit.oak.spi.state.AbstractNodeState.compareAgainstBaseState(Lorg/apache/jackrabbit/oak/spi/state/NodeState;Lorg/apache/jackrabbit/oak/spi/state/NodeStateDiff;)V
> (AbstractNodeState.java:150)
> at
> org.apache.jackrabbit.oak.kernel.KernelNodeState.compareAgainstBaseState(Lorg/apache/jackrabbit/oak/spi/state/NodeState;Lorg/apache/jackrabbit/oak/spi/state/NodeStateDiff;)V
> (KernelNodeState.java:286)
> at
> org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.merge(Lorg/apache/jackrabbit/oak/spi/state/NodeState;Lorg/apache/jackrabbit/oak/spi/state/NodeState;Lorg/apache/jackrabbit/oak/spi/state/NodeBuilder;Lorg/apache/jackrabbit/oak/spi/commit/ConflictHandler;)Lorg/apache/jackrabbit/oak/spi/state/NodeState;
> (MergingNodeStateDiff.java:69)
> at
> org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.merge(Lorg/apache/jackrabbit/oak/spi/state/NodeState;Lorg/apache/jackrabbit/oak/spi/state/NodeState;Lorg/apache/jackrabbit/oak/spi/commit/ConflictHandler;)Lorg/apache/jackrabbit/oak/spi/state/NodeState;
> (MergingNodeStateDiff.java:64)
> at
> org.apache.jackrabbit.oak.plugins.commit.ConflictHook.processCommit(Lorg/apache/jackrabbit/oak/spi/state/NodeState;Lorg/apache/jackrabbit/oak/spi/state/NodeState;)Lorg/apache/jackrabbit/oak/spi/state/NodeState;
> (ConflictHook.java:34)
> at
> org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(Lorg/apache/jackrabbit/oak/spi/state/NodeState;Lorg/apache/jackrabbit/oak/spi/state/NodeState;)Lorg/apache/jackrabbit/oak/spi/state/NodeState;
> (CompositeHook.java:59)
> at
> org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(Lorg/apache/jackrabbit/oak/spi/state/NodeState;Lorg/apache/jackrabbit/oak/spi/state/NodeState;)Lorg/apache/jackrabbit/oak/spi/state/NodeState;
> (CompositeHook.java:59)
> at
> org.apache.jackrabbit.oak.kernel.KernelNodeStoreBranch.merge(Lorg/apache/jackrabbit/oak/spi/commit/CommitHook;)Lorg/apache/jackrabbit/oak/spi/state/NodeState;
> (KernelNodeStoreBranch.java:128)
> at
> org.apache.jackrabbit.oak.core.RootImpl$2.run()Lorg/apache/jackrabbit/oak/api/CommitFailedException;
> (RootImpl.java:257)
> at org.apache.jackrabbit.oak.core.RootImpl$2.run()Ljava/lang/Object;
> (RootImpl.java:253)
> at
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;Ljava/security/AccessControlContext;)Ljava/lang/Object;
> (Native Method)
> at
> javax.security.auth.Subject.doAs(Ljavax/security/auth/Subject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (Subject.java:337)
> at org.apache.jackrabbit.oak.core.RootImpl.commit()V (RootImpl.java:252)
> at org.apache.jackrabbit.oak.jcr.SessionDelegate.save()V
> (SessionDelegate.java:255)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
