[
https://issues.apache.org/jira/browse/JCRVLT-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933704#comment-16933704
]
Mark Adamcin commented on JCRVLT-374:
-------------------------------------
[~joerghoh] suspect this is due to the default vaultfs config.xml
(org.apache.jackrabbit.vault-3.2.4.jar!/org/apache/jackrabbit/vault/fs/config/defaultConfig-1.1.xml)
not defining full coverage aggregates for children of the jcr:system tree.
{code:java}
<!--
Defines an aggregate that defines full coverage for certain node
types that cannot be covered by the default aggregator.
-->
<aggregate type="full" title="Full Coverage Aggregate">
<matches>
<include nodeType="rep:AccessControl" respectSupertype="true" />
<include nodeType="rep:Policy" respectSupertype="true" />
<include nodeType="cq:Widget" respectSupertype="true" />
<include nodeType="cq:WidgetCollection" respectSupertype="true" />
<include nodeType="cq:EditConfig" respectSupertype="true" />
<include nodeType="cq:WorkflowModel" respectSupertype="true" />
<include nodeType="vlt:FullCoverage" respectSupertype="true" />
<include nodeType="mix:language" respectSupertype="true" />
<include nodeType="sling:OsgiConfig" respectSupertype="true" />
</matches>
</aggregate>
{code}
I suspect if we added some of these child node types to those includes we would
see the package manager break them out into their own files:
{code:java}
[rep:system]
orderable
+ jcr:versionStorage (rep:versionStorage) = rep:versionStorage mandatory
protected ABORT
+ jcr:nodeTypes (rep:nodeTypes) = rep:nodeTypes mandatory protected ABORT
// @since 2.0
+ jcr:activities (rep:Activities) = rep:Activities mandatory protected ABORT
// @since 2.0
+ jcr:configurations (rep:Configurations) = rep:Configurations protected ABORT
+ * (nt:base) = nt:base IGNORE
// @since oak 1.0
+ rep:privileges (rep:Privileges) = rep:Privileges protected ABORT
{code}
And to break out directories within jcr:versionStorage, we may include
`nt:versionHistory` as well.
> assembling a content-package consumes much memory
> -------------------------------------------------
>
> Key: JCRVLT-374
> URL: https://issues.apache.org/jira/browse/JCRVLT-374
> Project: Jackrabbit FileVault
> Issue Type: Improvement
> Components: Packaging
> Affects Versions: 3.2.8
> Reporter: Jörg Hoh
> Priority: Major
> Attachments: filevault.log.gz
>
>
> I came across a situation that packaging a huge subtree
> (/jcr:system/jcr:versionStorage) (bad idea, I know) caused a huge spike in
> memory usage, which caused lots of FullGCs (due to AllocationFailures).
> I have several stacktraces from that time, which all look very similar to
> this one:
> {noformat}
> qtp1597826410-38130" prio=5 tid=0x94f2 nid=0xffffffff runnable
> java.lang.Thread.State: RUNNABLE
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeBuilder.createChildBuilder(SegmentNodeBuilder.java:147)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.getChildNode(MemoryNodeBuilder.java:330)
> at
> org.apache.jackrabbit.oak.core.SecureNodeBuilder.<init>(SecureNodeBuilder.java:110)
> at
> org.apache.jackrabbit.oak.core.SecureNodeBuilder.getChildNode(SecureNodeBuilder.java:327)
> at
> org.apache.jackrabbit.oak.core.MutableTree.getTree(MutableTree.java:288)
> at
> org.apache.jackrabbit.oak.core.MutableRoot.getTree(MutableRoot.java:220)
> at
> org.apache.jackrabbit.oak.core.MutableRoot.getTree(MutableRoot.java:69)
> at
> org.apache.jackrabbit.oak.jcr.session.WorkspaceImpl$1.getTypes(WorkspaceImpl.java:85)
> at
> org.apache.jackrabbit.oak.plugins.nodetype.ReadOnlyNodeTypeManager.isNodeType(ReadOnlyNodeTypeManager.java:293)
> at
> org.apache.jackrabbit.oak.jcr.session.NodeImpl$24.perform(NodeImpl.java:931)
> at
> org.apache.jackrabbit.oak.jcr.session.NodeImpl$24.perform(NodeImpl.java:926)
> at
> org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.perform(SessionDelegate.java:207)
> at
> org.apache.jackrabbit.oak.jcr.session.ItemImpl.perform(ItemImpl.java:112)
> at
> org.apache.jackrabbit.oak.jcr.session.NodeImpl.isNodeType(NodeImpl.java:926)
> at
> org.apache.jackrabbit.vault.fs.impl.aggregator.FileAggregator.matches(FileAggregator.java:66)
> at
> org.apache.jackrabbit.vault.fs.impl.AggregatorProvider.getAggregator(AggregatorProvider.java:68)
> at
> org.apache.jackrabbit.vault.fs.impl.AggregateManagerImpl.getAggregator(AggregateManagerImpl.java:455)
> at
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl.prepare(AggregateImpl.java:720)
> at
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl.prepare(AggregateImpl.java:733)
> at
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl.prepare(AggregateImpl.java:733)
> at
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl.prepare(AggregateImpl.java:733)
> at
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl.prepare(AggregateImpl.java:733)
> at
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl.prepare(AggregateImpl.java:733)
> at
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl.collect(AggregateImpl.java:684)
> at
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl.prepare(AggregateImpl.java:747)
> at
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl.load(AggregateImpl.java:657)
> at
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl.getArtifacts(AggregateImpl.java:259)
> at
> org.apache.jackrabbit.vault.fs.impl.VaultFileImpl.<init>(VaultFileImpl.java:101)
> at
> org.apache.jackrabbit.vault.fs.impl.VaultFileSystemImpl.<init>(VaultFileSystemImpl.java:120)
> at org.apache.jackrabbit.vault.fs.Mounter.mount(Mounter.java:64)
> at
> org.apache.jackrabbit.vault.packaging.impl.PackageManagerImpl.assemble(PackageManagerImpl.java:141)
> at
> org.apache.jackrabbit.vault.packaging.impl.PackageManagerImpl.assemble(PackageManagerImpl.java:102)
> at
> org.apache.jackrabbit.vault.packaging.impl.JcrPackageManagerImpl.assemble(JcrPackageManagerImpl.java:358)
> at
> org.apache.jackrabbit.vault.packaging.impl.JcrPackageManagerImpl.assemble(JcrPackageManagerImpl.java:324)
> {noformat}
> It seems to me that vault is traversing the complete tree and also storing
> some information of every traversed node in memory.
> For validation I enabled trace logging for {{org.apache.jackrabbit.vault.fs}}
> and tried to reproduce locally to package the complete
> {{/jcr:system/jcr:versionStorage}} in a package.
> {noformat}
> [...]
> 19.09.2019 20:06:08.792 *TRACE* [qtp681943839-1771]
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl Create Aggregate /jcr:system
> 19.09.2019 20:06:08.792 *TRACE* [qtp681943839-1771]
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl Collecting /jcr:system
> 19.09.2019 20:06:08.792 *TRACE* [qtp681943839-1771]
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl descending into /jcr:system
> (descend=false)
> 19.09.2019 20:06:08.792 *TRACE* [qtp681943839-1771]
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl including /jcr:system ->
> /jcr:system/jcr:primaryType
> 19.09.2019 20:06:08.792 *TRACE* [qtp681943839-1771]
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl including /jcr:system ->
> /jcr:system
> 19.09.2019 20:06:08.792 *TRACE* [qtp681943839-1771]
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl including /jcr:system ->
> /jcr:system/jcr:mixinTypes
> 19.09.2019 20:06:08.792 *TRACE* [qtp681943839-1771]
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl including /jcr:system ->
> /jcr:system/jcr:versionStorage
> 19.09.2019 20:06:08.793 *TRACE* [qtp681943839-1771]
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl descending into
> /jcr:system/jcr:versionStorage (descend=true)
> 19.09.2019 20:06:08.793 *TRACE* [qtp681943839-1771]
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl including /jcr:system ->
> /jcr:system/jcr:versionStorage/jcr:primaryType
> 19.09.2019 20:06:08.793 *TRACE* [qtp681943839-1771]
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl including /jcr:system ->
> /jcr:system/jcr:versionStorage/ee
> 19.09.2019 20:06:08.793 *TRACE* [qtp681943839-1771]
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl descending into
> /jcr:system/jcr:versionStorage/ee (descend=true)
> 19.09.2019 20:06:08.793 *TRACE* [qtp681943839-1771]
> org.apache.jackrabbit.vault.fs.impl.AggregateImpl including /jcr:system ->
> /jcr:system/jcr:versionStorage/ee/jcr:primaryType
> [...]
> {noformat}
> I found a lot of these "Including /jcr:system -> ..." statements in the log:
> {noformat}
> $ grep -c "AggregateImpl including" filevault.log
> 174425
> $
> {noformat}
> which is logged at [1]. And at [2] something is unconditionally added to a
> global variable. And I think that this is the problematic piece.
> I don't know the details of vault good enough to propose a solution, but I
> would love to have a less memory-intensive algorithm, for which the
> memory-usage does not grow linear with the number of nodes covered by the
> package rules.
> [1]
> https://github.com/apache/jackrabbit-filevault/blob/jackrabbit-filevault-3.2.8/vault-core/src/main/java/org/apache/jackrabbit/vault/fs/impl/AggregateImpl.java#L502
> [2]
> https://github.com/apache/jackrabbit-filevault/blob/jackrabbit-filevault-3.2.8/vault-core/src/main/java/org/apache/jackrabbit/vault/fs/impl/AggregateImpl.java#L507
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
