[
https://issues.apache.org/jira/browse/JENA-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762127#comment-17762127
]
Damien Obrist commented on JENA-2354:
-------------------------------------
[~andy] thanks for your analysis and effort!
{quote}It is the Caffeine cache
{quote}
How was the initial capacity set before the switch to Caffeine?
{quote}Do you need to keep all the databases permanently?
{quote}
Yes, we currently keep all databases permanently.
{quote}You can tune down the size of the cache with `StoreParams`
{quote}
Thanks for this insight and the code snippet! Based on the stack trace of the
OOM exception and the code of
[NodeTableCache.create|https://github.com/apache/jena/blob/0d5d10790650c4e47fc2a020e9da6c72598fb31b/jena-tdb2/src/main/java/org/apache/jena/tdb2/store/nodetable/NodeTableCache.java#L77-L79],
can you confirm that the following would be the ones I would need to tune?
{noformat}
nodeToIdCacheSize (default: 200000)
idToNodeCacheSize (default: 1000000)
missCacheSize (default: 1000){noformat}
However, I'm a bit hesitant as reducing the maximum cache size might have an
impact on performance depending on the size of the dataset. We might need to
reduce the cache size quite a bit to achieve a similar memory profile (i.e. a
low initial capacity), which could in turn potentially lead to a performance
degradation for bigger datasets.
Would it potentially make sense and be possible to also make the initial
capacity of the caches (i.e. the ratio) configurable via {{{}StoreParams{}}}?
> OOM / increased memory consumption with TDB 4.9.0
> -------------------------------------------------
>
> Key: JENA-2354
> URL: https://issues.apache.org/jira/browse/JENA-2354
> Project: Apache Jena
> Issue Type: Bug
> Components: TDB2
> Affects Versions: Jena 4.9.0
> Reporter: Damien Obrist
> Assignee: Andy Seaborne
> Priority: Major
> Fix For: Jena 4.10.0
>
> Attachments: tdb-4.8.0-run-2-memory.png, tdb-4.9.0-run-6-memory.png,
> tdb-4.9.0-run-7-memory.png, tdb-490-memory-increase-4.8.0-memory.png,
> tdb-490-memory-increase-4.9.0-exception.png,
> tdb-490-memory-increase-4.9.0-memory.png, tdb-490-memory-increase.zip
>
>
> I have observed that after upgrading from TDB 4.8.0 to 4.9.0, memory
> consumption seems to have increased substantially.
> Our application creates many TDB datasets, which are queried and updated
> concurrently. To profile the memory increase, I have implemented a small
> script, which creates up to 50 of our application-level objects, one after
> another. Each of these objects is backed by a TDB dataset under the hood,
> which gets created from scratch and filled with data (around 30k triples,
> 200MB folder size).
> The following shows the memory profiles when running the script against our
> application for both TDB 4.8.0 / 4.9.0 and using 1G of heap space.
> h3. TDB 4.8.0
> Memory increases only slightly and the script completes the creation of all
> 50 objects / TDB datasets:
> !tdb-4.8.0-run-2-memory.png|width=800,height=207!
> h3. TDB 4.9.0
> Memory increases fast and the application crashes with an OOM after having
> created 8 objects / TDB datasets, during the creation of the 9th:
> !tdb-4.9.0-run-6-memory.png|width=800,height=207!
> Stack trace:
> {code:java}
> java.lang.OutOfMemoryError: Java heap space
> at
> com.github.benmanes.caffeine.cache.FrequencySketch.ensureCapacity(FrequencySketch.java:92)
> at com.github.benmanes.caffeine.cache.SSMS.<init>(Unknown Source)
> at
> java.base/java.lang.invoke.LambdaForm$DMH/0x00000008028f7c40.newInvokeSpecial(LambdaForm$DMH)
> at
> java.base/java.lang.invoke.DelegatingMethodHandle$Holder.reinvoke_L(DelegatingMethodHandle$Holder)
> at
> java.base/java.lang.invoke.LambdaForm$MH/0x00000008028f7040.invoke_MT(LambdaForm$MH)
> at
> com.github.benmanes.caffeine.cache.LocalCacheFactory.loadFactory(LocalCacheFactory.java:86)
> at
> com.github.benmanes.caffeine.cache.LocalCacheFactory.newBoundedLocalCache(LocalCacheFactory.java:40)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.<init>(BoundedLocalCache.java:3947)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.<init>(BoundedLocalCache.java:3943)
> at com.github.benmanes.caffeine.cache.Caffeine.build(Caffeine.java:1051)
> at
> org.apache.jena.atlas.lib.cache.CacheCaffeine.<init>(CacheCaffeine.java:62)
> at
> org.apache.jena.atlas.lib.CacheFactory.createCache(CacheFactory.java:45)
> at
> org.apache.jena.atlas.lib.CacheFactory.createCache(CacheFactory.java:33)
> at
> org.apache.jena.tdb2.store.nodetable.NodeTableCache.createCache(NodeTableCache.java:101)
> at
> org.apache.jena.tdb2.store.nodetable.NodeTableCache.<init>(NodeTableCache.java:95)
> at
> org.apache.jena.tdb2.store.nodetable.NodeTableCache.create(NodeTableCache.java:87)
> at
> org.apache.jena.tdb2.store.TDB2StorageBuilder.addNodeTableCache(TDB2StorageBuilder.java:302)
> at
> org.apache.jena.tdb2.store.TDB2StorageBuilder.buildNodeTable(TDB2StorageBuilder.java:287)
> at
> org.apache.jena.tdb2.store.TDB2StorageBuilder.buildStorage(TDB2StorageBuilder.java:189)
> at
> org.apache.jena.tdb2.store.TDB2StorageBuilder.build(TDB2StorageBuilder.java:97)
> at org.apache.jena.tdb2.sys.StoreConnection.make(StoreConnection.java:95)
> at
> org.apache.jena.tdb2.sys.StoreConnection.connectCreate(StoreConnection.java:63)
> at
> org.apache.jena.tdb2.sys.DatabaseOps.createSwitchable(DatabaseOps.java:139)
> at org.apache.jena.tdb2.sys.DatabaseOps.create(DatabaseOps.java:86)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection.build(DatabaseConnection.java:112)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection.lambda$make$0(DatabaseConnection.java:74)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection$$Lambda$8502/0x00000008028e3c40.apply(Unknown
> Source)
> at
> java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection.make(DatabaseConnection.java:74)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:63)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:54)
> at org.apache.jena.tdb2.DatabaseMgr.DB_ConnectCreate(DatabaseMgr.java:41)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
