[
https://issues.apache.org/jira/browse/JENA-901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14368473#comment-14368473
]
Stian Soiland-Reyes edited comment on JENA-901 at 3/19/15 4:14 AM:
-------------------------------------------------------------------
You can call manually .reset() on the OntModel to clear the cache
{code}
/**
* Reset any internal caches. Some systems, such as the tabled backchainer,
* retain information after each query. A reset will wipe this information
preventing
* unbounded memory use at the expense of more expensive future queries. A
reset
* does not cause the raw data to be reconsulted and so is less expensive
than a rebind.
*/
public void reset();
{code}
But that does not really fix this bug - which is to limit the cache memory
usage.
I looked at using SoftReferences here - but it gets tricky as you would still
have a Map with the stale TriplePatterns (which would contain all those
millions of URIs). You can't combine this with a WeakHashMap - as the
TriplePatterns are compared on equality. Doing anything clever with clean-up
of stale soft references on insertion could potentially become quite heavy,
always iterating through the whole list. A ReferenceQueue combined with a Map
gets tricky, as you would also need to keep the TriplePattern key somewhere in
order to remove the whole Entry from the map (or do a slow iterate-purge at
that point). Guava's caches would be more mature for this kind of logic - see
https://code.google.com/p/guava-libraries/wiki/CachesExplained
So I started with a fix using a BoundedMap with a maximum setting.
https://github.com/apache/jena/compare/master...stain:JENA-901-LPDRuleEngine-bound-cache?expand=1
The cache capacity can be configured by setting the system property
`jena.rulesys.lp.max_cached_tabled_goals`
It is tricky to pick the right number - but we can find the lower bounds.
I used the [sizeof](https://github.com/dweiss/java-sizeof) library to see how
big the tabledGoals is after putting in 1024 entries from a very simple query-
and found the memory footprint of LPBRuleEngine to increasing with about 1500
bytes per each iteration of this loop:
{code}
for (int i=0; i<MAX*4096; i++) {
Node test = NodeFactory.createURI("test" + i);
infgraph.find(test, ty, C2).close();
}
{code}
.. even if it is calling .close(). So more investigation needed for this
memory leak. :(
To the profiler!
was (Author: soilandreyes):
You can call manually .reset() on the OntModel to clear the cache
/**
* Reset any internal caches. Some systems, such as the tabled backchainer,
* retain information after each query. A reset will wipe this information
preventing
* unbounded memory use at the expense of more expensive future queries. A
reset
* does not cause the raw data to be reconsulted and so is less expensive
than a rebind.
*/
public void reset();
But that does not really fix this bug - which is to limit the cache memory
usage.
I looked at using SoftReferences here - but it gets tricky as you would still
have a Map with the stale TriplePatterns (which would contain all those
millions of URIs). You can't combine this with a WeakHashMap - as the
TriplePatterns are compared on equality. Doing anything clever with clean-up
of stale soft references on insertion could potentially become quite heavy,
always iterating through the whole list. A ReferenceQueue combined with a Map
gets tricky, as you would also need to keep the TriplePattern key somewhere in
order to remove the whole Entry from the map (or do a slow iterate-purge at
that point). Guava's caches would be more mature for this kind of logic - see
https://code.google.com/p/guava-libraries/wiki/CachesExplained
So I started with a fix using a BoundedMap with a maximum setting.
https://github.com/apache/jena/compare/master...stain:JENA-901-LPDRuleEngine-bound-cache?expand=1
The cache capacity can be configured by setting the system property
`jena.rulesys.lp.max_cached_tabled_goals`
It is tricky to pick the right number - but we can find the lower bounds.
I used the [sizeof](https://github.com/dweiss/java-sizeof) library to see how
big the tabledGoals is after putting in 1024 entries from a very simple query-
and found the memory footprint of LPBRuleEngine to increasing with about 1500
bytes per each iteration of this loop:
for (int i=0; i<MAX*4096; i++) {
Node test = NodeFactory.createURI("test" + i);
infgraph.find(test, ty, C2).close();
}
.. even if it is calling .close(). So more investigation needed for this
memory leak. :(
To the profiler!
> Make the cache of LPBRuleEngine bounded to avoid out-of-memory
> --------------------------------------------------------------
>
> Key: JENA-901
> URL: https://issues.apache.org/jira/browse/JENA-901
> Project: Apache Jena
> Issue Type: Improvement
> Components: Reasoners
> Affects Versions: Jena 2.12.1
> Reporter: Jan De Beer
>
> The class "com.hp.hpl.jena.reasoner.rulesys.impl.LPBRuleEngine" uses an
> in-memory cache named "tabledGoals", which has no limit as to the size/number
> of entries stored.
> /** Table mapping tabled goals to generators for those goals.
> * This is here so that partial goal state can be shared across multiple
> queries. */
> protected HashMap<TriplePattern, Generator> tabledGoals = new HashMap<>();
> We have experienced out-of-memory issues because of the cache being filled
> with millions of entries in just a few days under normal query usage
> conditions and a heap memory set to 3GB.
> In our setup, we have a dataset containing multiple graphs, some of them are
> actual data graphs (backed by TDB), and then there are two which are ontology
> models using a "TransitiveReasoner" and an "OWLMicroFBRuleReasoner",
> respectively. A typical query may run over all the graphs in the dataset,
> including the ontology ones (see below for a query template). Eventhough the
> ontology graphs would not yield any additional results for data queries
> (which is fine), the above mentioned cache would still fill up with new
> entries.
> SELECT ?p ?o
> WHERE {
> GRAPH ?g {
> <some resource of interest> ?p ?o .
> }
> }
> As there is no upper bound to the cache, soon or later all available heap
> memory will be consumed by the cache, giving rise to an out-of-memory
> criticial error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
