Hello Developers,
Yesterday, I came back from a nice long christmas weekend and was
confrontend with the fact that one of the cmsc demo instances had ran
out of memory. I checked the access log files and this instance wasn't
used for more than a week. So it could not have been this issue
http://issues.apache.org/bugzilla/show_bug.cgi?id=37793
I didn't understand why it was complaining about memory after a week
doing "nothing". The jmap Object Histogram of java 5 showed this result.
The jvm hosts 2 mmbase war files
Size Count Class description
-------------------------------------------------------
25055520 1043980 java.util.HashMap$Entry
10993632 458068
org.mmbase.storage.search.implementation.BasicStepField
10851400 90350 java.util.HashMap$Entry[]
10757472 448228
org.mmbase.storage.search.implementation.BasicStepField
6275544 261481 java.util.ArrayList
3609240 90231 java.util.HashMap
2646912 82716
org.mmbase.storage.search.implementation.BasicFieldValueConstraint
2626816 82088
org.mmbase.storage.search.implementation.BasicFieldValueConstraint
2167200 54180 java.util.TreeMap
1197072 24939 org.mmbase.storage.search.implementation.NodeSearchQuery
1187040 24730 org.mmbase.storage.search.implementation.NodeSearchQuery
1185432 49393
org.mmbase.storage.search.implementation.BasicCompositeConstraint
1176816 49034
org.mmbase.storage.search.implementation.BasicCompositeConstraint
866608 54163 java.util.TreeMap$1
866464 54154 java.util.TreeSet
778144 24317
org.mmbase.storage.search.implementation.BasicFieldValueBetweenConstraint
777696 24303
org.mmbase.storage.search.implementation.BasicFieldValueBetweenConstraint
680232 28343 java.util.Hashtable$Entry
649152 27048 org.mmbase.storage.search.implementation.BasicStep
613680 25570 org.mmbase.storage.search.implementation.BasicStep
521408 16294 org.mmbase.util.LRUHashtable$LRUEntry
401472 3037 java.util.Hashtable$Entry[]
I was suprised about the fact that there were almost 25000
NodeSearchQuery instnaces in memory for each mmbase instance. We still
use the default caches.xml which has max values of 300 entries. After
some experiments it turns out the cache is filled by queries with a
timestamp which are executed by a cronjob. More suprising is that the
code explicitly sets the CachePolicy to NEVER. It turned out that the
NodeManager.getList() did nothing with the CachePolicy which is easily
fixed.
I did some profiling and found the problem. The QueryResultCache
registers observers and adds queries to a map which it should remove
when nodes of a specific type change. The CacheImplementation, which is
used by the QueryResultCache, can decide that queries are unsued and can
be removed. This is what the LRUHashtable does. The query is still in
the observer map after the CacheImplementation removed it. In other
words, the QueryResultCache observers queries which are not cached anymore.
As a side-effect, the performance of the cache release is bad, because
more queries have to be evaluated than actually cached. In my case it
was even worse, because the fields inside the query (selection,
constraint and sortorder) are not updated often which will keep them in
the observers.
At the moment, I am investigating how to solve this issue. Suggestions
are welcome.
I have created an issue in jira: http://www.mmbase.org/jira/browse/MMB-1367
Regards
Nico
_______________________________________________
Developers mailing list
[email protected]
http://lists.mmbase.org/mailman/listinfo/developers
