I have checked in a fix this morning, but I don't know how bad the
performance will be, Some parts of the new code tries to optimize, but
it looks a little ugly.
Now, I have spend some time in the cache code and I think MMbase could
use its caches better when it uses more regions which match the purpose
of the query. More specialized caches should be able to flush better.
The NodeListCache and RelatedNodesCache are not just QueryResultCaches.
They are used on special occasions and can be flushed based on those rules.
Nico
Nico Klasens wrote:
Hello Developers,
Yesterday, I came back from a nice long christmas weekend and was
confrontend with the fact that one of the cmsc demo instances had ran
out of memory. I checked the access log files and this instance wasn't
used for more than a week. So it could not have been this issue
http://issues.apache.org/bugzilla/show_bug.cgi?id=37793
I didn't understand why it was complaining about memory after a week
doing "nothing". The jmap Object Histogram of java 5 showed this
result. The jvm hosts 2 mmbase war files
Size Count Class description
-------------------------------------------------------
25055520 1043980 java.util.HashMap$Entry
10993632 458068
org.mmbase.storage.search.implementation.BasicStepField
10851400 90350 java.util.HashMap$Entry[]
10757472 448228
org.mmbase.storage.search.implementation.BasicStepField
6275544 261481 java.util.ArrayList
3609240 90231 java.util.HashMap
2646912 82716
org.mmbase.storage.search.implementation.BasicFieldValueConstraint
2626816 82088
org.mmbase.storage.search.implementation.BasicFieldValueConstraint
2167200 54180 java.util.TreeMap
1197072 24939
org.mmbase.storage.search.implementation.NodeSearchQuery
1187040 24730
org.mmbase.storage.search.implementation.NodeSearchQuery
1185432 49393
org.mmbase.storage.search.implementation.BasicCompositeConstraint
1176816 49034
org.mmbase.storage.search.implementation.BasicCompositeConstraint
866608 54163 java.util.TreeMap$1
866464 54154 java.util.TreeSet
778144 24317
org.mmbase.storage.search.implementation.BasicFieldValueBetweenConstraint
777696 24303
org.mmbase.storage.search.implementation.BasicFieldValueBetweenConstraint
680232 28343 java.util.Hashtable$Entry
649152 27048 org.mmbase.storage.search.implementation.BasicStep
613680 25570 org.mmbase.storage.search.implementation.BasicStep
521408 16294 org.mmbase.util.LRUHashtable$LRUEntry
401472 3037 java.util.Hashtable$Entry[]
I was suprised about the fact that there were almost 25000
NodeSearchQuery instnaces in memory for each mmbase instance. We still
use the default caches.xml which has max values of 300 entries. After
some experiments it turns out the cache is filled by queries with a
timestamp which are executed by a cronjob. More suprising is that the
code explicitly sets the CachePolicy to NEVER. It turned out that the
NodeManager.getList() did nothing with the CachePolicy which is easily
fixed.
I did some profiling and found the problem. The QueryResultCache
registers observers and adds queries to a map which it should remove
when nodes of a specific type change. The CacheImplementation, which
is used by the QueryResultCache, can decide that queries are unsued
and can be removed. This is what the LRUHashtable does. The query is
still in the observer map after the CacheImplementation removed it.
In other words, the QueryResultCache observers queries which are not
cached anymore.
As a side-effect, the performance of the cache release is bad, because
more queries have to be evaluated than actually cached. In my case it
was even worse, because the fields inside the query (selection,
constraint and sortorder) are not updated often which will keep them
in the observers.
At the moment, I am investigating how to solve this issue. Suggestions
are welcome.
I have created an issue in jira:
http://www.mmbase.org/jira/browse/MMB-1367
Regards
Nico
_______________________________________________
Developers mailing list
[email protected]
http://lists.mmbase.org/mailman/listinfo/developers
_______________________________________________
Developers mailing list
[email protected]
http://lists.mmbase.org/mailman/listinfo/developers
