Excellent work, Johannes.
Impressive. How long did it take to draw these conclusions from
profiling mmbase?
Ernst
Johannes Verelst wrote:
1.8.1 is a bugfix release, with major improvements on the terrain of
performance and memory-use, and some minor other improvements. I tried
to make a more complete overview in the release-notes.
Let me elaborate a bit on the performance and memory issues which I
fixed, in the hope that we don't get any new code that needs
refactoring later on. I found all of this issues using the JProfiler
profiler we just bought, and I can recommend it to everybody.
- I fixed several locations where loops like this occurred:
for (int i=0; i<something.size(); i++) {}
The 'something.size()' call is executed for every iteration, and with
many iterations this is quite costly. I refactored the most
expensive ones to calculate the size beforehand.
- At some places where large binary streams were used, they were
read/written byte-by-byte, instead of using bufferedreaders.
Especially when handling images this was quite a problem. I fixed many
of those, but there still is a place where I couldn't fix it (the
fix-corrupt-jpeg-images-transformer, it needs to read byte-by-byte and
inspect every byte). Maybe this should be redesigned: instead of
fixing the JPEG everytime it is sent out (and by that, processing
every image with this filter), it might be more sane to process the
images once before putting them in the database.
- Calculating hashcodes can be quite expensive, and when putting
objects in hashmaps/tables (like queries), the hashCode() method is
called quite often. When an object doesn't change, the hashcode
doesn't change, so there is no need in recalculating it.
- Several fixes in the Informix support code, because connections need
to be 'scrubbed' before they can be put back in a connectionpool, and
there was a query.toString() being executed for every query even for
debugging ... even when debugging was turned off.
- Several caches (like the RelatedNodesCache) contain references to
other private hashmaps. Whenever this happens, there must be an
overridden 'clear()' method that will also clear the referenced
private hashmaps. If this isn't done, this is a sure memory leak.
- In the taglib, very often the tag instances have references to
queries, nodelists, clouds etc. When these tags are pooled, they keep
their references and you have an instant memoryleak. Every .JSP file
has its own tagpool for every tag, so this means with many JSP files
(as we have), a possible huge memory leak.
I fixed many of these, but there is still a problem with exceptions in
mmbase 1.8.1. COnsider the following pseudo-code which is generated
for a jsp:
tag = tagpool.getTag("mm_import_1");
try {
tag.doStartTag();
Object o = null; out.println(o.getClass())
tag.doEndTag();
} catch (Exception e) {}
tagpool.putback(tag);
The cleanup that happens in the 'doEndTag()' will never be executed
because of the nullpointer exception on line 4. So, if a 500 occurs,
there will be a leaked object because the tag remains in the tagpool
with a reference which is not cleaned up.
For MMBase 1.8.2 I fixed this by implementing the 'TryCatchFinally'
tag interface, which adds a 'doFinally()' method for all tags, where
cleaning up can be done even when exceptions occurs. For people using
1.8.1 (or any version before that), I recommend disabling tagpooling.
Note: tagpooling is an option used for generating java code, so you
must clean your work directory after changing this setting!
The tagpooling-leak is probably the most important one for people now
running 1.8, so thats the main reason for me to send out this mail.
Also I want to point out that there are many 'hidden' bugs on
performance/memory level inside MMBase that you would normally not
find. It might be a good idea to do some profiling runs before every
release to make sure nothing stupid happens inside the code.
Johannes
_______________________________________________
Developers mailing list
[email protected]
http://lists.mmbase.org/mailman/listinfo/developers
_______________________________________________
Developers mailing list
[email protected]
http://lists.mmbase.org/mailman/listinfo/developers
