Werner,
My test program on jira (http://jira.codehaus.org/browse/CASTOR-1910)
creates a single resolver and is only one thread therefor i do not
synchronize access to it for the test - and still it eats memory!
However, i've seen that the number of descriptors does change over time
(shown in jProfiler) even though my test is using the same bit of xml
each time - so if the number of desriptors is changing then the cache is
still being *written* to and read from therefor not 100% thread safe.
e.g., after 60 iterations (each iteration is 100 marshal and unmarshals)
of my test app (with -Xms128M -Xmx512M) there are 1.9 million instances
of XMLFieldDescriptorImpl taking up 156Mb of memory!
Iteration 50 : 15328 millis [u:93 m:93 f:68 t:508]
Iteration 51 : 15703 millis [u:94 m:94 f:66 t:508]
Iteration 52 : 16344 millis [u:67 m:67 f:63 t:508]
Iteration 53 : 20375 millis [u:97 m:97 f:67 t:508]
Iteration 54 : 19578 millis [u:130 m:130 f:35 t:508]
Iteration 55 : 27015 millis [u:68 m:68 f:33 t:508]
Iteration 56 : 30672 millis [u:237 m:237 f:30 t:508]
Iteration 57 : 27985 millis [u:107 m:107 f:27 t:508]
Iteration 58 : 28188 millis [u:142 m:142 f:2 t:508]
Iteration 59 : 38953 millis [u:178 m:178 f:1 t:508]
Iteration 60 : 43844 millis [u:257 m:257 f:0 t:508]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
not to mention the time for the iterations!!
Thanks
Simon Lord
Werner Guttmann wrote:
Simon,
Please see inline.
Werner
-----Ursprüngliche Nachricht-----
Von: Simon Lord [mailto:[EMAIL PROTECTED]
Gesendet: Donnerstag, 15. März 2007 16:03
An: [email protected]
Betreff: Re: AW: [castor-user] [XML] XMLClassDescriptorResolver
XMLFieldDescriptorImpl cache possible memory leak
Hi there,
I've attached my small eclipse project to the JIRA issue i created:
http://jira.codehaus.org/browse/CASTOR-1910
Yes i was using the default memory in the test that i posted. The fact
is though that after the first iteration the memory should (in my mind)
stay the same as it's *exactly* the same piece of xml on the next
iteration - so it should need to load more descriptors.
Setting the memory to something larger doesn't fix the problem it just
takes a little longer to run out of memory! As it never 'levels off'
anywhere.
The cache of descriptors inside XMLClassDescriptorResolverImpl.java uses
java.util.HashMap which is known to not be thread safe
(http://java.sun.com/j2se/1.5.0/docs/api/java/util/HashMap.html) as it's
not synchronized. Whether that matters or not with the way the resolver
is used/setup i do not know - but i'd rather play it safe and use it in
a thread safe fashion ;-)
I think our desire to play it (over-ambitiously thread-safe) is actually
causing the problem here. Let me just repeat what Leander just said below:
However, in my opinion XMLClassDescriptorResolver does not need to be
thread safe. Only the initial initialization of such a resolver instance
needs to be thread safe. Later, whenever you use that instance in
marshaller and unmarshaller instances, they will only use the resolver
in a "read-only" way, and not modify the resolver, thus no thread
synchronization is neccessary. Right?
There's two distinctive phases in the life-cycle of an
XMLClassdescriptorResolver:
a) initialization.
b) usage for (un)marshalling.
Once a) has been completed, b) is 100% thread-safe. In other words, the usage
of one (and only one) XMLClassDescriptorResolver for your multi-threaded
application is 100% okay. You just need to makes sure that initialization
happens *before* usage.
Let me know how you get on.
Thanks
Simon Lord
M.-Leander Reimer wrote:
Hello Simon,
I am curious now, because I am also using the XMLClassDescriptorResolver
approach and our application that has many threads where unmarshalling
and marshalling happens, and I would like to know whether there is a
problem with that approach.
However, in my opinion XMLClassDescriptorResolver does not need to be
thread safe. Only the initial initialization of such a resolver instance
needs to be thread safe. Later, whenever you use that instance in
marshaller and unmarshaller instances, they will only use the resolver
in a "read-only" way, and not modify the resolver, thus no thread
synchronization is neccessary. Right?
Regarding your tests and the possible memory leak. Obviously, caching
things will require more memory. I am curious about your JVM settings
for the test that used the resolver. Have you increased the heap size or
did you run with the default 64MB?
> Iteration 7 : 6984 millis [u:31 m:31 f:5 t:63]
Total memory is at 63, so I guess the garbage collector kicked in, thus
the increase in time, but no more memory was available, thus the error.
Still I would like to see your unit test, and perhaps I will also do a
little testing with my implementation to check the memory consumption.
Regards,
Leander
Simon Lord schrieb:
Hello again Werner,
Thanks for the reply.
In the application i help to develop it is impossible for us to use
only
one resolver, as we have many threads needing to perform
(un)marshalling
operations and the implementation of XMLClassDescriptorResolver is not
thread safe (it uses HashMaps) and we cannot afford to block until the
resolver is available.
However, i've spent a little time creating a unit test that i believe
shows the symptoms of a memory leak. It also shows that using the
resolver does not speed up the (un)marshaling, in fact slowing it down
a
touch.
The test makes ten iterations of 100 unmarshals then marshals of the
same bit of xml. Without a resolver the test completes but with the
resolver the test fails on the 7th iteration with a java heap space
exception.
I'm going to create new issue on JIRA where i can attach source and
jars or a zip of the eclipse projct i've created if that easier, to
avoid
sending them to everyone on this list.
Here is the output from my test:
u = avg unmarshal time (millis)
m = avg marshal time (millis)
f = free mem (mb)
t = total mem (mb)
Test setup: 10 of 100 unmarshal then marshal, useResolver = false
Iteration 1 : 2047 millis [u:13 m:13 f:0 t:1]
Iteration 2 : 1265 millis [u:7 m:7 f:1 t:1]
Iteration 3 : 1250 millis [u:6 m:6 f:0 t:1]
Iteration 4 : 1188 millis [u:7 m:7 f:0 t:1]
Iteration 5 : 1219 millis [u:6 m:6 f:0 t:1]
Iteration 6 : 1171 millis [u:6 m:6 f:0 t:1]
Iteration 7 : 1204 millis [u:6 m:6 f:0 t:1]
Iteration 8 : 1234 millis [u:6 m:6 f:1 t:1]
Iteration 9 : 1156 millis [u:6 m:6 f:0 t:1]
Iteration 10 : 1203 millis [u:6 m:6 f:1 t:1]
Completed successfully
Test setup: 10 of 100 unmarshal then marshal, useResolver = true
Iteration 1 : 1235 millis [u:6 m:6 f:3 t:13]
Iteration 2 : 1265 millis [u:8 m:8 f:1 t:20]
Iteration 3 : 1532 millis [u:6 m:6 f:7 t:32]
Iteration 4 : 1890 millis [u:11 m:11 f:18 t:54]
Iteration 5 : 1797 millis [u:7 m:7 f:10 t:54]
Iteration 6 : 2047 millis [u:11 m:11 f:3 t:54]
Iteration 7 : 6984 millis [u:31 m:31 f:5 t:63]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
Thank you for your time
Simon Lord
Werner Guttmann wrote:
Hmm, you should *not* be creating a resolver for each root class, but
one resolver only for all classes potentially used throughout your
(un)marshalling operations. Once the resolver has loaded - in your
case - the descriptors for all classes found in that package, it will
cache them.
And the Unmarshaller will be able to retrieve the cached descriptors
as needed.
I hope this explains things in more detail.
Werner
-----Ursprüngliche Nachricht-----
Von: Simon Lord [mailto:[EMAIL PROTECTED]
Gesendet: Donnerstag, 15. März 2007 10:22
An: [email protected]
Betreff: Re: [castor-user] [XML] XMLClassDescriptorResolver
XMLFieldDescriptorImpl cache possible memory leak
Hi Steven,
Thanks for your speedy reply.
I understand that there is an object graph underneath the root
object,
hense i understand it needs to load the 3828 descriptors for the
object
graph underneath my root object. What i don't understand is why when
marshalling/unmarshalling the same root class (albeit with possibly
different values e.g., two objects of type person might have
different
names but it's still the same name field) the resolver is loading
more
descriptors but should in my opinion be using the same descriptors,
it
is only the values that have changed not the class/fields. I
understand
that in a complex object graph it may need to load more descriptors
for
different objects due to it having a larger object graph - but my
object
is staying relatively similiar each time - therefor i would expect
some
growth but not constant growth with each iteration.
Also i'm not trying to create a resolver for every class - i'm
creating
a resolver for each root class and reusing them (and yes i am setting
the resolver on the marshaller/unmashaller ;-). I don't think it is
thread safe either - so i've implemented it in a thread safe manner
I hope this sounds more like i'm implementing/understanding it all in
the right way.
I'm off to write a unit test now and do some further investigation :)
Thanks
Simon Lord
Steven Dolg wrote:
Hi Simon,
The 3828 FieldDescriptors indicate that the object graph your
(un)marshalling contains some more classes than the one you're
creating
the ClassDescriptorResolver for.
So the (Un)marshaller asks the ClassDescriptorResolver to find all
descriptors necessary for the whole object graph (not just the root
object).
This might lead to creating different ClassDescriptorResolvers (for
different root classes) that might contain almost identical
descriptors
(for all other objects in the object graph).
It shouldn't be necessary to create a ClassDescriptorResolver for
each
class individually so I'd suggest using only one resolver. (I'm not
entirely sure whether this is thread-safe or not)
And do not forget to set the created ClassDescriptorResolver at the
(Un)marshaller... ;-)
Unmarshaller unm = new Unmarshaller(...);
unm.setResolver(cdr);
Regards,
Steven
----- Original Message ----- From: "Simon Lord"
<[EMAIL PROTECTED]>
To: <[email protected]>
Sent: Wednesday, March 14, 2007 6:44 PM
Subject: [castor-user] [XML] XMLClassDescriptorResolver
XMLFieldDescriptorImpl cache possible memory leak
Afternoon all,
I've recently moved from using an old version of Castor (0.9.xxx)
to
the new stable 1.1. Mainly as i'd like to use the descriptor
caching.
However, after moving to castor 1.1 and creating an
XMLClassDescriptorResolver as per the xml-best-practices webpage,
running my application caused an out of memory expection
(heapspace).
So i ran the code under jProfiler and have seen symptoms of a
memory
leak (or symptoms of not using the resolver correctly ;-).
It my understanding that when marshalling/unmarshalling, the
XMLClassDescriptorResolver should be caching the descriptors for
use
in subsequent operations. Then on the subsequent operation (marshal
/
unmarshall) it shouldn't have to create those descriptors again
(assumming the same class/xml complex type is being used).
But what i've seen in jProfiler is that after one run of my code
there
have been 3828 XMLFieldDescriptorImpl objects created (triggering
garbage collection a few times doesn't change that number either)
then
when i run it again (no restart, just second iteration) it finishes
what it's doing and the count of XMLFieldDescriptorImpl objects is
now
7725!
It increases with each iteration, and after 10 iterations is 38901
objects, taking up 3.4 megabytes in memory - after a while this
gets
so big it causes the memory exception.
So it looks like it's created new XMLFieldDescriptorImpl for the
same
classes and xml complex types i used the first time around.
Has anyone else seen this behaviour before? Am i doing somthing
wrong
when creating / using the resolver?
Here's the code i use to contruct a resolver (sorry about the
formatting):
private XMLClassDescriptorResolver createResolver(Class clazz){
XMLClassDescriptorResolver cdResolver =
(XMLClassDescriptorResolver)ClassDescriptorResolverFactory.createClassDesc
riptorResolver(BindingType.XML);
cdResolver.setClassLoader(Thread.currentThread().getContextClassLoader());
try {
cdResolver.loadClassDescriptors(clazz.getPackage().toString());
return cdResolver;
} catch (ResolverException e) {
logger.warn(e.getMessage() +
":"+clazz.getPackage().toString(),e);
}
return null;
}
as you can see from the above code i create a resolver for each
Class
i marshal/unmarshall and store it for subsequent uses of the same
operation (so i'm not creating a new resolver each time as it might
appear)
Thank you for your time
Simon Lord
-------------------------------------------------------------------
--
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
--------------------------------------------------------------------
-
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
---------------------------------------------------------------------
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
---------------------------------------------------------------------
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
---------------------------------------------------------------------
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
---------------------------------------------------------------------
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
---------------------------------------------------------------------
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
---------------------------------------------------------------------
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
---------------------------------------------------------------------
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
