Hello Simon,
I am curious now, because I am also using the XMLClassDescriptorResolver
approach and our application that has many threads where unmarshalling
and marshalling happens, and I would like to know whether there is a
problem with that approach.
However, in my opinion XMLClassDescriptorResolver does not need to be
thread safe. Only the initial initialization of such a resolver instance
needs to be thread safe. Later, whenever you use that instance in
marshaller and unmarshaller instances, they will only use the resolver
in a "read-only" way, and not modify the resolver, thus no thread
synchronization is neccessary. Right?
Regarding your tests and the possible memory leak. Obviously, caching
things will require more memory. I am curious about your JVM settings
for the test that used the resolver. Have you increased the heap size or
did you run with the default 64MB?
> Iteration 7 : 6984 millis [u:31 m:31 f:5 t:63]
Total memory is at 63, so I guess the garbage collector kicked in, thus
the increase in time, but no more memory was available, thus the error.
Still I would like to see your unit test, and perhaps I will also do a
little testing with my implementation to check the memory consumption.
Regards,
Leander
Simon Lord schrieb:
Hello again Werner,
Thanks for the reply.
In the application i help to develop it is impossible for us to use only
one resolver, as we have many threads needing to perform (un)marshalling
operations and the implementation of XMLClassDescriptorResolver is not
thread safe (it uses HashMaps) and we cannot afford to block until the
resolver is available.
However, i've spent a little time creating a unit test that i believe
shows the symptoms of a memory leak. It also shows that using the
resolver does not speed up the (un)marshaling, in fact slowing it down a
touch.
The test makes ten iterations of 100 unmarshals then marshals of the
same bit of xml. Without a resolver the test completes but with the
resolver the test fails on the 7th iteration with a java heap space
exception.
I'm going to create new issue on JIRA where i can attach source and jars
or a zip of the eclipse projct i've created if that easier, to avoid
sending them to everyone on this list.
Here is the output from my test:
u = avg unmarshal time (millis)
m = avg marshal time (millis)
f = free mem (mb)
t = total mem (mb)
Test setup: 10 of 100 unmarshal then marshal, useResolver = false
Iteration 1 : 2047 millis [u:13 m:13 f:0 t:1]
Iteration 2 : 1265 millis [u:7 m:7 f:1 t:1]
Iteration 3 : 1250 millis [u:6 m:6 f:0 t:1]
Iteration 4 : 1188 millis [u:7 m:7 f:0 t:1]
Iteration 5 : 1219 millis [u:6 m:6 f:0 t:1]
Iteration 6 : 1171 millis [u:6 m:6 f:0 t:1]
Iteration 7 : 1204 millis [u:6 m:6 f:0 t:1]
Iteration 8 : 1234 millis [u:6 m:6 f:1 t:1]
Iteration 9 : 1156 millis [u:6 m:6 f:0 t:1]
Iteration 10 : 1203 millis [u:6 m:6 f:1 t:1]
Completed successfully
Test setup: 10 of 100 unmarshal then marshal, useResolver = true
Iteration 1 : 1235 millis [u:6 m:6 f:3 t:13]
Iteration 2 : 1265 millis [u:8 m:8 f:1 t:20]
Iteration 3 : 1532 millis [u:6 m:6 f:7 t:32]
Iteration 4 : 1890 millis [u:11 m:11 f:18 t:54]
Iteration 5 : 1797 millis [u:7 m:7 f:10 t:54]
Iteration 6 : 2047 millis [u:11 m:11 f:3 t:54]
Iteration 7 : 6984 millis [u:31 m:31 f:5 t:63]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
Thank you for your time
Simon Lord
Werner Guttmann wrote:
Hmm, you should *not* be creating a resolver for each root class, but
one resolver only for all classes potentially used throughout your
(un)marshalling operations. Once the resolver has loaded - in your
case - the descriptors for all classes found in that package, it will
cache them.
And the Unmarshaller will be able to retrieve the cached descriptors
as needed.
I hope this explains things in more detail.
Werner
-----Ursprüngliche Nachricht-----
Von: Simon Lord [mailto:[EMAIL PROTECTED]
Gesendet: Donnerstag, 15. März 2007 10:22
An: [email protected]
Betreff: Re: [castor-user] [XML] XMLClassDescriptorResolver
XMLFieldDescriptorImpl cache possible memory leak
Hi Steven,
Thanks for your speedy reply.
I understand that there is an object graph underneath the root object,
hense i understand it needs to load the 3828 descriptors for the object
graph underneath my root object. What i don't understand is why when
marshalling/unmarshalling the same root class (albeit with possibly
different values e.g., two objects of type person might have different
names but it's still the same name field) the resolver is loading more
descriptors but should in my opinion be using the same descriptors, it
is only the values that have changed not the class/fields. I understand
that in a complex object graph it may need to load more descriptors for
different objects due to it having a larger object graph - but my object
is staying relatively similiar each time - therefor i would expect some
growth but not constant growth with each iteration.
Also i'm not trying to create a resolver for every class - i'm creating
a resolver for each root class and reusing them (and yes i am setting
the resolver on the marshaller/unmashaller ;-). I don't think it is
thread safe either - so i've implemented it in a thread safe manner
I hope this sounds more like i'm implementing/understanding it all in
the right way.
I'm off to write a unit test now and do some further investigation :)
Thanks
Simon Lord
Steven Dolg wrote:
Hi Simon,
The 3828 FieldDescriptors indicate that the object graph your
(un)marshalling contains some more classes than the one you're creating
the ClassDescriptorResolver for.
So the (Un)marshaller asks the ClassDescriptorResolver to find all
descriptors necessary for the whole object graph (not just the root
object).
This might lead to creating different ClassDescriptorResolvers (for
different root classes) that might contain almost identical descriptors
(for all other objects in the object graph).
It shouldn't be necessary to create a ClassDescriptorResolver for each
class individually so I'd suggest using only one resolver. (I'm not
entirely sure whether this is thread-safe or not)
And do not forget to set the created ClassDescriptorResolver at the
(Un)marshaller... ;-)
Unmarshaller unm = new Unmarshaller(...);
unm.setResolver(cdr);
Regards,
Steven
----- Original Message ----- From: "Simon Lord"
<[EMAIL PROTECTED]>
To: <[email protected]>
Sent: Wednesday, March 14, 2007 6:44 PM
Subject: [castor-user] [XML] XMLClassDescriptorResolver
XMLFieldDescriptorImpl cache possible memory leak
Afternoon all,
I've recently moved from using an old version of Castor (0.9.xxx) to
the new stable 1.1. Mainly as i'd like to use the descriptor caching.
However, after moving to castor 1.1 and creating an
XMLClassDescriptorResolver as per the xml-best-practices webpage,
running my application caused an out of memory expection (heapspace).
So i ran the code under jProfiler and have seen symptoms of a memory
leak (or symptoms of not using the resolver correctly ;-).
It my understanding that when marshalling/unmarshalling, the
XMLClassDescriptorResolver should be caching the descriptors for use
in subsequent operations. Then on the subsequent operation (marshal /
unmarshall) it shouldn't have to create those descriptors again
(assumming the same class/xml complex type is being used).
But what i've seen in jProfiler is that after one run of my code there
have been 3828 XMLFieldDescriptorImpl objects created (triggering
garbage collection a few times doesn't change that number either) then
when i run it again (no restart, just second iteration) it finishes
what it's doing and the count of XMLFieldDescriptorImpl objects is now
7725!
It increases with each iteration, and after 10 iterations is 38901
objects, taking up 3.4 megabytes in memory - after a while this gets
so big it causes the memory exception.
So it looks like it's created new XMLFieldDescriptorImpl for the same
classes and xml complex types i used the first time around.
Has anyone else seen this behaviour before? Am i doing somthing wrong
when creating / using the resolver?
Here's the code i use to contruct a resolver (sorry about the
formatting):
private XMLClassDescriptorResolver createResolver(Class clazz){
XMLClassDescriptorResolver cdResolver =
(XMLClassDescriptorResolver)ClassDescriptorResolverFactory.createClassDesc
riptorResolver(BindingType.XML);
cdResolver.setClassLoader(Thread.currentThread().getContextClassLoader());
try {
cdResolver.loadClassDescriptors(clazz.getPackage().toString());
return cdResolver;
} catch (ResolverException e) {
logger.warn(e.getMessage() + ":"+clazz.getPackage().toString(),e);
}
return null;
}
as you can see from the above code i create a resolver for each Class
i marshal/unmarshall and store it for subsequent uses of the same
operation (so i'm not creating a new resolver each time as it might
appear)
Thank you for your time
Simon Lord
---------------------------------------------------------------------
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
---------------------------------------------------------------------
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
---------------------------------------------------------------------
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
---------------------------------------------------------------------
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
---------------------------------------------------------------------
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
---------------------------------------------------------------------
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
