Simon, Please see inline.
Werner > -----Ursprüngliche Nachricht----- > Von: Simon Lord [mailto:[EMAIL PROTECTED] > Gesendet: Donnerstag, 15. März 2007 16:03 > An: [email protected] > Betreff: Re: AW: [castor-user] [XML] XMLClassDescriptorResolver > XMLFieldDescriptorImpl cache possible memory leak > > Hi there, > > I've attached my small eclipse project to the JIRA issue i created: > http://jira.codehaus.org/browse/CASTOR-1910 > > Yes i was using the default memory in the test that i posted. The fact > is though that after the first iteration the memory should (in my mind) > stay the same as it's *exactly* the same piece of xml on the next > iteration - so it should need to load more descriptors. > > Setting the memory to something larger doesn't fix the problem it just > takes a little longer to run out of memory! As it never 'levels off' > anywhere. > > The cache of descriptors inside XMLClassDescriptorResolverImpl.java uses > java.util.HashMap which is known to not be thread safe > (http://java.sun.com/j2se/1.5.0/docs/api/java/util/HashMap.html) as it's > not synchronized. Whether that matters or not with the way the resolver > is used/setup i do not know - but i'd rather play it safe and use it in > a thread safe fashion ;-) I think our desire to play it (over-ambitiously thread-safe) is actually causing the problem here. Let me just repeat what Leander just said below: > > However, in my opinion XMLClassDescriptorResolver does not need to be > > thread safe. Only the initial initialization of such a resolver instance > > needs to be thread safe. Later, whenever you use that instance in > > marshaller and unmarshaller instances, they will only use the resolver > > in a "read-only" way, and not modify the resolver, thus no thread > > synchronization is neccessary. Right? There's two distinctive phases in the life-cycle of an XMLClassdescriptorResolver: a) initialization. b) usage for (un)marshalling. Once a) has been completed, b) is 100% thread-safe. In other words, the usage of one (and only one) XMLClassDescriptorResolver for your multi-threaded application is 100% okay. You just need to makes sure that initialization happens *before* usage. > > Let me know how you get on. > > > Thanks > > Simon Lord > > > M.-Leander Reimer wrote: > > Hello Simon, > > > > I am curious now, because I am also using the XMLClassDescriptorResolver > > approach and our application that has many threads where unmarshalling > > and marshalling happens, and I would like to know whether there is a > > problem with that approach. > > > > However, in my opinion XMLClassDescriptorResolver does not need to be > > thread safe. Only the initial initialization of such a resolver instance > > needs to be thread safe. Later, whenever you use that instance in > > marshaller and unmarshaller instances, they will only use the resolver > > in a "read-only" way, and not modify the resolver, thus no thread > > synchronization is neccessary. Right? > > > > Regarding your tests and the possible memory leak. Obviously, caching > > things will require more memory. I am curious about your JVM settings > > for the test that used the resolver. Have you increased the heap size or > > did you run with the default 64MB? > > > > > Iteration 7 : 6984 millis [u:31 m:31 f:5 t:63] > > > > Total memory is at 63, so I guess the garbage collector kicked in, thus > > the increase in time, but no more memory was available, thus the error. > > > > Still I would like to see your unit test, and perhaps I will also do a > > little testing with my implementation to check the memory consumption. > > > > Regards, > > Leander > > > > > > > > Simon Lord schrieb: > >> Hello again Werner, > >> > >> Thanks for the reply. > >> > >> In the application i help to develop it is impossible for us to use > only > >> one resolver, as we have many threads needing to perform > (un)marshalling > >> operations and the implementation of XMLClassDescriptorResolver is not > >> thread safe (it uses HashMaps) and we cannot afford to block until the > >> resolver is available. > >> > >> However, i've spent a little time creating a unit test that i believe > >> shows the symptoms of a memory leak. It also shows that using the > >> resolver does not speed up the (un)marshaling, in fact slowing it down > a > >> touch. > >> > >> The test makes ten iterations of 100 unmarshals then marshals of the > >> same bit of xml. Without a resolver the test completes but with the > >> resolver the test fails on the 7th iteration with a java heap space > >> exception. > >> > >> I'm going to create new issue on JIRA where i can attach source and > >> jars or a zip of the eclipse projct i've created if that easier, to > avoid > >> sending them to everyone on this list. > >> > >> Here is the output from my test: > >> > >> u = avg unmarshal time (millis) > >> m = avg marshal time (millis) > >> f = free mem (mb) > >> t = total mem (mb) > >> > >> > >> Test setup: 10 of 100 unmarshal then marshal, useResolver = false > >> Iteration 1 : 2047 millis [u:13 m:13 f:0 t:1] > >> Iteration 2 : 1265 millis [u:7 m:7 f:1 t:1] > >> Iteration 3 : 1250 millis [u:6 m:6 f:0 t:1] > >> Iteration 4 : 1188 millis [u:7 m:7 f:0 t:1] > >> Iteration 5 : 1219 millis [u:6 m:6 f:0 t:1] > >> Iteration 6 : 1171 millis [u:6 m:6 f:0 t:1] > >> Iteration 7 : 1204 millis [u:6 m:6 f:0 t:1] > >> Iteration 8 : 1234 millis [u:6 m:6 f:1 t:1] > >> Iteration 9 : 1156 millis [u:6 m:6 f:0 t:1] > >> Iteration 10 : 1203 millis [u:6 m:6 f:1 t:1] > >> Completed successfully > >> > >> Test setup: 10 of 100 unmarshal then marshal, useResolver = true > >> Iteration 1 : 1235 millis [u:6 m:6 f:3 t:13] > >> Iteration 2 : 1265 millis [u:8 m:8 f:1 t:20] > >> Iteration 3 : 1532 millis [u:6 m:6 f:7 t:32] > >> Iteration 4 : 1890 millis [u:11 m:11 f:18 t:54] > >> Iteration 5 : 1797 millis [u:7 m:7 f:10 t:54] > >> Iteration 6 : 2047 millis [u:11 m:11 f:3 t:54] > >> Iteration 7 : 6984 millis [u:31 m:31 f:5 t:63] > >> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > >> > >> > >> Thank you for your time > >> > >> Simon Lord > >> > >> > >> > >> > >> Werner Guttmann wrote: > >>> Hmm, you should *not* be creating a resolver for each root class, but > >>> one resolver only for all classes potentially used throughout your > >>> (un)marshalling operations. Once the resolver has loaded - in your > >>> case - the descriptors for all classes found in that package, it will > >>> cache them. > >>> And the Unmarshaller will be able to retrieve the cached descriptors > >>> as needed. > >>> > >>> I hope this explains things in more detail. > >>> > >>> Werner > >>> > >>>> -----Ursprüngliche Nachricht----- > >>>> Von: Simon Lord [mailto:[EMAIL PROTECTED] > >>>> Gesendet: Donnerstag, 15. März 2007 10:22 > >>>> An: [email protected] > >>>> Betreff: Re: [castor-user] [XML] XMLClassDescriptorResolver > >>>> XMLFieldDescriptorImpl cache possible memory leak > >>>> > >>>> Hi Steven, > >>>> > >>>> Thanks for your speedy reply. > >>>> > >>>> I understand that there is an object graph underneath the root > object, > >>>> hense i understand it needs to load the 3828 descriptors for the > object > >>>> graph underneath my root object. What i don't understand is why when > >>>> marshalling/unmarshalling the same root class (albeit with possibly > >>>> different values e.g., two objects of type person might have > different > >>>> names but it's still the same name field) the resolver is loading > more > >>>> descriptors but should in my opinion be using the same descriptors, > it > >>>> is only the values that have changed not the class/fields. I > understand > >>>> that in a complex object graph it may need to load more descriptors > for > >>>> different objects due to it having a larger object graph - but my > >>>> object > >>>> is staying relatively similiar each time - therefor i would expect > some > >>>> growth but not constant growth with each iteration. > >>>> > >>>> Also i'm not trying to create a resolver for every class - i'm > creating > >>>> a resolver for each root class and reusing them (and yes i am setting > >>>> the resolver on the marshaller/unmashaller ;-). I don't think it is > >>>> thread safe either - so i've implemented it in a thread safe manner > >>>> > >>>> I hope this sounds more like i'm implementing/understanding it all in > >>>> the right way. > >>>> > >>>> I'm off to write a unit test now and do some further investigation :) > >>>> > >>>> Thanks > >>>> > >>>> Simon Lord > >>>> > >>>> > >>>> > >>>> Steven Dolg wrote: > >>>>> Hi Simon, > >>>>> > >>>>> The 3828 FieldDescriptors indicate that the object graph your > >>>>> (un)marshalling contains some more classes than the one you're > >>>>> creating > >>>>> the ClassDescriptorResolver for. > >>>>> So the (Un)marshaller asks the ClassDescriptorResolver to find all > >>>>> descriptors necessary for the whole object graph (not just the root > >>>>> object). > >>>>> This might lead to creating different ClassDescriptorResolvers (for > >>>>> different root classes) that might contain almost identical > >>>>> descriptors > >>>>> (for all other objects in the object graph). > >>>>> > >>>>> It shouldn't be necessary to create a ClassDescriptorResolver for > each > >>>>> class individually so I'd suggest using only one resolver. (I'm not > >>>>> entirely sure whether this is thread-safe or not) > >>>>> > >>>>> And do not forget to set the created ClassDescriptorResolver at the > >>>>> (Un)marshaller... ;-) > >>>>> Unmarshaller unm = new Unmarshaller(...); > >>>>> unm.setResolver(cdr); > >>>>> > >>>>> Regards, > >>>>> Steven > >>>>> > >>>>> > >>>>> ----- Original Message ----- From: "Simon Lord" > >>>> <[EMAIL PROTECTED]> > >>>>> To: <[email protected]> > >>>>> Sent: Wednesday, March 14, 2007 6:44 PM > >>>>> Subject: [castor-user] [XML] XMLClassDescriptorResolver > >>>>> XMLFieldDescriptorImpl cache possible memory leak > >>>>> > >>>>> > >>>>>> Afternoon all, > >>>>>> > >>>>>> I've recently moved from using an old version of Castor (0.9.xxx) > to > >>>>>> the new stable 1.1. Mainly as i'd like to use the descriptor > caching. > >>>>>> > >>>>>> However, after moving to castor 1.1 and creating an > >>>>>> XMLClassDescriptorResolver as per the xml-best-practices webpage, > >>>>>> running my application caused an out of memory expection > (heapspace). > >>>>>> > >>>>>> So i ran the code under jProfiler and have seen symptoms of a > memory > >>>>>> leak (or symptoms of not using the resolver correctly ;-). > >>>>>> > >>>>>> It my understanding that when marshalling/unmarshalling, the > >>>>>> XMLClassDescriptorResolver should be caching the descriptors for > use > >>>>>> in subsequent operations. Then on the subsequent operation (marshal > / > >>>>>> unmarshall) it shouldn't have to create those descriptors again > >>>>>> (assumming the same class/xml complex type is being used). > >>>>>> > >>>>>> But what i've seen in jProfiler is that after one run of my code > >>>>>> there > >>>>>> have been 3828 XMLFieldDescriptorImpl objects created (triggering > >>>>>> garbage collection a few times doesn't change that number either) > >>>>>> then > >>>>>> when i run it again (no restart, just second iteration) it finishes > >>>>>> what it's doing and the count of XMLFieldDescriptorImpl objects is > >>>>>> now > >>>>>> 7725! > >>>>>> > >>>>>> It increases with each iteration, and after 10 iterations is 38901 > >>>>>> objects, taking up 3.4 megabytes in memory - after a while this > gets > >>>>>> so big it causes the memory exception. > >>>>>> > >>>>>> So it looks like it's created new XMLFieldDescriptorImpl for the > same > >>>>>> classes and xml complex types i used the first time around. > >>>>>> > >>>>>> > >>>>>> Has anyone else seen this behaviour before? Am i doing somthing > wrong > >>>>>> when creating / using the resolver? > >>>>>> > >>>>>> > >>>>>> Here's the code i use to contruct a resolver (sorry about the > >>>>>> formatting): > >>>>>> > >>>>>> private XMLClassDescriptorResolver createResolver(Class clazz){ > >>>>>> > >>>>>> XMLClassDescriptorResolver cdResolver = > >>>>>> > >>>> > (XMLClassDescriptorResolver)ClassDescriptorResolverFactory.createClassDesc > >>>> > >>>> riptorResolver(BindingType.XML); > >>>>>> > >>>>>> > >>>> > cdResolver.setClassLoader(Thread.currentThread().getContextClassLoader()); > >>>> > >>>>>> try { > >>>>>> cdResolver.loadClassDescriptors(clazz.getPackage().toString()); > >>>>>> return cdResolver; > >>>>>> } catch (ResolverException e) { > >>>>>> logger.warn(e.getMessage() + > ":"+clazz.getPackage().toString(),e); > >>>>>> } > >>>>>> return null; > >>>>>> } > >>>>>> > >>>>>> as you can see from the above code i create a resolver for each > Class > >>>>>> i marshal/unmarshall and store it for subsequent uses of the same > >>>>>> operation (so i'm not creating a new resolver each time as it might > >>>>>> appear) > >>>>>> > >>>>>> Thank you for your time > >>>>>> > >>>>>> Simon Lord > >>>>>> > >>>>>> ------------------------------------------------------------------- > -- > >>>>>> To unsubscribe from this list please visit: > >>>>>> > >>>>>> http://xircles.codehaus.org/manage_email > >>>>> > >>>>> -------------------------------------------------------------------- > - > >>>>> To unsubscribe from this list please visit: > >>>>> > >>>>> http://xircles.codehaus.org/manage_email > >>>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe from this list please visit: > >>>> > >>>> http://xircles.codehaus.org/manage_email > >>> > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe from this list please visit: > >>> > >>> http://xircles.codehaus.org/manage_email > >>> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe from this list please visit: > >> > >> http://xircles.codehaus.org/manage_email > >> > > > > --------------------------------------------------------------------- > > To unsubscribe from this list please visit: > > > > http://xircles.codehaus.org/manage_email > > > > --------------------------------------------------------------------- > To unsubscribe from this list please visit: > > http://xircles.codehaus.org/manage_email --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email
