I guess we'll be having a look at this in detail ..... Werner
> -----Ursprüngliche Nachricht----- > Von: Simon Lord [mailto:[EMAIL PROTECTED] > Gesendet: Donnerstag, 15. März 2007 16:36 > An: [email protected] > Betreff: Re: AW: AW: [castor-user] [XML] XMLClassDescriptorResolver > XMLFieldDescriptorImpl cache possible memory leak > > Werner, > > My test program on jira (http://jira.codehaus.org/browse/CASTOR-1910) > creates a single resolver and is only one thread therefor i do not > synchronize access to it for the test - and still it eats memory! > > However, i've seen that the number of descriptors does change over time > (shown in jProfiler) even though my test is using the same bit of xml > each time - so if the number of desriptors is changing then the cache is > still being *written* to and read from therefor not 100% thread safe. > > e.g., after 60 iterations (each iteration is 100 marshal and unmarshals) > of my test app (with -Xms128M -Xmx512M) there are 1.9 million instances > of XMLFieldDescriptorImpl taking up 156Mb of memory! > > Iteration 50 : 15328 millis [u:93 m:93 f:68 t:508] > Iteration 51 : 15703 millis [u:94 m:94 f:66 t:508] > Iteration 52 : 16344 millis [u:67 m:67 f:63 t:508] > Iteration 53 : 20375 millis [u:97 m:97 f:67 t:508] > Iteration 54 : 19578 millis [u:130 m:130 f:35 t:508] > Iteration 55 : 27015 millis [u:68 m:68 f:33 t:508] > Iteration 56 : 30672 millis [u:237 m:237 f:30 t:508] > Iteration 57 : 27985 millis [u:107 m:107 f:27 t:508] > Iteration 58 : 28188 millis [u:142 m:142 f:2 t:508] > Iteration 59 : 38953 millis [u:178 m:178 f:1 t:508] > Iteration 60 : 43844 millis [u:257 m:257 f:0 t:508] > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > > not to mention the time for the iterations!! > > > Thanks > > Simon Lord > > > > > > Werner Guttmann wrote: > > Simon, > > > > Please see inline. > > > > Werner > > > >> -----Ursprüngliche Nachricht----- > >> Von: Simon Lord [mailto:[EMAIL PROTECTED] > >> Gesendet: Donnerstag, 15. März 2007 16:03 > >> An: [email protected] > >> Betreff: Re: AW: [castor-user] [XML] XMLClassDescriptorResolver > >> XMLFieldDescriptorImpl cache possible memory leak > >> > >> Hi there, > >> > >> I've attached my small eclipse project to the JIRA issue i created: > >> http://jira.codehaus.org/browse/CASTOR-1910 > >> > >> Yes i was using the default memory in the test that i posted. The fact > >> is though that after the first iteration the memory should (in my mind) > >> stay the same as it's *exactly* the same piece of xml on the next > >> iteration - so it should need to load more descriptors. > >> > >> Setting the memory to something larger doesn't fix the problem it just > >> takes a little longer to run out of memory! As it never 'levels off' > >> anywhere. > >> > >> The cache of descriptors inside XMLClassDescriptorResolverImpl.java > uses > >> java.util.HashMap which is known to not be thread safe > >> (http://java.sun.com/j2se/1.5.0/docs/api/java/util/HashMap.html) as > it's > >> not synchronized. Whether that matters or not with the way the resolver > >> is used/setup i do not know - but i'd rather play it safe and use it in > >> a thread safe fashion ;-) > > > > I think our desire to play it (over-ambitiously thread-safe) is actually > causing the problem here. Let me just repeat what Leander just said below: > > > >>> However, in my opinion XMLClassDescriptorResolver does not need to be > >>> thread safe. Only the initial initialization of such a resolver > instance > >>> needs to be thread safe. Later, whenever you use that instance in > >>> marshaller and unmarshaller instances, they will only use the resolver > >>> in a "read-only" way, and not modify the resolver, thus no thread > >>> synchronization is neccessary. Right? > > > > There's two distinctive phases in the life-cycle of an > XMLClassdescriptorResolver: > > > > a) initialization. > > b) usage for (un)marshalling. > > > > Once a) has been completed, b) is 100% thread-safe. In other words, the > usage of one (and only one) XMLClassDescriptorResolver for your multi- > threaded application is 100% okay. You just need to makes sure that > initialization happens *before* usage. > > > >> Let me know how you get on. > >> > >> > >> Thanks > >> > >> Simon Lord > >> > >> > >> M.-Leander Reimer wrote: > >>> Hello Simon, > >>> > >>> I am curious now, because I am also using the > XMLClassDescriptorResolver > >>> approach and our application that has many threads where unmarshalling > >>> and marshalling happens, and I would like to know whether there is a > >>> problem with that approach. > >>> > >>> However, in my opinion XMLClassDescriptorResolver does not need to be > >>> thread safe. Only the initial initialization of such a resolver > instance > >>> needs to be thread safe. Later, whenever you use that instance in > >>> marshaller and unmarshaller instances, they will only use the resolver > >>> in a "read-only" way, and not modify the resolver, thus no thread > >>> synchronization is neccessary. Right? > >>> > >>> Regarding your tests and the possible memory leak. Obviously, caching > >>> things will require more memory. I am curious about your JVM settings > >>> for the test that used the resolver. Have you increased the heap size > or > >>> did you run with the default 64MB? > >>> > >>> > Iteration 7 : 6984 millis [u:31 m:31 f:5 t:63] > >>> > >>> Total memory is at 63, so I guess the garbage collector kicked in, > thus > >>> the increase in time, but no more memory was available, thus the > error. > >>> > >>> Still I would like to see your unit test, and perhaps I will also do a > >>> little testing with my implementation to check the memory consumption. > >>> > >>> Regards, > >>> Leander > >>> > >>> > >>> > >>> Simon Lord schrieb: > >>>> Hello again Werner, > >>>> > >>>> Thanks for the reply. > >>>> > >>>> In the application i help to develop it is impossible for us to use > >> only > >>>> one resolver, as we have many threads needing to perform > >> (un)marshalling > >>>> operations and the implementation of XMLClassDescriptorResolver is > not > >>>> thread safe (it uses HashMaps) and we cannot afford to block until > the > >>>> resolver is available. > >>>> > >>>> However, i've spent a little time creating a unit test that i believe > >>>> shows the symptoms of a memory leak. It also shows that using the > >>>> resolver does not speed up the (un)marshaling, in fact slowing it > down > >> a > >>>> touch. > >>>> > >>>> The test makes ten iterations of 100 unmarshals then marshals of the > >>>> same bit of xml. Without a resolver the test completes but with the > >>>> resolver the test fails on the 7th iteration with a java heap space > >>>> exception. > >>>> > >>>> I'm going to create new issue on JIRA where i can attach source and > >>>> jars or a zip of the eclipse projct i've created if that easier, to > >> avoid > >>>> sending them to everyone on this list. > >>>> > >>>> Here is the output from my test: > >>>> > >>>> u = avg unmarshal time (millis) > >>>> m = avg marshal time (millis) > >>>> f = free mem (mb) > >>>> t = total mem (mb) > >>>> > >>>> > >>>> Test setup: 10 of 100 unmarshal then marshal, useResolver = false > >>>> Iteration 1 : 2047 millis [u:13 m:13 f:0 t:1] > >>>> Iteration 2 : 1265 millis [u:7 m:7 f:1 t:1] > >>>> Iteration 3 : 1250 millis [u:6 m:6 f:0 t:1] > >>>> Iteration 4 : 1188 millis [u:7 m:7 f:0 t:1] > >>>> Iteration 5 : 1219 millis [u:6 m:6 f:0 t:1] > >>>> Iteration 6 : 1171 millis [u:6 m:6 f:0 t:1] > >>>> Iteration 7 : 1204 millis [u:6 m:6 f:0 t:1] > >>>> Iteration 8 : 1234 millis [u:6 m:6 f:1 t:1] > >>>> Iteration 9 : 1156 millis [u:6 m:6 f:0 t:1] > >>>> Iteration 10 : 1203 millis [u:6 m:6 f:1 t:1] > >>>> Completed successfully > >>>> > >>>> Test setup: 10 of 100 unmarshal then marshal, useResolver = true > >>>> Iteration 1 : 1235 millis [u:6 m:6 f:3 t:13] > >>>> Iteration 2 : 1265 millis [u:8 m:8 f:1 t:20] > >>>> Iteration 3 : 1532 millis [u:6 m:6 f:7 t:32] > >>>> Iteration 4 : 1890 millis [u:11 m:11 f:18 t:54] > >>>> Iteration 5 : 1797 millis [u:7 m:7 f:10 t:54] > >>>> Iteration 6 : 2047 millis [u:11 m:11 f:3 t:54] > >>>> Iteration 7 : 6984 millis [u:31 m:31 f:5 t:63] > >>>> Exception in thread "main" java.lang.OutOfMemoryError: Java heap > space > >>>> > >>>> > >>>> Thank you for your time > >>>> > >>>> Simon Lord > >>>> > >>>> > >>>> > >>>> > >>>> Werner Guttmann wrote: > >>>>> Hmm, you should *not* be creating a resolver for each root class, > but > >>>>> one resolver only for all classes potentially used throughout your > >>>>> (un)marshalling operations. Once the resolver has loaded - in your > >>>>> case - the descriptors for all classes found in that package, it > will > >>>>> cache them. > >>>>> And the Unmarshaller will be able to retrieve the cached descriptors > >>>>> as needed. > >>>>> > >>>>> I hope this explains things in more detail. > >>>>> > >>>>> Werner > >>>>> > >>>>>> -----Ursprüngliche Nachricht----- > >>>>>> Von: Simon Lord [mailto:[EMAIL PROTECTED] > >>>>>> Gesendet: Donnerstag, 15. März 2007 10:22 > >>>>>> An: [email protected] > >>>>>> Betreff: Re: [castor-user] [XML] XMLClassDescriptorResolver > >>>>>> XMLFieldDescriptorImpl cache possible memory leak > >>>>>> > >>>>>> Hi Steven, > >>>>>> > >>>>>> Thanks for your speedy reply. > >>>>>> > >>>>>> I understand that there is an object graph underneath the root > >> object, > >>>>>> hense i understand it needs to load the 3828 descriptors for the > >> object > >>>>>> graph underneath my root object. What i don't understand is why > when > >>>>>> marshalling/unmarshalling the same root class (albeit with possibly > >>>>>> different values e.g., two objects of type person might have > >> different > >>>>>> names but it's still the same name field) the resolver is loading > >> more > >>>>>> descriptors but should in my opinion be using the same descriptors, > >> it > >>>>>> is only the values that have changed not the class/fields. I > >> understand > >>>>>> that in a complex object graph it may need to load more descriptors > >> for > >>>>>> different objects due to it having a larger object graph - but my > >>>>>> object > >>>>>> is staying relatively similiar each time - therefor i would expect > >> some > >>>>>> growth but not constant growth with each iteration. > >>>>>> > >>>>>> Also i'm not trying to create a resolver for every class - i'm > >> creating > >>>>>> a resolver for each root class and reusing them (and yes i am > setting > >>>>>> the resolver on the marshaller/unmashaller ;-). I don't think it is > >>>>>> thread safe either - so i've implemented it in a thread safe manner > >>>>>> > >>>>>> I hope this sounds more like i'm implementing/understanding it all > in > >>>>>> the right way. > >>>>>> > >>>>>> I'm off to write a unit test now and do some further investigation > :) > >>>>>> > >>>>>> Thanks > >>>>>> > >>>>>> Simon Lord > >>>>>> > >>>>>> > >>>>>> > >>>>>> Steven Dolg wrote: > >>>>>>> Hi Simon, > >>>>>>> > >>>>>>> The 3828 FieldDescriptors indicate that the object graph your > >>>>>>> (un)marshalling contains some more classes than the one you're > >>>>>>> creating > >>>>>>> the ClassDescriptorResolver for. > >>>>>>> So the (Un)marshaller asks the ClassDescriptorResolver to find all > >>>>>>> descriptors necessary for the whole object graph (not just the > root > >>>>>>> object). > >>>>>>> This might lead to creating different ClassDescriptorResolvers > (for > >>>>>>> different root classes) that might contain almost identical > >>>>>>> descriptors > >>>>>>> (for all other objects in the object graph). > >>>>>>> > >>>>>>> It shouldn't be necessary to create a ClassDescriptorResolver for > >> each > >>>>>>> class individually so I'd suggest using only one resolver. (I'm > not > >>>>>>> entirely sure whether this is thread-safe or not) > >>>>>>> > >>>>>>> And do not forget to set the created ClassDescriptorResolver at > the > >>>>>>> (Un)marshaller... ;-) > >>>>>>> Unmarshaller unm = new Unmarshaller(...); > >>>>>>> unm.setResolver(cdr); > >>>>>>> > >>>>>>> Regards, > >>>>>>> Steven > >>>>>>> > >>>>>>> > >>>>>>> ----- Original Message ----- From: "Simon Lord" > >>>>>> <[EMAIL PROTECTED]> > >>>>>>> To: <[email protected]> > >>>>>>> Sent: Wednesday, March 14, 2007 6:44 PM > >>>>>>> Subject: [castor-user] [XML] XMLClassDescriptorResolver > >>>>>>> XMLFieldDescriptorImpl cache possible memory leak > >>>>>>> > >>>>>>> > >>>>>>>> Afternoon all, > >>>>>>>> > >>>>>>>> I've recently moved from using an old version of Castor (0.9.xxx) > >> to > >>>>>>>> the new stable 1.1. Mainly as i'd like to use the descriptor > >> caching. > >>>>>>>> However, after moving to castor 1.1 and creating an > >>>>>>>> XMLClassDescriptorResolver as per the xml-best-practices webpage, > >>>>>>>> running my application caused an out of memory expection > >> (heapspace). > >>>>>>>> So i ran the code under jProfiler and have seen symptoms of a > >> memory > >>>>>>>> leak (or symptoms of not using the resolver correctly ;-). > >>>>>>>> > >>>>>>>> It my understanding that when marshalling/unmarshalling, the > >>>>>>>> XMLClassDescriptorResolver should be caching the descriptors for > >> use > >>>>>>>> in subsequent operations. Then on the subsequent operation > (marshal > >> / > >>>>>>>> unmarshall) it shouldn't have to create those descriptors again > >>>>>>>> (assumming the same class/xml complex type is being used). > >>>>>>>> > >>>>>>>> But what i've seen in jProfiler is that after one run of my code > >>>>>>>> there > >>>>>>>> have been 3828 XMLFieldDescriptorImpl objects created (triggering > >>>>>>>> garbage collection a few times doesn't change that number either) > >>>>>>>> then > >>>>>>>> when i run it again (no restart, just second iteration) it > finishes > >>>>>>>> what it's doing and the count of XMLFieldDescriptorImpl objects > is > >>>>>>>> now > >>>>>>>> 7725! > >>>>>>>> > >>>>>>>> It increases with each iteration, and after 10 iterations is > 38901 > >>>>>>>> objects, taking up 3.4 megabytes in memory - after a while this > >> gets > >>>>>>>> so big it causes the memory exception. > >>>>>>>> > >>>>>>>> So it looks like it's created new XMLFieldDescriptorImpl for the > >> same > >>>>>>>> classes and xml complex types i used the first time around. > >>>>>>>> > >>>>>>>> > >>>>>>>> Has anyone else seen this behaviour before? Am i doing somthing > >> wrong > >>>>>>>> when creating / using the resolver? > >>>>>>>> > >>>>>>>> > >>>>>>>> Here's the code i use to contruct a resolver (sorry about the > >>>>>>>> formatting): > >>>>>>>> > >>>>>>>> private XMLClassDescriptorResolver createResolver(Class clazz){ > >>>>>>>> > >>>>>>>> XMLClassDescriptorResolver cdResolver = > >>>>>>>> > >> > (XMLClassDescriptorResolver)ClassDescriptorResolverFactory.createClassDesc > >>>>>> riptorResolver(BindingType.XML); > >>>>>>>> > >> > cdResolver.setClassLoader(Thread.currentThread().getContextClassLoader()); > >>>>>>>> try { > >>>>>>>> cdResolver.loadClassDescriptors(clazz.getPackage().toString()); > >>>>>>>> return cdResolver; > >>>>>>>> } catch (ResolverException e) { > >>>>>>>> logger.warn(e.getMessage() + > >> ":"+clazz.getPackage().toString(),e); > >>>>>>>> } > >>>>>>>> return null; > >>>>>>>> } > >>>>>>>> > >>>>>>>> as you can see from the above code i create a resolver for each > >> Class > >>>>>>>> i marshal/unmarshall and store it for subsequent uses of the same > >>>>>>>> operation (so i'm not creating a new resolver each time as it > might > >>>>>>>> appear) > >>>>>>>> > >>>>>>>> Thank you for your time > >>>>>>>> > >>>>>>>> Simon Lord > >>>>>>>> > >>>>>>>> ----------------------------------------------------------------- > -- > >> -- > >>>>>>>> To unsubscribe from this list please visit: > >>>>>>>> > >>>>>>>> http://xircles.codehaus.org/manage_email > >>>>>>> ------------------------------------------------------------------ > -- > >> - > >>>>>>> To unsubscribe from this list please visit: > >>>>>>> > >>>>>>> http://xircles.codehaus.org/manage_email > >>>>>>> > >>>>>> ------------------------------------------------------------------- > -- > >>>>>> To unsubscribe from this list please visit: > >>>>>> > >>>>>> http://xircles.codehaus.org/manage_email > >>>>> > >>>>> -------------------------------------------------------------------- > - > >>>>> To unsubscribe from this list please visit: > >>>>> > >>>>> http://xircles.codehaus.org/manage_email > >>>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe from this list please visit: > >>>> > >>>> http://xircles.codehaus.org/manage_email > >>>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe from this list please visit: > >>> > >>> http://xircles.codehaus.org/manage_email > >>> > >> --------------------------------------------------------------------- > >> To unsubscribe from this list please visit: > >> > >> http://xircles.codehaus.org/manage_email > > > > > > --------------------------------------------------------------------- > > To unsubscribe from this list please visit: > > > > http://xircles.codehaus.org/manage_email > > > > --------------------------------------------------------------------- > To unsubscribe from this list please visit: > > http://xircles.codehaus.org/manage_email --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email
