I think we can improve deserialization performance in general and solve this specific issue if we make ObjectStore.objectMap (this is the AbstractReferenceMap causing trouble) transient. There's no value is saving and restoring this map. There's another map called "changes", which is a regular HashMap, and that needs to be preserved.
Now the implementation can be a little tricky, as "objectMap" is provided by the DI factory and we need to create an empty map of the same type on deserialization. This may require refactoring of ObjectStore creation flow. Should be doable though... Andrus On Feb 22, 2012, at 1:34 PM, Andrew Lindesay wrote: > Hi there; > > It does appear that the problem is in an Apache Commons abstract superclass. > What is happening is that whilst running; > > AbstractReferenceMap.doReadObject(..) > > ...the method... > > AbstractReferenceMap.checkCapacity(..) > > ...is invoked and it seems to be doubling the size of the Map's entry array > on de-serialization. Through sequential serialization + de-serialization > this yields the exponential growth in the memory consumption. > > It seems that the problem is caused by the 'threshold' value being calculated > after to populating the Map during de-serialization. > > I have re-built 3.0-STABLE with commons 3.2.1 and it still seems to exhibit > the same problem. > > The release cycles for commons seems to be quite long these days. Hmmm..... > > Regards; > > On 18/02/12 11:50 AM, Andrew Lindesay wrote: >> Hi Andrus; >> >> It does seem rather strange. >> >> I'm starting the application with maven (from Eclipse) and >> jetty-maven-plugin. No hot redeploy. No actual container per se. The >> class-loader should be the same so shouldn't be re-creating "isolated >> singletons or statics". I'm using a Filter to serialize-->deserialize >> the attributes on the HttpSession in order to test how some >> infrastructure handles serialization-->deserialization of the session >> attributes. >> >> If I explicitly don't serialize-->deserialize the ObjectContext then >> there is no problem; so serializing ObjectContext as the root object >> does seem to cause the problem. >> >> The incremental increase in the memory consumption seems to be roughly >> exponential in relation to the number of deserializations of the >> ObjectContext. That may be a clue; I may download Apache commons and see >> if I can see anything obvious there in the LRU map. Like you say though; >> it probably shouldn't be serializing this stuff anyway though. >> >> It looks like "SQLTemplateResourceManager" is created via reflection so >> it may be hard to profile for memory usage. >> >> I'll log a Jira when I get a moment. >> >> Regards; >> >> On 17/02/12 11:29 PM, Andrus Adamchik wrote: >>> Interesting... I didn't realize we have a static Velocity runtime in >>> the app. This must to be changed to instance variable with correct >>> scope, (and if needed shut down properly when ServerRuntime is >>> shutdown). Could you please log a Jira - we need to investigate. >>> >>> BTW, are you doing hot redeploy of your app? This in combination with >>> static Velocity runtime might produce the observed behavior. >>> >>> Andrus >>> >>> >>> On Feb 17, 2012, at 1:19 PM, Andrew Lindesay wrote: >>> >>>> Hi; >>>> >>>> It looks like it is holding onto lots (hundreds of megs) of; >>>> >>>> org.apache.commons.collections.map.AbstractHashedMap$HashEntry[] >>>> > org.apache.commons.collecitons.map.LRUMap >>>> >> org.apache.cayenne.access.jdbc.SQLTemplateResourceManager >>>> >>> ...velocity >>>> >>>> Not quite understanding yet... as it seems like this wouldn't be >>>> something stored on an object context. >>>> >>>> Regards; >>>> >>>> On 16/02/12 7:50 PM, Andrus Adamchik wrote: >>>>>>> an OutOfMemory -- even with substantial quantities of memory >>>>>>> available. >>>>> >>>>> This is a hint actually.. Maybe the problem is not heap size, but >>>>> perm gen size instead? Try increasing that: >>>>> >>>>> -XX:MaxPermSize=256m >>>>> >>>>> On Feb 16, 2012, at 9:44 AM, Andrus Adamchik wrote: >>>>> >>>>>> This is strange indeed. Haven't seen such behavior. May need some >>>>>> profiling on your end I guess. >>>>>> >>>>>> Andrus >>>>>> >>>>>> >>>>>> On Feb 14, 2012, at 1:30 PM, Andrew Lindesay wrote: >>>>>>> Hello; >>>>>>> >>>>>>> I have a little question on the ObjectContext in 3.0.2. >>>>>>> >>>>>>> I'm keeping an ObjectContext in an HttpSession instance's >>>>>>> attributes for re-use between request-response cycles. When I >>>>>>> serialize this ObjectContext instance, it (+ other objects) stores >>>>>>> out to< 8k in my test case, but then deserializing the >>>>>>> ObjectContext causes an OutOfMemory -- even with substantial >>>>>>> quantities of memory available. >>>>>>> >>>>>>> Has anybody else seen this or is there anything to be weary of >>>>>>> before I look into it further? >>>>>>> >>>>>>> Regards >>>>>>> >>>>>>> -- >>>>>>> Andrew Lindesay >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Andrew Lindesay >>>> www.silvereye.co.nz >>>> >>> >>> >> >> > > > -- > Andrew Lindesay > www.silvereye.co.nz >
