Yup, I have put much time into figuring out and squashing these issues.
Comments inline.
On Tue, Mar 22, 2011 at 4:02 AM, Niels <[email protected]> wrote:
> I have been trying to figure out a memory leak in app-schema (and I think,
> perhaps even in geoserver in general).
>
> The problem is that the app-schema unit tests run out of memory when they
> are ran in a batch by maven, never when they are run by themselves. While
> maven is running the tests, data is accumulated on the heap and at some
> point it will run out and crash. This is a serious bug.
>
> I have figured out quite a bit about it, using Java Memory Analyzer.
> The data that is accumulating is mainly XSD schema information
> (XSDElementDeclarationImpl etc...).
> But no features or types or kept alive.
>
> The other thing I figured out is that the GeoServerImpl , Catalog,
> ResourcePool, etc.. objects are not disposed of. For every test that has
> been run, these objects are kept on the heap! Although, the abstract test
> class does get rid of them. My first assumption was that these objects where
> somehow also keeping XSD information in memory, but I was wrong: it is the
> other way around.
>
> I have included a screenshot in the attachment that shows the path that
> keeps the GeoserverImpl alive, through the XSD classes. I have also added a
> second screenshot of another GeoserverImpl instance's path ( in the same
> memory dump). If you look at the addresses you can also see where these
> paths are the same and where they split up.
> XSD schema's can be imported in to each other, and that is what is going on
> here.
>
> Here is a summary of my findings
> 1. Almost all XSD classes are singletons, therefore static and kept alive.
>
Done intentionally. They have to be cached since they are so expensive to
create.
> 2. org.geoserver.wfs.xml.v1_1_0.WFS is an exception this rule, it is
> initiated and contains, indirectly, a link to the running geoserverimpl.
>
Yes this is an issue, one i very much want to kill. I did some work to fix
this but app-schema soon because a blocker. It is kind of a separate issue
but long story short when we build up a schema object we need to iterate
over every type since app-schema types have dependencies among them. If we
can figure out how to process those dependencies rather than just build a
schema from all types we get the side affect of getting rid of the
WFSConfiguration singleton... which is not a source of much pain.
3. For every instance of geoserverimpl, a new
> org.geoserver.wfs.xml.v1_1_0.WFS is created and *imported* in all the other
> (static) XSD classes (OGC, GML, etc).
>
It should just be GML, but yeah, part of the work is to fix this as well..
reversing the importing dependency so as not to modify the gml schema.
> 4. These imports accumulate - For each test a new import is added.
>
Take a look at GML.buildSchema and you should see two adapters which attempt
to manage this. ReferencingDirectiveLeakPreventer which removes duplicate
import statements so they do not accumulate. And
SubstitutionGroupLeakPreventer which prevents the same problem but with the
gml _Feature substitution group.
>
> But what I cannot find, is where and how this
> org.geoserver.wfs.xml.v1_1_0.WFS is imported in to the other XSD classes. Is
> there anyone who can point me in the right direction?
>
> Also, even if I do find the place where this happens - the question remains
> what to do.
> 1. Just try to "undo" the import when closing down geoserver? In that case
> I definitely need to figure out how the import is happening in the first
> place.
> 2. Just clear out all of the static XSD classes, and rebuilt them each
> time.
>
>
Definitely (1) or be prepared to wait hours for your tests to finish :)
Again if we can figure out the app-schema issue this will get a lot better.
Again the issue being that when we build an XSDSchema object for a complex
feature type, we need some way to traverse the feature type dependency
graph.. and pull in any dependencies into the XSDSchema object rather than
just build a schema with them all. Also note that this is an issue
that severely limits GeoServer to containing many layers.
Regards
>
> --
> *Niels Charlier*
>
> Software Engineer
> CSIRO Earth Science and Resource Engineering
> Phone: +61 8 6436 8914
>
> Australian Resources Research Centre
> 26 Dick Perry Avenue, Kensington WA 6151
>
>
> ------------------------------------------------------------------------------
> Enable your software for Intel(R) Active Management Technology to meet the
> growing manageability and security demands of your customers. Businesses
> are taking advantage of Intel(R) vPro (TM) technology - will your software
> be a part of the solution? Download the Intel(R) Manageability Checker
> today! http://p.sf.net/sfu/intel-dev2devmar
> _______________________________________________
> Geoserver-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/geoserver-devel
>
>
--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.
------------------------------------------------------------------------------
Enable your software for Intel(R) Active Management Technology to meet the
growing manageability and security demands of your customers. Businesses
are taking advantage of Intel(R) vPro (TM) technology - will your software
be a part of the solution? Download the Intel(R) Manageability Checker
today! http://p.sf.net/sfu/intel-dev2devmar
_______________________________________________
Geotools-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/geotools-devel
