I
don't know enough about the JVM implementation to speak definitively, but I have
seen the very same behavior you're referring to. After enlisting the help
of IBM support on the issue, we both came to the conclusion that the JVM's stack
had a bunch of XML-related objects in it and nothing we could do
programmatically forced those objects to be collected. It eventually goes
down, but it took maybe 30 minutes to an hour, after which more runs would bump
the memory usage back up.
I was
very skeptical of this answer and had thought that once the request was
done and those objects I'd created (in the DOM, particularly) went out of scope,
I'd see the JVM's memory usage go back down. It just never did happen that
way. After getting into the tweaking of the JVM heap, I learned about how
the JVM recovers memory and it has more to do with what percentage of the heap
is taken up, what space needs to be allocated to new objects being created, and
what your maximum and minimum settings are. I am fairly confident that I
understand this process well enough now to say that you'll only see a reduction
in memory usage if your heap is 70-80% full and the system isn't continuously
creating new objects. If the heap is getting close to 70-80%, but is able
to reclaim enough to fulfill the need for new objects, then the memory used by
old objects won't ever be collected until a certain amount of time has
passed. I guess the garbage collector just figures that if you don't need
the memory those old objects are using, then there's no reason to spend
processor cycles in collecting it.
I'm
sure there are those on this list that are much more qualified to expound on
these issues than me, but this is just my own personal experience with this
problem.
After
months of trying different things and getting some help from IBM, I believe that
one of the downfalls to XML and XSLT-based applications being used in large (and
very scalable) applications is that they simply require a massive amount of
RAM to support. IBM support flatly said that our use of XML and XSLT was
the reason for slow performance and not any hardware or JVM-related
issues.
If
someone else has found a way to do anything meaningful over an extended period
of time (i.e. not one transform per minute, like in many testing and development
situations) without having a bare minimum of 1-2 GBs of RAM, I'd be interested
to hear your solution.
Thanks!
Jon
Brisbin
=+=+=+=+=+=+=+=+=+=+=+=
www.jbrisbin.net
417.682.6157
(h/o)
417.850.1644 (cel)
-----Original Message-----
From: Damon Horrell [mailto:[EMAIL PROTECTED]
Sent: Thursday, May 22, 2003 6:54 PM
To: 'Jon Brisbin'; [EMAIL PROTECTED]
Subject: RE: Reusing Templates doesn't free memoryI don't think its just garbage collection. The call to transformer.transform doesn't free all the memory that it uses up. I expected it to do its thing and then free up all the temporary structures used after it has finished. It is not a memory leak though because it doesn't keep using more and more each time I call it with the same document but instead reaches a limit, so it must be some sort of deliberate cache.The memory is an issue for me because it slows the whole process down heaps which defeats the purpose of using the same Templates object to process multiple documents in the first place. (This is just running on a desktop PC with 0.5Gb RAM and after running a few documents through this the process is using over 300Mb which is never releases, and this causes it to start running very slowly.)Its not a problem that the memory usage gets high while it is running but I would like it to drop back down after the transfromation has finished. Any ideas?Thanks, Damon
-----Original Message-----
From: Jon Brisbin [mailto:[EMAIL PROTECTED]
Sent: Friday, 23 May 2003 11:19 a.m.
To: Damon Horrell; [EMAIL PROTECTED]
Subject: RE: Reusing Templates doesn't free memoryI recently worked on a large-scale XML and XSLT application framework and one of the things we've found is that the garbage collector can be a problem when it comes to using XSLT in a large application with loads of requests. You may end up having to use a thread-safe object that reuses things as much as possible. It's a pain, I know, but it seems to help. You should also DEFINATELY tweak your JVM settings to fiddle with the heap size. If you don't, don't be surprised to start seeing OutOfMemory errors as your applications and transformations get larger.BTW- 100 MBs of memory usage in a nice-sized XSLT application is really not that much :-) If I could keep the memory usage under 100 MBs on my framework application under even moderate load, I'd be tickled! :-) Under a heavy load, I was able to push memory usage up to about 1.5 GBs, sustained :-)
Thanks!
Jon Brisbin
=+=+=+=+=+=+=+=+=+=+=+=
www.jbrisbin.net
417.682.6157 (h/o)
417.850.1644 (cel)
-----Original Message-----
From: Damon Horrell [mailto:[EMAIL PROTECTED]
Sent: Thursday, May 22, 2003 6:08 PM
To: '[EMAIL PROTECTED]'
Subject: Reusing Templates doesn't free memoryHi,I have written a little server program that uses a Templates object to avoid reparsing my XSLT each time I want to process a document. I am finding that it chews up an awful lots of memory (like 100+ Mb) though which does not get released when the transformation is finished. If I process the same document a few times the memory increases and then remains static after about the 3rd time, but it grows higher still if I process other documents.I am not sure if this is because it is caching the source document or something. I have experimented with noth XPathContext.release and SourceTreeManager but to no effect.I don't want anything other than the Templates object to be retained in memory after each transformation is run.Does anybody know where the memory goes and how I can release it?Has anyone else sucessfully written a similar application to efficiently process multiple source documents through the same transformation?Thanks, Damon
BEGIN:VCARD VERSION:2.1 N:Brisbin;Jon FN:Jon Brisbin NICKNAME:Jon ORG:Jbrisbin.Net TEL;WORK;VOICE:(417) 682-6157 TEL;HOME;VOICE:(417) 682-6157 TEL;CELL;VOICE:(417) 850-1644 ADR;WORK:;;394 S.E. 1st Lane;Lamar;MO;64759;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:394 S.E. 1st Lane=0D=0ALamar, MO 64759=0D=0AUnited States of America URL;WORK:http://www.jbrisbin.net EMAIL;PREF;INTERNET:[EMAIL PROTECTED] REV:20030520T160909Z END:VCARD
