On May 8, 2008, at 08:40, Jean-François El Fouly wrote:
Hi
Jeremias Maerki a écrit :
And my next problem is to find a way to force memory recycling
after this long and hefty FOP processing, but until further
investigated this is OT ;-)
You probably didn't get my hint earlier but with the new image
loading framework you should actually get away with lower memory
settings. In my tests I have been able to produce PDF with little
text and many images with 40MB of VM memory which wasn't possible
with 0.94 and earlier.
Well, I got the hint, but it seems in contradiction with what I read.
There are, of course, other factors to take into account than simply
document/image sizes.
Which Java VM are you using? Practically every time someone tells us
about memory/GC issues, it appears they are using an implementation
other than Sun (IBM, GNU...)
Up to now, we still have to find out why precisely non-Sun VMs have
difficulties with FOP...
What options does the Java VM offer to tweak the GC? What options
does it use by default?
So to take the picture from a bit higher:
- all XSL-FO transformation + FOP generation now work OK.
- this generates 20-30 documents (chapters) for a grand total of
about 150 Mb, to be bound together by iText.
- source XML is 1.5 Mb
- 1011 PNG images for a total of 151 Mb, the largest image is 715 kb.
Now the figures:
- XML -> XSL-FO transformation + FOP generation take 15 minutes on
a pretty decent DELL Server (running Debian 4.0) having all the
physical RAM possible (staging server for several customers)
How large would the resulting FO-files be if you dump them to the
filesystem? The XML by itself says very little. From a 1.5MB XML, you
could get a FO of a few KB or one of 26MB, depending on the stylesheet.
Does the stylesheet adhere to XSLT best practices? Does it generate a
lot of redundant fo:blocks, fo:inlines?
- JVM has 2000 Mb (which is BTW the grand max on this processor/
server/OS/JVM architecture)
<snip />
On my end, that has proven to be more than enough to generate one
page-sequence with a table of 15 columns, spanning 500+ pages. (Note:
only text-content, no images; more a test to check the memory usage
without doing anything special, just a whole lot of FOs)
If I try to investigate memory usage using Runtime.getRuntime
().getFreeMemory() and logging the figures with log4j, these are
the figures I get:
- before XSLT + FOP: 1900 Mb free/2000 Mb
- end of XSLT + FOP: 241 Mb free
Yikes! That looks troublesome indeed... :/
- set FopFactory instance to null as a desperate hint to the GC
that FOP objects could be/should be recycled
- I force garbage collection using System.gc() [OK, in an
application server this is a brute force approach, but could not
see a more clever maneuver ATM]
A nit, for the record: There is no such thing as 'forcing garbage
collection'. The most you can do with System.gc() is indicate to the
VM that it should run the GC as soon as possible. Admitted, most
implementations do run the algorithm virtually immediately upon
execution of the statement, but the Java spec does not mandate such
behavior. In theory, if the VM is too busy, it could still postpone
the actual GC-run, until it acquires the necessary resources...
<snip />
Now I don't take runtime.getXXXMemory() for bible word but at least
it "looks like" the Xalan + FOP subsystem hogs 1500 Mb of RAM which
I cannot recover.
So I hired the team member who's competent in profiler usage next
week but I must say at the moment I'm still stuck :-(
If you're not on a Sun VM, then I have a very vague feeling that he's
going to discover the issue to be related to arrays, a special type
of object, but I could be wrong about this. Someone once reported
that the VM seemed to hold on to a lot of arrays. When profiling, he
discovered that the arrays were referenced nowhere, but still the GC
did not clean them up.
Of course I've made my homework and read the f...riendly manual
before daring to ask.
Did I miss any important indication ?
I don't think so, but it seems we might do well by putting some of
the info concerning JVM/GC implementation we have gathered so far, on
the website or a Wiki.
Cheers
Andreas
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
