Hi Andrew,
Your best starting point is to get rid of XMLBeans. XMLBeans always
stores raw XML in memory, so there's no way to avoid the memory issues
with large messages.
ADB would avoid some of the overhead, in that it would convert the XML
message to an object graph, which would typically be a factor of 2-5x
smaller than the raw XML data (depending on the type of data in your
message).
JiBX would do at least as well as ADB in terms of the reduced-size
object graph. If you have some way of breaking up the response data into
more easily-digestible chunks, JiBX would also allow you to do
piece-meal processing of the message (by creating a fake collection, for
instance, where your data objects expose an addXXX() method which just
writes the object being added to some backing store rather than adding
it to an in-memory collection). Piece-meal processing is the only way
you can dramatically decrease your memory usage and handle messages of
effectively unlimited size without a problem.
- Dennis
--
Dennis M. Sosnoski
SOA and Web Services in Java
Axis2 Training and Consulting
http://www.sosnoski.com - http://www.sosnoski.co.nz
Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117
Andrew Bruno wrote:
Mr J,
But its just one data element that I need, so there is no concept many
rows, etc.
As a matter of fact, I am breaking up the call in 2 calls, one for all
the metadata, and one for the actual mime content.
i.e.
<m:ResponseMessages>
<m:GetItemResponseMessage ResponseClass="Success">
<m:ResponseCode>NoError</m:ResponseCode>
<m:Items>
<t:Message>
<t:MimeContent CharacterSet="UTF-8">RnJvbTogTGVlIF.....
large mime base64 encoded email......... 120Meg++ of encoded
data</t:MimeContent>
....
I only ever request one message at a time. Its just that when the
MimeContent is greater then 10Meg, OutOfmemory occurs.
It appears to be the case because the parse loads all the content into
memory.
In theory, this element can have data of any size coming back.
On Mon, May 18, 2009 at 6:54 PM, J. Hondius <[email protected]
<mailto:[email protected]>> wrote:
I'd think about a different design of the webservice calls.
You should try to avoid real big results.
Split into more calls.
Something like:
One to get a overview list
Another to get details
Or:
One call to get the size like the SQL count() does
combined with
Add parameters to your to limit the result: like start_at,
number_of_results
my 2c
Andrew Bruno schreef:
Hello all
I was wondering how some of you may be dealing with web
service calls
that result in extremely large data responses?
I have been struggling in trying to change the way the parsing
of the
XML response works, as I am getting out of memory errors
java.lang.OutOfMemoryError: Java heap space
at
org.apache.xmlbeans.impl.store.CharUtil.allocate(CharUtil.java:397)
at
org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:506)
at
org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:419)
at
org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:489)
at
org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:2911)
at
org.apache.xmlbeans.impl.store.Cur$CurLoadContext.stripText(Cur.java:3113)
at
org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:3126)
at
org.apache.xmlbeans.impl.store.Locale.loadXMLStreamReader(Locale.java:1154)
at
org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:843)
at
org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:826)
at
org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:231)
.....
Is there a way to change the parser to use a temp file rather then
trying to buffer the response in memory?
Should I be directing this question to the developers list?
Or should I be thinking differently on solving this problem?
Please any ideas :(
Thank you
Andrew
