Ah, I hadn't seen noticed/remembered this dealt with a single huge
base64 string. That's just plain bad design for the web service... but
no surprise if Exchange is involved.
No, you can't handle this cleanly with JiBX or any other data binding
tool I'm aware of. The only way you *could* handle it would be by
getting an XMLStreamReader and reading the data directly, with . The
XMLStreamReader interface provides next() and getTextCharacters()
methods which in theory could be used to get this text a chunk at a
time. Assuming the parser involved actually implements these as
intended, you can get a block of text at a time and run the base64
decoding on that block, then move on to the next block. A lot of work to
implement, though.
Andrew, you might want to try switching to ADB or JiBX anyway, at least
to try for an easier solution than working directly with the
XMLStreamReader. Either of these should use less memory than XMLBeans
for the same data, so as long as your text strings are only very large
and not unbounded these might let you squeak by.
- Dennis
Andreas Veithen wrote:
Dennis,
I think that the Web service is actually Microsoft Exchange, so
changing it is not an option. Does JiBX support caching a base64 value
(which is not represented using MTOM) on disk instead of memory?
Andrew,
You might still want to give MTOM a try. Many servers switch to MTOM
if the request is sent as MTOM. I can't believe the Microsoft Exchange
can only return inline base64.
Regards,
Andreas
On Mon, May 18, 2009 at 13:22, Dennis Sosnoski <[email protected]> wrote:
Hi Andrew,
Your best starting point is to get rid of XMLBeans. XMLBeans always stores
raw XML in memory, so there's no way to avoid the memory issues with large
messages.
ADB would avoid some of the overhead, in that it would convert the XML
message to an object graph, which would typically be a factor of 2-5x
smaller than the raw XML data (depending on the type of data in your
message).
JiBX would do at least as well as ADB in terms of the reduced-size object
graph. If you have some way of breaking up the response data into more
easily-digestible chunks, JiBX would also allow you to do piece-meal
processing of the message (by creating a fake collection, for instance,
where your data objects expose an addXXX() method which just writes the
object being added to some backing store rather than adding it to an
in-memory collection). Piece-meal processing is the only way you can
dramatically decrease your memory usage and handle messages of effectively
unlimited size without a problem.
- Dennis
--
Dennis M. Sosnoski
SOA and Web Services in Java
Axis2 Training and Consulting
http://www.sosnoski.com - http://www.sosnoski.co.nz
Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117
Andrew Bruno wrote:
Mr J,
But its just one data element that I need, so there is no concept many
rows, etc.
As a matter of fact, I am breaking up the call in 2 calls, one for all the
metadata, and one for the actual mime content.
i.e.
<m:ResponseMessages>
<m:GetItemResponseMessage ResponseClass="Success">
<m:ResponseCode>NoError</m:ResponseCode>
<m:Items>
<t:Message>
<t:MimeContent CharacterSet="UTF-8">RnJvbTogTGVlIF..... large
mime base64 encoded email......... 120Meg++ of encoded data</t:MimeContent>
....
I only ever request one message at a time. Its just that when the
MimeContent is greater then 10Meg, OutOfmemory occurs.
It appears to be the case because the parse loads all the content into
memory.
In theory, this element can have data of any size coming back.
On Mon, May 18, 2009 at 6:54 PM, J. Hondius <[email protected]
<mailto:[email protected]>> wrote:
I'd think about a different design of the webservice calls.
You should try to avoid real big results.
Split into more calls.
Something like:
One to get a overview list
Another to get details
Or:
One call to get the size like the SQL count() does
combined with
Add parameters to your to limit the result: like start_at,
number_of_results
my 2c
Andrew Bruno schreef:
Hello all
I was wondering how some of you may be dealing with web
service calls
that result in extremely large data responses?
I have been struggling in trying to change the way the parsing
of the
XML response works, as I am getting out of memory errors
java.lang.OutOfMemoryError: Java heap space
at
org.apache.xmlbeans.impl.store.CharUtil.allocate(CharUtil.java:397)
at
org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:506)
at
org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:419)
at
org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:489)
at
org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:2911)
at
org.apache.xmlbeans.impl.store.Cur$CurLoadContext.stripText(Cur.java:3113)
at
org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:3126)
at
org.apache.xmlbeans.impl.store.Locale.loadXMLStreamReader(Locale.java:1154)
at
org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:843)
at
org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:826)
at
org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:231)
.....
Is there a way to change the parser to use a temp file rather then
trying to buffer the response in memory?
Should I be directing this question to the developers list?
Or should I be thinking differently on solving this problem?
Please any ideas :(
Thank you
Andrew
