Dennis, I think that the Web service is actually Microsoft Exchange, so changing it is not an option. Does JiBX support caching a base64 value (which is not represented using MTOM) on disk instead of memory?
Andrew, You might still want to give MTOM a try. Many servers switch to MTOM if the request is sent as MTOM. I can't believe the Microsoft Exchange can only return inline base64. Regards, Andreas On Mon, May 18, 2009 at 13:22, Dennis Sosnoski <[email protected]> wrote: > Hi Andrew, > > Your best starting point is to get rid of XMLBeans. XMLBeans always stores > raw XML in memory, so there's no way to avoid the memory issues with large > messages. > > ADB would avoid some of the overhead, in that it would convert the XML > message to an object graph, which would typically be a factor of 2-5x > smaller than the raw XML data (depending on the type of data in your > message). > > JiBX would do at least as well as ADB in terms of the reduced-size object > graph. If you have some way of breaking up the response data into more > easily-digestible chunks, JiBX would also allow you to do piece-meal > processing of the message (by creating a fake collection, for instance, > where your data objects expose an addXXX() method which just writes the > object being added to some backing store rather than adding it to an > in-memory collection). Piece-meal processing is the only way you can > dramatically decrease your memory usage and handle messages of effectively > unlimited size without a problem. > > - Dennis > > -- > Dennis M. Sosnoski > SOA and Web Services in Java > Axis2 Training and Consulting > http://www.sosnoski.com - http://www.sosnoski.co.nz > Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117 > > > > Andrew Bruno wrote: >> >> Mr J, >> >> But its just one data element that I need, so there is no concept many >> rows, etc. >> >> As a matter of fact, I am breaking up the call in 2 calls, one for all the >> metadata, and one for the actual mime content. >> >> i.e. >> >> <m:ResponseMessages> >> <m:GetItemResponseMessage ResponseClass="Success"> >> <m:ResponseCode>NoError</m:ResponseCode> >> <m:Items> >> <t:Message> >> <t:MimeContent CharacterSet="UTF-8">RnJvbTogTGVlIF..... large >> mime base64 encoded email......... 120Meg++ of encoded data</t:MimeContent> >> .... >> >> I only ever request one message at a time. Its just that when the >> MimeContent is greater then 10Meg, OutOfmemory occurs. >> >> It appears to be the case because the parse loads all the content into >> memory. >> >> In theory, this element can have data of any size coming back. >> >> >> >> >> On Mon, May 18, 2009 at 6:54 PM, J. Hondius <[email protected] >> <mailto:[email protected]>> wrote: >> >> I'd think about a different design of the webservice calls. >> You should try to avoid real big results. >> Split into more calls. >> >> Something like: >> One to get a overview list >> Another to get details >> >> Or: >> One call to get the size like the SQL count() does >> combined with >> Add parameters to your to limit the result: like start_at, >> number_of_results >> >> my 2c >> >> Andrew Bruno schreef: >> >> Hello all >> >> I was wondering how some of you may be dealing with web >> service calls >> that result in extremely large data responses? >> >> I have been struggling in trying to change the way the parsing >> of the >> XML response works, as I am getting out of memory errors >> >> java.lang.OutOfMemoryError: Java heap space >> at >> org.apache.xmlbeans.impl.store.CharUtil.allocate(CharUtil.java:397) >> at >> >> org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:506) >> at >> >> org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:419) >> at >> >> org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:489) >> at >> >> org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:2911) >> at >> >> org.apache.xmlbeans.impl.store.Cur$CurLoadContext.stripText(Cur.java:3113) >> at >> >> org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:3126) >> at >> >> org.apache.xmlbeans.impl.store.Locale.loadXMLStreamReader(Locale.java:1154) >> at >> >> org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:843) >> at >> >> org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:826) >> at >> >> org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:231) >> ..... >> >> Is there a way to change the parser to use a temp file rather then >> trying to buffer the response in memory? >> >> Should I be directing this question to the developers list? >> >> Or should I be thinking differently on solving this problem? >> >> Please any ideas :( >> >> Thank you >> Andrew >> >
