Andreas, When I changed to MTOM, Exchange threw a Schema validation error.
My solution so far has been in implementing my own XML pull parser. When I detect the ELEMENT_TYPE MimeContent, I start to write the CDATA to a temporary file. I also cant believe that I have not found an easier way. If JiBX definately provided filesystem caching of a single large CDATA element, i.e. basically give me an inputstream, then I'd be wrapped. Dennis? Thanks guys Andrew On Mon, May 18, 2009 at 9:33 PM, Andreas Veithen <[email protected]>wrote: > Dennis, > > I think that the Web service is actually Microsoft Exchange, so > changing it is not an option. Does JiBX support caching a base64 value > (which is not represented using MTOM) on disk instead of memory? > > Andrew, > > You might still want to give MTOM a try. Many servers switch to MTOM > if the request is sent as MTOM. I can't believe the Microsoft Exchange > can only return inline base64. > > Regards, > > Andreas > > On Mon, May 18, 2009 at 13:22, Dennis Sosnoski <[email protected]> wrote: > > Hi Andrew, > > > > Your best starting point is to get rid of XMLBeans. XMLBeans always > stores > > raw XML in memory, so there's no way to avoid the memory issues with > large > > messages. > > > > ADB would avoid some of the overhead, in that it would convert the XML > > message to an object graph, which would typically be a factor of 2-5x > > smaller than the raw XML data (depending on the type of data in your > > message). > > > > JiBX would do at least as well as ADB in terms of the reduced-size object > > graph. If you have some way of breaking up the response data into more > > easily-digestible chunks, JiBX would also allow you to do piece-meal > > processing of the message (by creating a fake collection, for instance, > > where your data objects expose an addXXX() method which just writes the > > object being added to some backing store rather than adding it to an > > in-memory collection). Piece-meal processing is the only way you can > > dramatically decrease your memory usage and handle messages of > effectively > > unlimited size without a problem. > > > > - Dennis > > > > -- > > Dennis M. Sosnoski > > SOA and Web Services in Java > > Axis2 Training and Consulting > > http://www.sosnoski.com - http://www.sosnoski.co.nz > > Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117 > > > > > > > > Andrew Bruno wrote: > >> > >> Mr J, > >> > >> But its just one data element that I need, so there is no concept many > >> rows, etc. > >> > >> As a matter of fact, I am breaking up the call in 2 calls, one for all > the > >> metadata, and one for the actual mime content. > >> > >> i.e. > >> > >> <m:ResponseMessages> > >> <m:GetItemResponseMessage ResponseClass="Success"> > >> <m:ResponseCode>NoError</m:ResponseCode> > >> <m:Items> > >> <t:Message> > >> <t:MimeContent CharacterSet="UTF-8">RnJvbTogTGVlIF..... large > >> mime base64 encoded email......... 120Meg++ of encoded > data</t:MimeContent> > >> .... > >> > >> I only ever request one message at a time. Its just that when the > >> MimeContent is greater then 10Meg, OutOfmemory occurs. > >> > >> It appears to be the case because the parse loads all the content into > >> memory. > >> > >> In theory, this element can have data of any size coming back. > >> > >> > >> > >> > >> On Mon, May 18, 2009 at 6:54 PM, J. Hondius <[email protected] > >> <mailto:[email protected]>> wrote: > >> > >> I'd think about a different design of the webservice calls. > >> You should try to avoid real big results. > >> Split into more calls. > >> > >> Something like: > >> One to get a overview list > >> Another to get details > >> > >> Or: > >> One call to get the size like the SQL count() does > >> combined with > >> Add parameters to your to limit the result: like start_at, > >> number_of_results > >> > >> my 2c > >> > >> Andrew Bruno schreef: > >> > >> Hello all > >> > >> I was wondering how some of you may be dealing with web > >> service calls > >> that result in extremely large data responses? > >> > >> I have been struggling in trying to change the way the parsing > >> of the > >> XML response works, as I am getting out of memory errors > >> > >> java.lang.OutOfMemoryError: Java heap space > >> at > >> > org.apache.xmlbeans.impl.store.CharUtil.allocate(CharUtil.java:397) > >> at > >> > >> org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:506) > >> at > >> > >> org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:419) > >> at > >> > >> org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:489) > >> at > >> > >> org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:2911) > >> at > >> > >> > org.apache.xmlbeans.impl.store.Cur$CurLoadContext.stripText(Cur.java:3113) > >> at > >> > >> org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:3126) > >> at > >> > >> > org.apache.xmlbeans.impl.store.Locale.loadXMLStreamReader(Locale.java:1154) > >> at > >> > >> org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:843) > >> at > >> > >> org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:826) > >> at > >> > >> > > org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:231) > >> ..... > >> > >> Is there a way to change the parser to use a temp file rather > then > >> trying to buffer the response in memory? > >> > >> Should I be directing this question to the developers list? > >> > >> Or should I be thinking differently on solving this problem? > >> > >> Please any ideas :( > >> > >> Thank you > >> Andrew > >> > > >
