Re: Most efficient way to unmarshall a large web service response

[bob anderson]

Hi,

I'm not sure if this is helpful.
I use Camel Spring XML DSL to update a Master Data table from a suppliers
database. The Master Data is accessible to us via a webservice. The server
this runs on has only 500Mb of heap memory allocated. I use CXF to handle
the Webservice call and had issues with the payload size - its
approximately 180,000 records with approx size of 200Mb (so smaller than
yours). I am not technical and cobbled the solution from reading Camel
documentation, examples and copying snippets from the web..

I found a reference on stack overflow (can't remember the link) that
suggested disabling the payload limits
<!-- disable XML payload limits -->
    <bean
class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
        <property name="targetClass" value="java.lang.System" />
        <property name="targetMethod" value="setProperty" />
        <property name="arguments">
            <list>
                <value>org.apache.cxf.stax.maxChildElements</value>
                <value>1000000</value>
            </list>
        </property>
    </bean>

AND I stream the the data splitting on the individual record within the
response, also increased the continuation timeout value - as I was timing
out too.
I set the exchange body to the soap request and then make the SOAP call to
the web service as follows..

<!-- set the SOAP header action -->
<setHeader headerName="SOAPAction">
             <simple>http://MyWebServiceURL/GetMasterdataAction</simple>
  </setHeader>
        <!-- Call the webservice -->
         <to uri="cxf://http://MyWebServiceURL/Latest/Data.svc?
                wsdlURL=http://MyWebServiceURL/Latest/Data.svc?Wsdl&amp;
                serviceName={
http://wcf.isolutions.it/Their.API.Data/1.0/Data&amp;

dataFormat=PAYLOAD&amp;allowStreaming=true&amp;continuationTimeout=80000&amp;
                defaultOperationName=GetMasterdataAction"/>


          <!-- split and tokenize the incoming stream of XML on Record
Element -->
            <split streaming="true" parallelProcessing="true"
executorServiceRef="ThreadPool">
              <xtokenize>//b:Record</xtokenize>
         <!-- do stuff here with the record -->
           </split>

The solution works..... takes some time though ....





On Fri, Dec 8, 2017 at 11:27 PM, Gokul J <gokuljaya...@icloud.com> wrote:

> Hi All,
>
>   So we use Apache camel in order to consume web services via spring WA
> and recently we are being plagued with OOM issues relating to heap size .
> The response is around 675 MB in size and we had to increase the heap by 5
> GB to successfully unmarshall it . We use vanilla jaxb data format . Is
> there anything we can do to reduce the heap usage even if it costs us some
> time ? Any ideas would be highly appreciated.
>
> Sent from my iPhone




-- 
Bob Anderson
+27 (0) 82 389 0335
[image: View my profile on LinkedIn]
<http://ng.linkedin.com/pub/bob-anderson/2/25/9b5>