Hi Aditya, The best way is to use the smooks mediator to process huge XML payloads. Here is the sample for the smooks mediator. [1]
[1] https://docs.wso2.com/display/ESB490/Sample+658%3A+Huge+XML+Message+Processing+with+Smooks+Mediator Thanks, Madhawa On Wed, Mar 14, 2018 at 11:19 PM, aditya shivankar < [email protected]> wrote: > Respected Sir, > > Removed the logger, but still not able to process large files. > Requirement is to process files of size upto GB's . > Please Guide. > > With Regards, > Aditya > > On Wed, Mar 14, 2018 at 8:08 PM, Ruwan Abeykoon <[email protected]> wrote: > >> Hi Adithya, >> Can you try this without the log mediator. >> Log mediator is usually tries to decode the payload. We call that a >> content aware mediator. The content aware mediators has the issue decoding >> large payloads. >> >> Cheers, >> Ruwan >> >> On Wed, Mar 14, 2018 at 6:43 PM, aditya shivankar < >> [email protected]> wrote: >> >>> Respected Sir, >>> >>> Forgot to mention ,Sometimes I am getting OutOfMemoryError as well >>> >>> java.lang.OutOfMemoryError: Java heap space >>> Dumping heap to C:/wso2ei-6.1.1/wso2ei-6.1.1/repository/logs/heap-dump.hprof >>> ... >>> Unable to create C:/wso2ei-6.1.1/wso2ei-6.1.1/r >>> epository/logs/heap-dump.hprof: File exists >>> [2018-03-14 17:16:01,228] [EI-Core] ERROR - NativeWorkerPool Uncaught >>> exception >>> java.lang.OutOfMemoryError: Java heap space >>> at java.util.Arrays.copyOf(Unknown Source) >>> at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown >>> Source) >>> at java.lang.AbstractStringBuilder.append(Unknown Source) >>> at java.lang.StringBuffer.append(Unknown Source) >>> at org.apache.log4j.helpers.PatternParser$LiteralPatternConvert >>> er.format(PatternParser.java:419) >>> >>> With Regards, >>> Aditya >>> >>> On Wed, Mar 14, 2018 at 4:46 PM, aditya shivankar < >>> [email protected]> wrote: >>> >>>> added the missing attachment from previous mail >>>> >>>> On Wed, Mar 14, 2018 at 4:44 PM, aditya shivankar < >>>> [email protected]> wrote: >>>> >>>>> Respected Sir/Madam, >>>>> >>>>> I am able to achieve below file processing using vfs, and datamapper. >>>>> But when trying to process files more than size of 8 MB or more the flow >>>>> is >>>>> failing with out of memory. Please guide >>>>> >>>>> step 1. I am using vfs to read one xml file from one location. >>>>> step 2. Then using datamapper to convert the xml file to csv. >>>>> step 3. Then writing the converted output to another location. >>>>> >>>>> >>>>> <?xml version="1.0" encoding="UTF-8"?> >>>>> <proxy name="FlatFileProxy" startOnLoad="true" transports="vfs" xmlns=" >>>>> http://ws.apache.org/ns/synapse";> >>>>> <target> >>>>> <inSequence> >>>>> <log level="custom"> >>>>> <property name="property_name" value=""Inside >>>>> FlatFile ***************************************** ""/> >>>>> </log> >>>>> <property name="FORCE_SC_ACCEPTED" scope="axis2" >>>>> type="STRING" value="true"/> >>>>> <property name="REST_URL_POSTFIX" scope="axis2" >>>>> type="STRING" value=""/> >>>>> <datamapper config="gov:datamapper/xmlToCsv1.dmc" >>>>> inputSchema="gov:datamapper/xmlToCsv1_inputSchema.json" >>>>> inputType="XML" outputSchema="gov:datamapper/xmlToCsv1_outputSchema.json" >>>>> outputType="CSV"/> >>>>> <property expression="fn:concat(fn:subst >>>>> ring-after(get-property('MessageID'), 'urn:uuid:'), '.csv')" >>>>> name="transport.vfs.ReplyFileName" scope="transport" type="STRING"/> >>>>> <property name="OUT_ONLY" scope="default" type="STRING" >>>>> value="true"/> >>>>> <property name="messageType" scope="axis2" type="STRING" >>>>> value="application/csv"/> >>>>> <property action="remove" name="LAST_MODIFIED" >>>>> scope="transport"/> >>>>> <send> >>>>> <endpoint> >>>>> <address uri="vfs:file:///C:/Flatfile/output"/> >>>>> </endpoint> >>>>> </send> >>>>> </inSequence> >>>>> <outSequence/> >>>>> <faultSequence/> >>>>> </target> >>>>> <parameter name="transport.PollInterval">50ms</parameter> >>>>> <parameter name="transport.vfs.FileURI">f >>>>> ile:///C:/Flatfile/input</parameter> >>>>> <parameter name="transport.vfs.ContentTyp >>>>> e">application/xml</parameter> >>>>> <parameter name="transport.vfs.ActionAfte >>>>> rProcess">MOVE</parameter> >>>>> <parameter name="transport.vfs.MoveAfterF >>>>> ailure">file:///C:/Flatfile/failure</parameter> >>>>> <parameter name="transport.vfs.ActionAfte >>>>> rFailure">MOVE</parameter> >>>>> <parameter name="transport.vfs.FileNamePa >>>>> ttern">.*\.xml</parameter> >>>>> <parameter name="transport.vfs.Locking">disable</parameter> >>>>> <parameter name="transport.vfs.MoveAfterP >>>>> rocess">file:///C:/Flatfile/orgFilesProcessedSuccessfully</parameter> >>>>> </proxy> >>>>> >>>>> >>>>> sometimes it is not able to delete .lock files from input folder. >>>>> sometimes timeout error. >>>>> >>>>> etc.. >>>>> >>>>> Sample input attached. >>>>> >>>>> With regards, >>>>> Aditya >>>>> >>>> >>>> >>> >>> _______________________________________________ >>> Dev mailing list >>> [email protected] >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Madhawa Gunasekara* Senior Software Engineer WSO2 Inc.; http://wso2.com lean.enterprise.middleware mobile: +94 719411002 <+94+719411002> blog: *http://madhawa-gunasekara.blogspot.com <http://madhawa-gunasekara.blogspot.com>* linkedin: *http://lk.linkedin.com/in/mgunasekara <http://lk.linkedin.com/in/mgunasekara>*
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
