Re: [Dev] Processing Big/Huge/Large files in wso2 using vfs.Facing issues

2018-03-14 Thread Madhawa Gunasekara 
Hi Aditya,

The best way is to use the smooks mediator to process huge XML payloads.
Here is the sample for the smooks mediator. [1]

[1]
https://docs.wso2.com/display/ESB490/Sample+658%3A+Huge+XML+Message+Processing+with+Smooks+Mediator

Thanks,
Madhawa

On Wed, Mar 14, 2018 at 11:19 PM, aditya shivankar <
shivankar.adit...@gmail.com> wrote:

> Respected Sir,
>
> Removed the logger, but still not able to process large files.
> Requirement is to process files of size upto GB's .
> Please Guide.
>
> With Regards,
> Aditya
>
> On Wed, Mar 14, 2018 at 8:08 PM, Ruwan Abeykoon  wrote:
>
>> Hi Adithya,
>> Can you try this without the log mediator.
>> Log mediator is usually tries to decode the payload. We call that a
>> content aware mediator. The content aware mediators has the issue decoding
>> large payloads.
>>
>> Cheers,
>> Ruwan
>>
>> On Wed, Mar 14, 2018 at 6:43 PM, aditya shivankar <
>> shivankar.adit...@gmail.com> wrote:
>>
>>> Respected Sir,
>>>
>>> Forgot to mention ,Sometimes I am getting OutOfMemoryError as well
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>> Dumping heap to C:/wso2ei-6.1.1/wso2ei-6.1.1/repository/logs/heap-dump.hprof
>>> ...
>>> Unable to create C:/wso2ei-6.1.1/wso2ei-6.1.1/r
>>> epository/logs/heap-dump.hprof: File exists
>>> [2018-03-14 17:16:01,228] [EI-Core] ERROR - NativeWorkerPool Uncaught
>>> exception
>>> java.lang.OutOfMemoryError: Java heap space
>>> at java.util.Arrays.copyOf(Unknown Source)
>>> at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown
>>> Source)
>>> at java.lang.AbstractStringBuilder.append(Unknown Source)
>>> at java.lang.StringBuffer.append(Unknown Source)
>>> at org.apache.log4j.helpers.PatternParser$LiteralPatternConvert
>>> er.format(PatternParser.java:419)
>>>
>>> With Regards,
>>> Aditya
>>>
>>> On Wed, Mar 14, 2018 at 4:46 PM, aditya shivankar <
>>> shivankar.adit...@gmail.com> wrote:
>>>
 added the missing attachment from previous mail

 On Wed, Mar 14, 2018 at 4:44 PM, aditya shivankar <
 shivankar.adit...@gmail.com> wrote:

> Respected Sir/Madam,
>
> I am able to achieve below file processing using vfs, and datamapper.
> But when trying to process files more than size of 8 MB or more the flow 
> is
> failing with out of memory. Please guide
>
> step 1. I am using vfs to read one xml file from one location.
> step 2. Then using datamapper to convert the xml file to csv.
> step 3. Then writing the converted output to another location.
>
>
> 
> http://ws.apache.org/ns/synapse";>
> 
> 
> 
> 
> 
>  type="STRING" value="true"/>
>  type="STRING" value=""/>
>  inputSchema="gov:datamapper/xmlToCsv1_inputSchema.json"
> inputType="XML" outputSchema="gov:datamapper/xmlToCsv1_outputSchema.json"
> outputType="CSV"/>
>  name="transport.vfs.ReplyFileName" scope="transport" type="STRING"/>
>  value="true"/>
>  value="application/csv"/>
>  scope="transport"/>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 50ms
> f
> ile:///C:/Flatfile/input
> application/xml
> MOVE
> file:///C:/Flatfile/failure
> MOVE
> .*\.xml
> disable
> file:///C:/Flatfile/orgFilesProcessedSuccessfully
> 
>
>
> sometimes it is not able to delete .lock files from input folder.
> sometimes timeout error.
>
> etc..
>
> Sample input attached.
>
> With regards,
> Aditya
>


>>>
>>> ___
>>> Dev mailing list
>>> Dev@wso2.org
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>>
>>
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
*Madhawa Gunasekara*
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware

mobile: +94 719411002 <+94+719411002>
blog: *http://madhawa-gunasekara.blogspot.com
*
linkedin: *http://lk.linkedin.com/in/mgunasekara
*
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] Processing Big/Huge/Large files in wso2 using vfs.Facing issues

2018-03-14 Thread aditya shivankar 
Respected Sir,

Removed the logger, but still not able to process large files.
Requirement is to process files of size upto GB's .
Please Guide.

With Regards,
Aditya

On Wed, Mar 14, 2018 at 8:08 PM, Ruwan Abeykoon  wrote:

> Hi Adithya,
> Can you try this without the log mediator.
> Log mediator is usually tries to decode the payload. We call that a
> content aware mediator. The content aware mediators has the issue decoding
> large payloads.
>
> Cheers,
> Ruwan
>
> On Wed, Mar 14, 2018 at 6:43 PM, aditya shivankar <
> shivankar.adit...@gmail.com> wrote:
>
>> Respected Sir,
>>
>> Forgot to mention ,Sometimes I am getting OutOfMemoryError as well
>>
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to C:/wso2ei-6.1.1/wso2ei-6.1.1/repository/logs/heap-dump.hprof
>> ...
>> Unable to create C:/wso2ei-6.1.1/wso2ei-6.1.1/r
>> epository/logs/heap-dump.hprof: File exists
>> [2018-03-14 17:16:01,228] [EI-Core] ERROR - NativeWorkerPool Uncaught
>> exception
>> java.lang.OutOfMemoryError: Java heap space
>> at java.util.Arrays.copyOf(Unknown Source)
>> at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
>> at java.lang.AbstractStringBuilder.append(Unknown Source)
>> at java.lang.StringBuffer.append(Unknown Source)
>> at org.apache.log4j.helpers.PatternParser$LiteralPatternConvert
>> er.format(PatternParser.java:419)
>>
>> With Regards,
>> Aditya
>>
>> On Wed, Mar 14, 2018 at 4:46 PM, aditya shivankar <
>> shivankar.adit...@gmail.com> wrote:
>>
>>> added the missing attachment from previous mail
>>>
>>> On Wed, Mar 14, 2018 at 4:44 PM, aditya shivankar <
>>> shivankar.adit...@gmail.com> wrote:
>>>
 Respected Sir/Madam,

 I am able to achieve below file processing using vfs, and datamapper.
 But when trying to process files more than size of 8 MB or more the flow is
 failing with out of memory. Please guide

 step 1. I am using vfs to read one xml file from one location.
 step 2. Then using datamapper to convert the xml file to csv.
 step 3. Then writing the converted output to another location.


 
 http://ws.apache.org/ns/synapse";>
 
 
 
 
 
 >>> type="STRING" value="true"/>
 >>> type="STRING" value=""/>
 >>> inputSchema="gov:datamapper/xmlToCsv1_inputSchema.json"
 inputType="XML" outputSchema="gov:datamapper/xmlToCsv1_outputSchema.json"
 outputType="CSV"/>
 >>> name="transport.vfs.ReplyFileName" scope="transport" type="STRING"/>
 >>> value="true"/>
 >>> value="application/csv"/>
 >>> scope="transport"/>
 
 
 
 
 
 
 
 
 
 50ms
 f
 ile:///C:/Flatfile/input
 application/xml
 MOVE
 file:///C:/Flatfile/failure
 MOVE
 .*\.xml
 disable
 file:///C:/Flatfile/orgFilesProcessedSuccessfully
 


 sometimes it is not able to delete .lock files from input folder.
 sometimes timeout error.

 etc..

 Sample input attached.

 With regards,
 Aditya

>>>
>>>
>>
>> ___
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] Processing Big/Huge/Large files in wso2 using vfs.Facing issues

2018-03-14 Thread Ruwan Abeykoon 
Hi Adithya,
Can you try this without the log mediator.
Log mediator is usually tries to decode the payload. We call that a content
aware mediator. The content aware mediators has the issue decoding large
payloads.

Cheers,
Ruwan

On Wed, Mar 14, 2018 at 6:43 PM, aditya shivankar <
shivankar.adit...@gmail.com> wrote:

> Respected Sir,
>
> Forgot to mention ,Sometimes I am getting OutOfMemoryError as well
>
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to C:/wso2ei-6.1.1/wso2ei-6.1.1/repository/logs/heap-dump.hprof
> ...
> Unable to create C:/wso2ei-6.1.1/wso2ei-6.1.1/repository/logs/heap-dump.hprof:
> File exists
> [2018-03-14 17:16:01,228] [EI-Core] ERROR - NativeWorkerPool Uncaught
> exception
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Unknown Source)
> at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
> at java.lang.AbstractStringBuilder.append(Unknown Source)
> at java.lang.StringBuffer.append(Unknown Source)
> at org.apache.log4j.helpers.PatternParser$LiteralPatternConverter.
> format(PatternParser.java:419)
>
> With Regards,
> Aditya
>
> On Wed, Mar 14, 2018 at 4:46 PM, aditya shivankar <
> shivankar.adit...@gmail.com> wrote:
>
>> added the missing attachment from previous mail
>>
>> On Wed, Mar 14, 2018 at 4:44 PM, aditya shivankar <
>> shivankar.adit...@gmail.com> wrote:
>>
>>> Respected Sir/Madam,
>>>
>>> I am able to achieve below file processing using vfs, and datamapper.
>>> But when trying to process files more than size of 8 MB or more the flow is
>>> failing with out of memory. Please guide
>>>
>>> step 1. I am using vfs to read one xml file from one location.
>>> step 2. Then using datamapper to convert the xml file to csv.
>>> step 3. Then writing the converted output to another location.
>>>
>>>
>>> 
>>> http://ws.apache.org/ns/synapse";>
>>> 
>>> 
>>> 
>>> 
>>> 
>>> >> type="STRING" value="true"/>
>>> >> type="STRING" value=""/>
>>> >> inputSchema="gov:datamapper/xmlToCsv1_inputSchema.json" inputType="XML"
>>> outputSchema="gov:datamapper/xmlToCsv1_outputSchema.json"
>>> outputType="CSV"/>
>>> >> name="transport.vfs.ReplyFileName" scope="transport" type="STRING"/>
>>> >> value="true"/>
>>> >> value="application/csv"/>
>>> >> scope="transport"/>
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 50ms
>>> f
>>> ile:///C:/Flatfile/input
>>> application/xml
>>> MOVE
>>> file:///C:/Flatfile/failure
>>> MOVE
>>> .*\.xml
>>> disable
>>> file:///C:/Flatfile/orgFilesProcessedSuccessfully
>>> 
>>>
>>>
>>> sometimes it is not able to delete .lock files from input folder.
>>> sometimes timeout error.
>>>
>>> etc..
>>>
>>> Sample input attached.
>>>
>>> With regards,
>>> Aditya
>>>
>>
>>
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] Processing Big/Huge/Large files in wso2 using vfs.Facing issues

2018-03-14 Thread aditya shivankar 
Respected Sir,

Forgot to mention ,Sometimes I am getting OutOfMemoryError as well

java.lang.OutOfMemoryError: Java heap space
Dumping heap to
C:/wso2ei-6.1.1/wso2ei-6.1.1/repository/logs/heap-dump.hprof ...
Unable to create
C:/wso2ei-6.1.1/wso2ei-6.1.1/repository/logs/heap-dump.hprof: File exists
[2018-03-14 17:16:01,228] [EI-Core] ERROR - NativeWorkerPool Uncaught
exception
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuffer.append(Unknown Source)
at
org.apache.log4j.helpers.PatternParser$LiteralPatternConverter.format(PatternParser.java:419)

With Regards,
Aditya

On Wed, Mar 14, 2018 at 4:46 PM, aditya shivankar <
shivankar.adit...@gmail.com> wrote:

> added the missing attachment from previous mail
>
> On Wed, Mar 14, 2018 at 4:44 PM, aditya shivankar <
> shivankar.adit...@gmail.com> wrote:
>
>> Respected Sir/Madam,
>>
>> I am able to achieve below file processing using vfs, and datamapper. But
>> when trying to process files more than size of 8 MB or more the flow is
>> failing with out of memory. Please guide
>>
>> step 1. I am using vfs to read one xml file from one location.
>> step 2. Then using datamapper to convert the xml file to csv.
>> step 3. Then writing the converted output to another location.
>>
>>
>> 
>> http://ws.apache.org/ns/synapse";>
>> 
>> 
>> 
>> 
>> 
>> > type="STRING" value="true"/>
>> > value=""/>
>> > inputSchema="gov:datamapper/xmlToCsv1_inputSchema.json" inputType="XML"
>> outputSchema="gov:datamapper/xmlToCsv1_outputSchema.json"
>> outputType="CSV"/>
>> > name="transport.vfs.ReplyFileName" scope="transport" type="STRING"/>
>> > value="true"/>
>> > value="application/csv"/>
>> > scope="transport"/>
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 50ms
>> f
>> ile:///C:/Flatfile/input
>> application/xml
>> MOVE
>> file:///C:/Flatfile/
>> failure
>> MOVE
>> .*\.xml
>> disable
>> file:///C:/Flatfile/orgFilesProcessedSuccessfully
>> 
>>
>>
>> sometimes it is not able to delete .lock files from input folder.
>> sometimes timeout error.
>>
>> etc..
>>
>> Sample input attached.
>>
>> With regards,
>> Aditya
>>
>
>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] Processing Big/Huge/Large files in wso2 using vfs.Facing issues

2018-03-14 Thread aditya shivankar 
Respected Sir/Madam,

I am able to achieve below file processing using vfs, and datamapper. But
when trying to process files more than size of 8 MB or more the flow is
failing with out of memory. Please guide

step 1. I am using vfs to read one xml file from one location.
step 2. Then using datamapper to convert the xml file to csv.
step 3. Then writing the converted output to another location.



http://ws.apache.org/ns/synapse";>





















50ms
file:///C:/Flatfile/input
application/xml
MOVE
file:///C:/Flatfile/failure
MOVE
.*\.xml
disable
file:///C:/Flatfile/orgFilesProcessedSuccessfully



sometimes it is not able to delete .lock files from input folder.
sometimes timeout error.

etc..

Sample input attached.

With regards,
Aditya
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev