Re: [Dev] Streaming File transfer with File Inbound/File Connector

[Kasun Indrasiri]

IMO, file connector is quite simple to use compared to the vfs sender for
various file operations (for example just have a look at 'append' operation
which is support in both sender and connector).

On Wed, Apr 6, 2016 at 9:20 PM, Malaka Silva  wrote:

> Hi Kasun,
>
> I don't think we need to cover any functionality that is already available
> with ESB.
> Any use case that we need the vfs sender from connector?
>
> On Mon, Mar 14, 2016 at 10:40 AM, Kasun Indrasiri  wrote:
>
>>
>>
>> On Mon, Mar 14, 2016 at 9:57 AM, Malaka Silva  wrote:
>>
>>> Hi Kasun,
>>>
>>> I don't think we have the same functionality of VFS sender in file
>>> connector. We have only focused on use cases not covered with VFS transport.
>>>
>>
>> Ok. I think we need to make sure things are consistent across the
>> transport and the file connector. Can we please include this as an
>> improvement for the next version of the connector.
>>
>>
>>>
>> @Vivekananthan - Please confirm this?
>>>
>>> On Sat, Mar 12, 2016 at 4:39 PM, Kasun Indrasiri  wrote:
>>>
 This is great. Thanks a lot Malaka.
 Also, if we use the file connector as the outbound channel, will it
 work in the same way?

 On Tue, Mar 1, 2016 at 12:36 PM, Malaka Silva  wrote:

> Hi All,
>
> Please ignore my previous comments. This can be done with current
> implementation.
>
> if (builder instanceof DataSourceMessageBuilder &&
> "true".equals(streaming)) {
> dataSource = ManagedDataSourceFactory.create(new
> FileObjectDataSource(file, contentType));
> in = null;
> } else {
> in = new
> AutoCloseInputStream(file.getContent().getInputStream());
> dataSource = null;
> }
> ..
> OMElement documentElement;
> if (in != null) {
> documentElement = builder.processDocument(in,
> contentType, axis2MsgCtx);
> } else {
> documentElement =
> ((DataSourceMessageBuilder)builder).processDocument(dataSource,
> contentType, axis2MsgCtx);
> }
> ...
> if(dataSource != null) {
> dataSource.destroy();
> }
>
> This is
> ​because ​
> message builder able to build messages from
> ​​
> DataSource objects.
> ​
> ​
> DataSource​
>  by definition the data from a DataSource can be read
> ​
> multiple times,
> ​builders that implement ​
> this interface
> ​can​
>  avoid storing the message content in memory.
> ​ ​
> If a message builder implements this interface and the
> ​file/vfs
>  is able to provide the message payload as a data source, then the
> method defined by this interface should be preferred over the method
> defined by Builder.
> ​ This
>  helps optimizing
> ​PT​
>  with
> ​vfs/file​.
> The builder will typically expose the data source directly or
> indirectly through the returned OMElement, e.g. by adding to the tree an
> OMText or OMDataSource node referencing the data source.
>
> ​I have checked this with inbound but there is a fix we need to do. I
> have done it in [1]. I have done several tests with and without streaming
> from 5mb to 1gb files.
>
> Without streaming [2] and [3] will show the memory growth. With
> streaming [4]. Without streaming most of the time ESB went OOM​.
>
> Find the related configs and axis2 changes in [5] and [6].
>
> [1] https://wso2.org/jira/browse/ESBJAVA-4458
>
> [2]
>
> [image: Inline image 1]
>
> [3]
>
> [image: Inline image 2]
>
> [4]
>
> [image: Inline image 3]
>
> [5]
>
> http://ws.apache.org/ns/synapse";  name="load"
>  sequence="request"  onError="fault" protocol="file" suspend="false">
>
>   1
>   true
>    name="transport.vfs.ContentType">application/file
>    name="transport.vfs.LockReleaseSameNode">false
>   false
>    name="transport.vfs.ActionAfterFailure">DELETE
>   true
>   true
>    name="transport.vfs.ActionAfterProcess">DELETE
>    name="transport.vfs.FileURI">file:///home/wso2/work/tmp/file/in
>   false
>   true
>   enable
>    name="transport.vfs.FileSortAscending">true
>    name="transport.vfs.FileSortAttribute">NONE
>   false
>
> 
>
> 
> 
> 
> 
>  value="true"/>
> 
> 
> 
> 
>
> [6]
>  class="org.apache.axis2.format.BinaryBuilder"/>
>
> 
> class="org.wso2.carbon.relay.ExpandingMessageFormatter"/>
>>>

Re: [Dev] Streaming File transfer with File Inbound/File Connector

[Malaka Silva]

Hi Kasun,

I don't think we need to cover any functionality that is already available
with ESB.
Any use case that we need the vfs sender from connector?

On Mon, Mar 14, 2016 at 10:40 AM, Kasun Indrasiri  wrote:

>
>
> On Mon, Mar 14, 2016 at 9:57 AM, Malaka Silva  wrote:
>
>> Hi Kasun,
>>
>> I don't think we have the same functionality of VFS sender in file
>> connector. We have only focused on use cases not covered with VFS transport.
>>
>
> Ok. I think we need to make sure things are consistent across the
> transport and the file connector. Can we please include this as an
> improvement for the next version of the connector.
>
>
>>
> @Vivekananthan - Please confirm this?
>>
>> On Sat, Mar 12, 2016 at 4:39 PM, Kasun Indrasiri  wrote:
>>
>>> This is great. Thanks a lot Malaka.
>>> Also, if we use the file connector as the outbound channel, will it work
>>> in the same way?
>>>
>>> On Tue, Mar 1, 2016 at 12:36 PM, Malaka Silva  wrote:
>>>
 Hi All,

 Please ignore my previous comments. This can be done with current
 implementation.

 if (builder instanceof DataSourceMessageBuilder &&
 "true".equals(streaming)) {
 dataSource = ManagedDataSourceFactory.create(new
 FileObjectDataSource(file, contentType));
 in = null;
 } else {
 in = new
 AutoCloseInputStream(file.getContent().getInputStream());
 dataSource = null;
 }
 ..
 OMElement documentElement;
 if (in != null) {
 documentElement = builder.processDocument(in,
 contentType, axis2MsgCtx);
 } else {
 documentElement =
 ((DataSourceMessageBuilder)builder).processDocument(dataSource,
 contentType, axis2MsgCtx);
 }
 ...
 if(dataSource != null) {
 dataSource.destroy();
 }

 This is
 ​because ​
 message builder able to build messages from
 ​​
 DataSource objects.
 ​
 ​
 DataSource​
  by definition the data from a DataSource can be read
 ​
 multiple times,
 ​builders that implement ​
 this interface
 ​can​
  avoid storing the message content in memory.
 ​ ​
 If a message builder implements this interface and the
 ​file/vfs
  is able to provide the message payload as a data source, then the
 method defined by this interface should be preferred over the method
 defined by Builder.
 ​ This
  helps optimizing
 ​PT​
  with
 ​vfs/file​.
 The builder will typically expose the data source directly or
 indirectly through the returned OMElement, e.g. by adding to the tree an
 OMText or OMDataSource node referencing the data source.

 ​I have checked this with inbound but there is a fix we need to do. I
 have done it in [1]. I have done several tests with and without streaming
 from 5mb to 1gb files.

 Without streaming [2] and [3] will show the memory growth. With
 streaming [4]. Without streaming most of the time ESB went OOM​.

 Find the related configs and axis2 changes in [5] and [6].

 [1] https://wso2.org/jira/browse/ESBJAVA-4458

 [2]

 [image: Inline image 1]

 [3]

 [image: Inline image 2]

 [4]

 [image: Inline image 3]

 [5]

 http://ws.apache.org/ns/synapse";  name="load"
  sequence="request"  onError="fault" protocol="file" suspend="false">

   1
   true
   >>> name="transport.vfs.ContentType">application/file
   >>> name="transport.vfs.LockReleaseSameNode">false
   false
   >>> name="transport.vfs.ActionAfterFailure">DELETE
   true
   true
   >>> name="transport.vfs.ActionAfterProcess">DELETE
   >>> name="transport.vfs.FileURI">file:///home/wso2/work/tmp/file/in
   false
   true
   enable
   true
   NONE
   false

 

 
 
 
 
 >>> value="true"/>
 
 
 
 

 [6]
 >>> class="org.apache.axis2.format.BinaryBuilder"/>

 >>>
 class="org.wso2.carbon.relay.ExpandingMessageFormatter"/>

 On Mon, Feb 29, 2016 at 12:02 PM, Malaka Silva  wrote:

> Hi Kasun,
>
> Currently no OOB solution with ESB 4.9.0. File always gets build
> before mediation.
>
> However use case mentioned can be handled with Schedule task -> File
> connector search -> File connector copy
>
> +1 for OOB solution.
>
> We can  do this for ESB 5.0. Also there is an option to do this as a
> custom inbound. So that this can be used by previous ESB versions as well.

Re: [Dev] Streaming File transfer with File Inbound/File Connector

[Yashothara Shanmugarajah]

Yes Malaka

Best Regards,
Yashothara.S

Software Engineer
WSO2


On Mon, Apr 4, 2016 at 9:14 AM, Malaka Silva  wrote:

> Hi Yashothara,
>
> Can you start working on this please.
>
> On Mon, Mar 14, 2016 at 11:41 AM, Vivekananthan Sivanayagam <
> vivekanant...@wso2.com> wrote:
>
>> Hi Malaka,
>>
>> I have created Jira [1].
>>
>> [1] https://wso2.org/jira/browse/ESBCONNECT-86
>>
>>
>>
>>
>> *Thank youVivekananthan Sivanayagam*
>>
>> *Associate Software Engineer | WSO2*
>>
>> *E:vivekanant...@wso2.com *
>> *M:+94752786138 <%2B94752786138>*
>>
>> On Mon, Mar 14, 2016 at 10:59 AM, Malaka Silva  wrote:
>>
>>> Hi Vivekananthan,
>>>
>>> Can we create a Jira and follow up on this please?
>>>
>>> On Mon, Mar 14, 2016 at 10:40 AM, Kasun Indrasiri 
>>> wrote:
>>>


 On Mon, Mar 14, 2016 at 9:57 AM, Malaka Silva  wrote:

> Hi Kasun,
>
> I don't think we have the same functionality of VFS sender in file
> connector. We have only focused on use cases not covered with VFS 
> transport.
>

 Ok. I think we need to make sure things are consistent across the
 transport and the file connector. Can we please include this as an
 improvement for the next version of the connector.


>
 @Vivekananthan - Please confirm this?
>
> On Sat, Mar 12, 2016 at 4:39 PM, Kasun Indrasiri 
> wrote:
>
>> This is great. Thanks a lot Malaka.
>> Also, if we use the file connector as the outbound channel, will it
>> work in the same way?
>>
>> On Tue, Mar 1, 2016 at 12:36 PM, Malaka Silva 
>> wrote:
>>
>>> Hi All,
>>>
>>> Please ignore my previous comments. This can be done with current
>>> implementation.
>>>
>>> if (builder instanceof DataSourceMessageBuilder &&
>>> "true".equals(streaming)) {
>>> dataSource = ManagedDataSourceFactory.create(new
>>> FileObjectDataSource(file, contentType));
>>> in = null;
>>> } else {
>>> in = new
>>> AutoCloseInputStream(file.getContent().getInputStream());
>>> dataSource = null;
>>> }
>>> ..
>>> OMElement documentElement;
>>> if (in != null) {
>>> documentElement = builder.processDocument(in,
>>> contentType, axis2MsgCtx);
>>> } else {
>>> documentElement =
>>> ((DataSourceMessageBuilder)builder).processDocument(dataSource,
>>> contentType, axis2MsgCtx);
>>> }
>>> ...
>>> if(dataSource != null) {
>>> dataSource.destroy();
>>> }
>>>
>>> This is
>>> ​because ​
>>> message builder able to build messages from
>>> ​​
>>> DataSource objects.
>>> ​
>>> ​
>>> DataSource​
>>>  by definition the data from a DataSource can be read
>>> ​
>>> multiple times,
>>> ​builders that implement ​
>>> this interface
>>> ​can​
>>>  avoid storing the message content in memory.
>>> ​ ​
>>> If a message builder implements this interface and the
>>> ​file/vfs
>>>  is able to provide the message payload as a data source, then the
>>> method defined by this interface should be preferred over the method
>>> defined by Builder.
>>> ​ This
>>>  helps optimizing
>>> ​PT​
>>>  with
>>> ​vfs/file​.
>>> The builder will typically expose the data source directly or
>>> indirectly through the returned OMElement, e.g. by adding to the tree an
>>> OMText or OMDataSource node referencing the data source.
>>>
>>> ​I have checked this with inbound but there is a fix we need to do.
>>> I have done it in [1]. I have done several tests with and without 
>>> streaming
>>> from 5mb to 1gb files.
>>>
>>> Without streaming [2] and [3] will show the memory growth. With
>>> streaming [4]. Without streaming most of the time ESB went OOM​.
>>>
>>> Find the related configs and axis2 changes in [5] and [6].
>>>
>>> [1] https://wso2.org/jira/browse/ESBJAVA-4458
>>>
>>> [2]
>>>
>>> [image: Inline image 1]
>>>
>>> [3]
>>>
>>> [image: Inline image 2]
>>>
>>> [4]
>>>
>>> [image: Inline image 3]
>>>
>>> [5]
>>>
>>> http://ws.apache.org/ns/synapse";
>>>  name="load"  sequence="request"  onError="fault" protocol="file"
>>> suspend="false">
>>>
>>>   1
>>>   true
>>>   >> name="transport.vfs.ContentType">application/file
>>>   >> name="transport.vfs.LockReleaseSameNode">false
>>>   >> name="transport.vfs.AutoLockRelease">false
>>>   >> name="transport.vfs.ActionAfterFailure">DELETE
>>>   true
>>>   true
>>>   >> name="transport.vfs.ActionAfterProcess">DELETE
>>>   >> 

Re: [Dev] Streaming File transfer with File Inbound/File Connector

[Malaka Silva]

Hi Yashothara,

Can you start working on this please.

On Mon, Mar 14, 2016 at 11:41 AM, Vivekananthan Sivanayagam <
vivekanant...@wso2.com> wrote:

> Hi Malaka,
>
> I have created Jira [1].
>
> [1] https://wso2.org/jira/browse/ESBCONNECT-86
>
>
>
>
> *Thank youVivekananthan Sivanayagam*
>
> *Associate Software Engineer | WSO2*
>
> *E:vivekanant...@wso2.com *
> *M:+94752786138 <%2B94752786138>*
>
> On Mon, Mar 14, 2016 at 10:59 AM, Malaka Silva  wrote:
>
>> Hi Vivekananthan,
>>
>> Can we create a Jira and follow up on this please?
>>
>> On Mon, Mar 14, 2016 at 10:40 AM, Kasun Indrasiri  wrote:
>>
>>>
>>>
>>> On Mon, Mar 14, 2016 at 9:57 AM, Malaka Silva  wrote:
>>>
 Hi Kasun,

 I don't think we have the same functionality of VFS sender in file
 connector. We have only focused on use cases not covered with VFS 
 transport.

>>>
>>> Ok. I think we need to make sure things are consistent across the
>>> transport and the file connector. Can we please include this as an
>>> improvement for the next version of the connector.
>>>
>>>

>>> @Vivekananthan - Please confirm this?

 On Sat, Mar 12, 2016 at 4:39 PM, Kasun Indrasiri 
 wrote:

> This is great. Thanks a lot Malaka.
> Also, if we use the file connector as the outbound channel, will it
> work in the same way?
>
> On Tue, Mar 1, 2016 at 12:36 PM, Malaka Silva  wrote:
>
>> Hi All,
>>
>> Please ignore my previous comments. This can be done with current
>> implementation.
>>
>> if (builder instanceof DataSourceMessageBuilder &&
>> "true".equals(streaming)) {
>> dataSource = ManagedDataSourceFactory.create(new
>> FileObjectDataSource(file, contentType));
>> in = null;
>> } else {
>> in = new
>> AutoCloseInputStream(file.getContent().getInputStream());
>> dataSource = null;
>> }
>> ..
>> OMElement documentElement;
>> if (in != null) {
>> documentElement = builder.processDocument(in,
>> contentType, axis2MsgCtx);
>> } else {
>> documentElement =
>> ((DataSourceMessageBuilder)builder).processDocument(dataSource,
>> contentType, axis2MsgCtx);
>> }
>> ...
>> if(dataSource != null) {
>> dataSource.destroy();
>> }
>>
>> This is
>> ​because ​
>> message builder able to build messages from
>> ​​
>> DataSource objects.
>> ​
>> ​
>> DataSource​
>>  by definition the data from a DataSource can be read
>> ​
>> multiple times,
>> ​builders that implement ​
>> this interface
>> ​can​
>>  avoid storing the message content in memory.
>> ​ ​
>> If a message builder implements this interface and the
>> ​file/vfs
>>  is able to provide the message payload as a data source, then the
>> method defined by this interface should be preferred over the method
>> defined by Builder.
>> ​ This
>>  helps optimizing
>> ​PT​
>>  with
>> ​vfs/file​.
>> The builder will typically expose the data source directly or
>> indirectly through the returned OMElement, e.g. by adding to the tree an
>> OMText or OMDataSource node referencing the data source.
>>
>> ​I have checked this with inbound but there is a fix we need to do. I
>> have done it in [1]. I have done several tests with and without streaming
>> from 5mb to 1gb files.
>>
>> Without streaming [2] and [3] will show the memory growth. With
>> streaming [4]. Without streaming most of the time ESB went OOM​.
>>
>> Find the related configs and axis2 changes in [5] and [6].
>>
>> [1] https://wso2.org/jira/browse/ESBJAVA-4458
>>
>> [2]
>>
>> [image: Inline image 1]
>>
>> [3]
>>
>> [image: Inline image 2]
>>
>> [4]
>>
>> [image: Inline image 3]
>>
>> [5]
>>
>> http://ws.apache.org/ns/synapse";
>>  name="load"  sequence="request"  onError="fault" protocol="file"
>> suspend="false">
>>
>>   1
>>   true
>>   > name="transport.vfs.ContentType">application/file
>>   > name="transport.vfs.LockReleaseSameNode">false
>>   > name="transport.vfs.AutoLockRelease">false
>>   > name="transport.vfs.ActionAfterFailure">DELETE
>>   true
>>   true
>>   > name="transport.vfs.ActionAfterProcess">DELETE
>>   > name="transport.vfs.FileURI">file:///home/wso2/work/tmp/file/in
>>   > name="transport.vfs.DistributedLock">false
>>   true
>>   enable
>>   > name="transport.vfs.FileSortAscending">true
>>   > name="transport.vfs.FileSortAttribute">NONE
>>   fals

Re: [Dev] Streaming File transfer with File Inbound/File Connector

[Vivekananthan Sivanayagam]

Hi Malaka,

I have created Jira [1].

[1] https://wso2.org/jira/browse/ESBCONNECT-86




*Thank youVivekananthan Sivanayagam*

*Associate Software Engineer | WSO2*

*E:vivekanant...@wso2.com *
*M:+94752786138*

On Mon, Mar 14, 2016 at 10:59 AM, Malaka Silva  wrote:

> Hi Vivekananthan,
>
> Can we create a Jira and follow up on this please?
>
> On Mon, Mar 14, 2016 at 10:40 AM, Kasun Indrasiri  wrote:
>
>>
>>
>> On Mon, Mar 14, 2016 at 9:57 AM, Malaka Silva  wrote:
>>
>>> Hi Kasun,
>>>
>>> I don't think we have the same functionality of VFS sender in file
>>> connector. We have only focused on use cases not covered with VFS transport.
>>>
>>
>> Ok. I think we need to make sure things are consistent across the
>> transport and the file connector. Can we please include this as an
>> improvement for the next version of the connector.
>>
>>
>>>
>> @Vivekananthan - Please confirm this?
>>>
>>> On Sat, Mar 12, 2016 at 4:39 PM, Kasun Indrasiri  wrote:
>>>
 This is great. Thanks a lot Malaka.
 Also, if we use the file connector as the outbound channel, will it
 work in the same way?

 On Tue, Mar 1, 2016 at 12:36 PM, Malaka Silva  wrote:

> Hi All,
>
> Please ignore my previous comments. This can be done with current
> implementation.
>
> if (builder instanceof DataSourceMessageBuilder &&
> "true".equals(streaming)) {
> dataSource = ManagedDataSourceFactory.create(new
> FileObjectDataSource(file, contentType));
> in = null;
> } else {
> in = new
> AutoCloseInputStream(file.getContent().getInputStream());
> dataSource = null;
> }
> ..
> OMElement documentElement;
> if (in != null) {
> documentElement = builder.processDocument(in,
> contentType, axis2MsgCtx);
> } else {
> documentElement =
> ((DataSourceMessageBuilder)builder).processDocument(dataSource,
> contentType, axis2MsgCtx);
> }
> ...
> if(dataSource != null) {
> dataSource.destroy();
> }
>
> This is
> ​because ​
> message builder able to build messages from
> ​​
> DataSource objects.
> ​
> ​
> DataSource​
>  by definition the data from a DataSource can be read
> ​
> multiple times,
> ​builders that implement ​
> this interface
> ​can​
>  avoid storing the message content in memory.
> ​ ​
> If a message builder implements this interface and the
> ​file/vfs
>  is able to provide the message payload as a data source, then the
> method defined by this interface should be preferred over the method
> defined by Builder.
> ​ This
>  helps optimizing
> ​PT​
>  with
> ​vfs/file​.
> The builder will typically expose the data source directly or
> indirectly through the returned OMElement, e.g. by adding to the tree an
> OMText or OMDataSource node referencing the data source.
>
> ​I have checked this with inbound but there is a fix we need to do. I
> have done it in [1]. I have done several tests with and without streaming
> from 5mb to 1gb files.
>
> Without streaming [2] and [3] will show the memory growth. With
> streaming [4]. Without streaming most of the time ESB went OOM​.
>
> Find the related configs and axis2 changes in [5] and [6].
>
> [1] https://wso2.org/jira/browse/ESBJAVA-4458
>
> [2]
>
> [image: Inline image 1]
>
> [3]
>
> [image: Inline image 2]
>
> [4]
>
> [image: Inline image 3]
>
> [5]
>
> http://ws.apache.org/ns/synapse";  name="load"
>  sequence="request"  onError="fault" protocol="file" suspend="false">
>
>   1
>   true
>    name="transport.vfs.ContentType">application/file
>    name="transport.vfs.LockReleaseSameNode">false
>   false
>    name="transport.vfs.ActionAfterFailure">DELETE
>   true
>   true
>    name="transport.vfs.ActionAfterProcess">DELETE
>    name="transport.vfs.FileURI">file:///home/wso2/work/tmp/file/in
>   false
>   true
>   enable
>    name="transport.vfs.FileSortAscending">true
>    name="transport.vfs.FileSortAttribute">NONE
>   false
>
> 
>
> 
> 
> 
> 
>  value="true"/>
> 
> 
> 
> 
>
> [6]
>  class="org.apache.axis2.format.BinaryBuilder"/>
>
> 
> class="org.wso2.carbon.relay.ExpandingMessageFormatter"/>
>
> On Mon, Feb 29, 2016 at 12:02 PM, Malaka Silva 
> wrote:
>>>

Re: [Dev] Streaming File transfer with File Inbound/File Connector

[Malaka Silva]

Hi Vivekananthan,

Can we create a Jira and follow up on this please?

On Mon, Mar 14, 2016 at 10:40 AM, Kasun Indrasiri  wrote:

>
>
> On Mon, Mar 14, 2016 at 9:57 AM, Malaka Silva  wrote:
>
>> Hi Kasun,
>>
>> I don't think we have the same functionality of VFS sender in file
>> connector. We have only focused on use cases not covered with VFS transport.
>>
>
> Ok. I think we need to make sure things are consistent across the
> transport and the file connector. Can we please include this as an
> improvement for the next version of the connector.
>
>
>>
> @Vivekananthan - Please confirm this?
>>
>> On Sat, Mar 12, 2016 at 4:39 PM, Kasun Indrasiri  wrote:
>>
>>> This is great. Thanks a lot Malaka.
>>> Also, if we use the file connector as the outbound channel, will it work
>>> in the same way?
>>>
>>> On Tue, Mar 1, 2016 at 12:36 PM, Malaka Silva  wrote:
>>>
 Hi All,

 Please ignore my previous comments. This can be done with current
 implementation.

 if (builder instanceof DataSourceMessageBuilder &&
 "true".equals(streaming)) {
 dataSource = ManagedDataSourceFactory.create(new
 FileObjectDataSource(file, contentType));
 in = null;
 } else {
 in = new
 AutoCloseInputStream(file.getContent().getInputStream());
 dataSource = null;
 }
 ..
 OMElement documentElement;
 if (in != null) {
 documentElement = builder.processDocument(in,
 contentType, axis2MsgCtx);
 } else {
 documentElement =
 ((DataSourceMessageBuilder)builder).processDocument(dataSource,
 contentType, axis2MsgCtx);
 }
 ...
 if(dataSource != null) {
 dataSource.destroy();
 }

 This is
 ​because ​
 message builder able to build messages from
 ​​
 DataSource objects.
 ​
 ​
 DataSource​
  by definition the data from a DataSource can be read
 ​
 multiple times,
 ​builders that implement ​
 this interface
 ​can​
  avoid storing the message content in memory.
 ​ ​
 If a message builder implements this interface and the
 ​file/vfs
  is able to provide the message payload as a data source, then the
 method defined by this interface should be preferred over the method
 defined by Builder.
 ​ This
  helps optimizing
 ​PT​
  with
 ​vfs/file​.
 The builder will typically expose the data source directly or
 indirectly through the returned OMElement, e.g. by adding to the tree an
 OMText or OMDataSource node referencing the data source.

 ​I have checked this with inbound but there is a fix we need to do. I
 have done it in [1]. I have done several tests with and without streaming
 from 5mb to 1gb files.

 Without streaming [2] and [3] will show the memory growth. With
 streaming [4]. Without streaming most of the time ESB went OOM​.

 Find the related configs and axis2 changes in [5] and [6].

 [1] https://wso2.org/jira/browse/ESBJAVA-4458

 [2]

 [image: Inline image 1]

 [3]

 [image: Inline image 2]

 [4]

 [image: Inline image 3]

 [5]

 http://ws.apache.org/ns/synapse";  name="load"
  sequence="request"  onError="fault" protocol="file" suspend="false">

   1
   true
   >>> name="transport.vfs.ContentType">application/file
   >>> name="transport.vfs.LockReleaseSameNode">false
   false
   >>> name="transport.vfs.ActionAfterFailure">DELETE
   true
   true
   >>> name="transport.vfs.ActionAfterProcess">DELETE
   >>> name="transport.vfs.FileURI">file:///home/wso2/work/tmp/file/in
   false
   true
   enable
   true
   NONE
   false

 

 
 
 
 
 >>> value="true"/>
 
 
 
 

 [6]
 >>> class="org.apache.axis2.format.BinaryBuilder"/>

 >>>
 class="org.wso2.carbon.relay.ExpandingMessageFormatter"/>

 On Mon, Feb 29, 2016 at 12:02 PM, Malaka Silva  wrote:

> Hi Kasun,
>
> Currently no OOB solution with ESB 4.9.0. File always gets build
> before mediation.
>
> However use case mentioned can be handled with Schedule task -> File
> connector search -> File connector copy
>
> +1 for OOB solution.
>
> We can  do this for ESB 5.0. Also there is an option to do this as a
> custom inbound. So that this can be used by previous ESB versions as well.
>
> So default inbound can be used, if someone needs to do a mediation on
>

Re: [Dev] Streaming File transfer with File Inbound/File Connector

[Kasun Indrasiri]

On Mon, Mar 14, 2016 at 9:57 AM, Malaka Silva  wrote:

> Hi Kasun,
>
> I don't think we have the same functionality of VFS sender in file
> connector. We have only focused on use cases not covered with VFS transport.
>

Ok. I think we need to make sure things are consistent across the transport
and the file connector. Can we please include this as an improvement for
the next version of the connector.


>
@Vivekananthan - Please confirm this?
>
> On Sat, Mar 12, 2016 at 4:39 PM, Kasun Indrasiri  wrote:
>
>> This is great. Thanks a lot Malaka.
>> Also, if we use the file connector as the outbound channel, will it work
>> in the same way?
>>
>> On Tue, Mar 1, 2016 at 12:36 PM, Malaka Silva  wrote:
>>
>>> Hi All,
>>>
>>> Please ignore my previous comments. This can be done with current
>>> implementation.
>>>
>>> if (builder instanceof DataSourceMessageBuilder &&
>>> "true".equals(streaming)) {
>>> dataSource = ManagedDataSourceFactory.create(new
>>> FileObjectDataSource(file, contentType));
>>> in = null;
>>> } else {
>>> in = new
>>> AutoCloseInputStream(file.getContent().getInputStream());
>>> dataSource = null;
>>> }
>>> ..
>>> OMElement documentElement;
>>> if (in != null) {
>>> documentElement = builder.processDocument(in,
>>> contentType, axis2MsgCtx);
>>> } else {
>>> documentElement =
>>> ((DataSourceMessageBuilder)builder).processDocument(dataSource,
>>> contentType, axis2MsgCtx);
>>> }
>>> ...
>>> if(dataSource != null) {
>>> dataSource.destroy();
>>> }
>>>
>>> This is
>>> ​because ​
>>> message builder able to build messages from
>>> ​​
>>> DataSource objects.
>>> ​
>>> ​
>>> DataSource​
>>>  by definition the data from a DataSource can be read
>>> ​
>>> multiple times,
>>> ​builders that implement ​
>>> this interface
>>> ​can​
>>>  avoid storing the message content in memory.
>>> ​ ​
>>> If a message builder implements this interface and the
>>> ​file/vfs
>>>  is able to provide the message payload as a data source, then the
>>> method defined by this interface should be preferred over the method
>>> defined by Builder.
>>> ​ This
>>>  helps optimizing
>>> ​PT​
>>>  with
>>> ​vfs/file​.
>>> The builder will typically expose the data source directly or indirectly
>>> through the returned OMElement, e.g. by adding to the tree an OMText or
>>> OMDataSource node referencing the data source.
>>>
>>> ​I have checked this with inbound but there is a fix we need to do. I
>>> have done it in [1]. I have done several tests with and without streaming
>>> from 5mb to 1gb files.
>>>
>>> Without streaming [2] and [3] will show the memory growth. With
>>> streaming [4]. Without streaming most of the time ESB went OOM​.
>>>
>>> Find the related configs and axis2 changes in [5] and [6].
>>>
>>> [1] https://wso2.org/jira/browse/ESBJAVA-4458
>>>
>>> [2]
>>>
>>> [image: Inline image 1]
>>>
>>> [3]
>>>
>>> [image: Inline image 2]
>>>
>>> [4]
>>>
>>> [image: Inline image 3]
>>>
>>> [5]
>>>
>>> http://ws.apache.org/ns/synapse";  name="load"
>>>  sequence="request"  onError="fault" protocol="file" suspend="false">
>>>
>>>   1
>>>   true
>>>   >> name="transport.vfs.ContentType">application/file
>>>   >> name="transport.vfs.LockReleaseSameNode">false
>>>   false
>>>   >> name="transport.vfs.ActionAfterFailure">DELETE
>>>   true
>>>   true
>>>   >> name="transport.vfs.ActionAfterProcess">DELETE
>>>   >> name="transport.vfs.FileURI">file:///home/wso2/work/tmp/file/in
>>>   false
>>>   true
>>>   enable
>>>   true
>>>   NONE
>>>   false
>>>
>>> 
>>>
>>> 
>>> 
>>> 
>>> 
>>> >> value="true"/>
>>> 
>>> 
>>> 
>>> 
>>>
>>> [6]
>>> >> class="org.apache.axis2.format.BinaryBuilder"/>
>>>
>>> >>
>>> class="org.wso2.carbon.relay.ExpandingMessageFormatter"/>
>>>
>>> On Mon, Feb 29, 2016 at 12:02 PM, Malaka Silva  wrote:
>>>
 Hi Kasun,

 Currently no OOB solution with ESB 4.9.0. File always gets build before
 mediation.

 However use case mentioned can be handled with Schedule task -> File
 connector search -> File connector copy

 +1 for OOB solution.

 We can  do this for ESB 5.0. Also there is an option to do this as a
 custom inbound. So that this can be used by previous ESB versions as well.

 So default inbound can be used, if someone needs to do a mediation on
 file content and custom inbound for PT file use case. WDYT?


 On Mon, Feb 29, 2016 at 11:49 AM, Kasun Indrasiri 
 wrote:

> Hi Malaka,
>
> Do we support the $subject? Basically we use an Inbound as the source
> and use file connector as the destination. If this is not supp

Re: [Dev] Streaming File transfer with File Inbound/File Connector

[Vivekananthan Sivanayagam]

Hi,

We don't have the same functionality of VFS sender in File Connector.




*Thank youVivekananthan Sivanayagam*

*Associate Software Engineer | WSO2*

*E:vivekanant...@wso2.com *
*M:+94752786138*

On Mon, Mar 14, 2016 at 9:57 AM, Malaka Silva  wrote:

> Hi Kasun,
>
> I don't think we have the same functionality of VFS sender in file
> connector. We have only focused on use cases not covered with VFS transport.
>
> @Vivekananthan - Please confirm this?
>
> On Sat, Mar 12, 2016 at 4:39 PM, Kasun Indrasiri  wrote:
>
>> This is great. Thanks a lot Malaka.
>> Also, if we use the file connector as the outbound channel, will it work
>> in the same way?
>>
>> On Tue, Mar 1, 2016 at 12:36 PM, Malaka Silva  wrote:
>>
>>> Hi All,
>>>
>>> Please ignore my previous comments. This can be done with current
>>> implementation.
>>>
>>> if (builder instanceof DataSourceMessageBuilder &&
>>> "true".equals(streaming)) {
>>> dataSource = ManagedDataSourceFactory.create(new
>>> FileObjectDataSource(file, contentType));
>>> in = null;
>>> } else {
>>> in = new
>>> AutoCloseInputStream(file.getContent().getInputStream());
>>> dataSource = null;
>>> }
>>> ..
>>> OMElement documentElement;
>>> if (in != null) {
>>> documentElement = builder.processDocument(in,
>>> contentType, axis2MsgCtx);
>>> } else {
>>> documentElement =
>>> ((DataSourceMessageBuilder)builder).processDocument(dataSource,
>>> contentType, axis2MsgCtx);
>>> }
>>> ...
>>> if(dataSource != null) {
>>> dataSource.destroy();
>>> }
>>>
>>> This is
>>> ​because ​
>>> message builder able to build messages from
>>> ​​
>>> DataSource objects.
>>> ​
>>> ​
>>> DataSource​
>>>  by definition the data from a DataSource can be read
>>> ​
>>> multiple times,
>>> ​builders that implement ​
>>> this interface
>>> ​can​
>>>  avoid storing the message content in memory.
>>> ​ ​
>>> If a message builder implements this interface and the
>>> ​file/vfs
>>>  is able to provide the message payload as a data source, then the
>>> method defined by this interface should be preferred over the method
>>> defined by Builder.
>>> ​ This
>>>  helps optimizing
>>> ​PT​
>>>  with
>>> ​vfs/file​.
>>> The builder will typically expose the data source directly or indirectly
>>> through the returned OMElement, e.g. by adding to the tree an OMText or
>>> OMDataSource node referencing the data source.
>>>
>>> ​I have checked this with inbound but there is a fix we need to do. I
>>> have done it in [1]. I have done several tests with and without streaming
>>> from 5mb to 1gb files.
>>>
>>> Without streaming [2] and [3] will show the memory growth. With
>>> streaming [4]. Without streaming most of the time ESB went OOM​.
>>>
>>> Find the related configs and axis2 changes in [5] and [6].
>>>
>>> [1] https://wso2.org/jira/browse/ESBJAVA-4458
>>>
>>> [2]
>>>
>>> [image: Inline image 1]
>>>
>>> [3]
>>>
>>> [image: Inline image 2]
>>>
>>> [4]
>>>
>>> [image: Inline image 3]
>>>
>>> [5]
>>>
>>> http://ws.apache.org/ns/synapse";  name="load"
>>>  sequence="request"  onError="fault" protocol="file" suspend="false">
>>>
>>>   1
>>>   true
>>>   >> name="transport.vfs.ContentType">application/file
>>>   >> name="transport.vfs.LockReleaseSameNode">false
>>>   false
>>>   >> name="transport.vfs.ActionAfterFailure">DELETE
>>>   true
>>>   true
>>>   >> name="transport.vfs.ActionAfterProcess">DELETE
>>>   >> name="transport.vfs.FileURI">file:///home/wso2/work/tmp/file/in
>>>   false
>>>   true
>>>   enable
>>>   true
>>>   NONE
>>>   false
>>>
>>> 
>>>
>>> 
>>> 
>>> 
>>> 
>>> >> value="true"/>
>>> 
>>> 
>>> 
>>> 
>>>
>>> [6]
>>> >> class="org.apache.axis2.format.BinaryBuilder"/>
>>>
>>> >>
>>> class="org.wso2.carbon.relay.ExpandingMessageFormatter"/>
>>>
>>> On Mon, Feb 29, 2016 at 12:02 PM, Malaka Silva  wrote:
>>>
 Hi Kasun,

 Currently no OOB solution with ESB 4.9.0. File always gets build before
 mediation.

 However use case mentioned can be handled with Schedule task -> File
 connector search -> File connector copy

 +1 for OOB solution.

 We can  do this for ESB 5.0. Also there is an option to do this as a
 custom inbound. So that this can be used by previous ESB versions as well.

 So default inbound can be used, if someone needs to do a mediation on
 file content and custom inbound for PT file use case. WDYT?


 On Mon, Feb 29, 2016 at 11:49 AM, Kasun Indrasiri 
 wrote:

> Hi Malaka,
>
> Do we support the $subject? Basically we use an Inbound as the source
> and use file connector as the destination. If th

Re: [Dev] Streaming File transfer with File Inbound/File Connector

[Malaka Silva]

Hi Kasun,

I don't think we have the same functionality of VFS sender in file
connector. We have only focused on use cases not covered with VFS transport.

@Vivekananthan - Please confirm this?

On Sat, Mar 12, 2016 at 4:39 PM, Kasun Indrasiri  wrote:

> This is great. Thanks a lot Malaka.
> Also, if we use the file connector as the outbound channel, will it work
> in the same way?
>
> On Tue, Mar 1, 2016 at 12:36 PM, Malaka Silva  wrote:
>
>> Hi All,
>>
>> Please ignore my previous comments. This can be done with current
>> implementation.
>>
>> if (builder instanceof DataSourceMessageBuilder &&
>> "true".equals(streaming)) {
>> dataSource = ManagedDataSourceFactory.create(new
>> FileObjectDataSource(file, contentType));
>> in = null;
>> } else {
>> in = new
>> AutoCloseInputStream(file.getContent().getInputStream());
>> dataSource = null;
>> }
>> ..
>> OMElement documentElement;
>> if (in != null) {
>> documentElement = builder.processDocument(in,
>> contentType, axis2MsgCtx);
>> } else {
>> documentElement =
>> ((DataSourceMessageBuilder)builder).processDocument(dataSource,
>> contentType, axis2MsgCtx);
>> }
>> ...
>> if(dataSource != null) {
>> dataSource.destroy();
>> }
>>
>> This is
>> ​because ​
>> message builder able to build messages from
>> ​​
>> DataSource objects.
>> ​
>> ​
>> DataSource​
>>  by definition the data from a DataSource can be read
>> ​
>> multiple times,
>> ​builders that implement ​
>> this interface
>> ​can​
>>  avoid storing the message content in memory.
>> ​ ​
>> If a message builder implements this interface and the
>> ​file/vfs
>>  is able to provide the message payload as a data source, then the method
>> defined by this interface should be preferred over the method defined by
>> Builder.
>> ​ This
>>  helps optimizing
>> ​PT​
>>  with
>> ​vfs/file​.
>> The builder will typically expose the data source directly or indirectly
>> through the returned OMElement, e.g. by adding to the tree an OMText or
>> OMDataSource node referencing the data source.
>>
>> ​I have checked this with inbound but there is a fix we need to do. I
>> have done it in [1]. I have done several tests with and without streaming
>> from 5mb to 1gb files.
>>
>> Without streaming [2] and [3] will show the memory growth. With streaming
>> [4]. Without streaming most of the time ESB went OOM​.
>>
>> Find the related configs and axis2 changes in [5] and [6].
>>
>> [1] https://wso2.org/jira/browse/ESBJAVA-4458
>>
>> [2]
>>
>> [image: Inline image 1]
>>
>> [3]
>>
>> [image: Inline image 2]
>>
>> [4]
>>
>> [image: Inline image 3]
>>
>> [5]
>>
>> http://ws.apache.org/ns/synapse";  name="load"
>>  sequence="request"  onError="fault" protocol="file" suspend="false">
>>
>>   1
>>   true
>>   > name="transport.vfs.ContentType">application/file
>>   > name="transport.vfs.LockReleaseSameNode">false
>>   false
>>   > name="transport.vfs.ActionAfterFailure">DELETE
>>   true
>>   true
>>   > name="transport.vfs.ActionAfterProcess">DELETE
>>   > name="transport.vfs.FileURI">file:///home/wso2/work/tmp/file/in
>>   false
>>   true
>>   enable
>>   true
>>   NONE
>>   false
>>
>> 
>>
>> 
>> 
>> 
>> 
>> > value="true"/>
>> 
>> 
>> 
>> 
>>
>> [6]
>> > class="org.apache.axis2.format.BinaryBuilder"/>
>>
>> >
>> class="org.wso2.carbon.relay.ExpandingMessageFormatter"/>
>>
>> On Mon, Feb 29, 2016 at 12:02 PM, Malaka Silva  wrote:
>>
>>> Hi Kasun,
>>>
>>> Currently no OOB solution with ESB 4.9.0. File always gets build before
>>> mediation.
>>>
>>> However use case mentioned can be handled with Schedule task -> File
>>> connector search -> File connector copy
>>>
>>> +1 for OOB solution.
>>>
>>> We can  do this for ESB 5.0. Also there is an option to do this as a
>>> custom inbound. So that this can be used by previous ESB versions as well.
>>>
>>> So default inbound can be used, if someone needs to do a mediation on
>>> file content and custom inbound for PT file use case. WDYT?
>>>
>>>
>>> On Mon, Feb 29, 2016 at 11:49 AM, Kasun Indrasiri 
>>> wrote:
>>>
 Hi Malaka,

 Do we support the $subject? Basically we use an Inbound as the source
 and use file connector as the destination. If this is not supported yet, we
 got to add this to ESB 5.

 Thanks,
 Kasun

 --
 Kasun Indrasiri
 Software Architect
 WSO2, Inc.; http://wso2.com
 lean.enterprise.middleware

 cell: +94 77 556 5206
 Blog : http://kasunpanorama.blogspot.com/

>>>
>>>
>>>
>>> --
>>>
>>> Best Regards,
>>>
>>> Malaka Silva
>>> Senior Tech Lead
>>> M: +94 777 219 791
>>> Tel : 94 11 214 5345
>>> Fax

Re: [Dev] Streaming File transfer with File Inbound/File Connector

[Kasun Indrasiri]

This is great. Thanks a lot Malaka.
Also, if we use the file connector as the outbound channel, will it work in
the same way?

On Tue, Mar 1, 2016 at 12:36 PM, Malaka Silva  wrote:

> Hi All,
>
> Please ignore my previous comments. This can be done with current
> implementation.
>
> if (builder instanceof DataSourceMessageBuilder &&
> "true".equals(streaming)) {
> dataSource = ManagedDataSourceFactory.create(new
> FileObjectDataSource(file, contentType));
> in = null;
> } else {
> in = new
> AutoCloseInputStream(file.getContent().getInputStream());
> dataSource = null;
> }
> ..
> OMElement documentElement;
> if (in != null) {
> documentElement = builder.processDocument(in, contentType,
> axis2MsgCtx);
> } else {
> documentElement =
> ((DataSourceMessageBuilder)builder).processDocument(dataSource,
> contentType, axis2MsgCtx);
> }
> ...
> if(dataSource != null) {
> dataSource.destroy();
> }
>
> This is
> ​because ​
> message builder able to build messages from
> ​​
> DataSource objects.
> ​
> ​
> DataSource​
>  by definition the data from a DataSource can be read
> ​
> multiple times,
> ​builders that implement ​
> this interface
> ​can​
>  avoid storing the message content in memory.
> ​ ​
> If a message builder implements this interface and the
> ​file/vfs
>  is able to provide the message payload as a data source, then the method
> defined by this interface should be preferred over the method defined by
> Builder.
> ​ This
>  helps optimizing
> ​PT​
>  with
> ​vfs/file​.
> The builder will typically expose the data source directly or indirectly
> through the returned OMElement, e.g. by adding to the tree an OMText or
> OMDataSource node referencing the data source.
>
> ​I have checked this with inbound but there is a fix we need to do. I have
> done it in [1]. I have done several tests with and without streaming from
> 5mb to 1gb files.
>
> Without streaming [2] and [3] will show the memory growth. With streaming
> [4]. Without streaming most of the time ESB went OOM​.
>
> Find the related configs and axis2 changes in [5] and [6].
>
> [1] https://wso2.org/jira/browse/ESBJAVA-4458
>
> [2]
>
> [image: Inline image 1]
>
> [3]
>
> [image: Inline image 2]
>
> [4]
>
> [image: Inline image 3]
>
> [5]
>
> http://ws.apache.org/ns/synapse";  name="load"
>  sequence="request"  onError="fault" protocol="file" suspend="false">
>
>   1
>   true
>name="transport.vfs.ContentType">application/file
>   false
>   false
>   DELETE
>   true
>   true
>   DELETE
>name="transport.vfs.FileURI">file:///home/wso2/work/tmp/file/in
>   false
>   true
>   enable
>   true
>   NONE
>   false
>
> 
>
> 
> 
> 
> 
>  value="true"/>
> 
> 
> 
> 
>
> [6]
>  class="org.apache.axis2.format.BinaryBuilder"/>
>
> 
> class="org.wso2.carbon.relay.ExpandingMessageFormatter"/>
>
> On Mon, Feb 29, 2016 at 12:02 PM, Malaka Silva  wrote:
>
>> Hi Kasun,
>>
>> Currently no OOB solution with ESB 4.9.0. File always gets build before
>> mediation.
>>
>> However use case mentioned can be handled with Schedule task -> File
>> connector search -> File connector copy
>>
>> +1 for OOB solution.
>>
>> We can  do this for ESB 5.0. Also there is an option to do this as a
>> custom inbound. So that this can be used by previous ESB versions as well.
>>
>> So default inbound can be used, if someone needs to do a mediation on
>> file content and custom inbound for PT file use case. WDYT?
>>
>>
>> On Mon, Feb 29, 2016 at 11:49 AM, Kasun Indrasiri  wrote:
>>
>>> Hi Malaka,
>>>
>>> Do we support the $subject? Basically we use an Inbound as the source
>>> and use file connector as the destination. If this is not supported yet, we
>>> got to add this to ESB 5.
>>>
>>> Thanks,
>>> Kasun
>>>
>>> --
>>> Kasun Indrasiri
>>> Software Architect
>>> WSO2, Inc.; http://wso2.com
>>> lean.enterprise.middleware
>>>
>>> cell: +94 77 556 5206
>>> Blog : http://kasunpanorama.blogspot.com/
>>>
>>
>>
>>
>> --
>>
>> Best Regards,
>>
>> Malaka Silva
>> Senior Tech Lead
>> M: +94 777 219 791
>> Tel : 94 11 214 5345
>> Fax :94 11 2145300
>> Skype : malaka.sampath.silva
>> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
>> Blog : http://mrmalakasilva.blogspot.com/
>>
>> WSO2, Inc.
>> lean . enterprise . middleware
>> http://www.wso2.com/
>> http://www.wso2.com/about/team/malaka-silva/
>> 
>> https://store.wso2.com/store/
>>
>> Save a tree -Conserve nature & Save the world for your future. Print this
>> email only if it is absolutely necessary.
>>
>
>
>
> --
>
> Best Regards,
>
> Malaka Silva
> Senior Tech Lead
> M: +94 777 219 791
>

Re: [Dev] Streaming File transfer with File Inbound/File Connector

[Malaka Silva]

Hi All,

Please ignore my previous comments. This can be done with current
implementation.

if (builder instanceof DataSourceMessageBuilder &&
"true".equals(streaming)) {
dataSource = ManagedDataSourceFactory.create(new
FileObjectDataSource(file, contentType));
in = null;
} else {
in = new
AutoCloseInputStream(file.getContent().getInputStream());
dataSource = null;
}
..
OMElement documentElement;
if (in != null) {
documentElement = builder.processDocument(in, contentType,
axis2MsgCtx);
} else {
documentElement =
((DataSourceMessageBuilder)builder).processDocument(dataSource,
contentType, axis2MsgCtx);
}
...
if(dataSource != null) {
dataSource.destroy();
}

This is
​because ​
message builder able to build messages from
​​
DataSource objects.
​
​
DataSource​
 by definition the data from a DataSource can be read
​
multiple times,
​builders that implement ​
this interface
​can​
 avoid storing the message content in memory.
​ ​
If a message builder implements this interface and the
​file/vfs
 is able to provide the message payload as a data source, then the method
defined by this interface should be preferred over the method defined by
Builder.
​ This
 helps optimizing
​PT​
 with
​vfs/file​.
The builder will typically expose the data source directly or indirectly
through the returned OMElement, e.g. by adding to the tree an OMText or
OMDataSource node referencing the data source.

​I have checked this with inbound but there is a fix we need to do. I have
done it in [1]. I have done several tests with and without streaming from
5mb to 1gb files.

Without streaming [2] and [3] will show the memory growth. With streaming
[4]. Without streaming most of the time ESB went OOM​.

Find the related configs and axis2 changes in [5] and [6].

[1] https://wso2.org/jira/browse/ESBJAVA-4458

[2]

[image: Inline image 1]

[3]

[image: Inline image 2]

[4]

[image: Inline image 3]

[5]

http://ws.apache.org/ns/synapse";  name="load"
 sequence="request"  onError="fault" protocol="file" suspend="false">
   
  1
  true
  application/file
  false
  false
  DELETE
  true
  true
  DELETE
  file:///home/wso2/work/tmp/file/in
  false
  true
  enable
  true
  NONE
  false
   












[6]




On Mon, Feb 29, 2016 at 12:02 PM, Malaka Silva  wrote:

> Hi Kasun,
>
> Currently no OOB solution with ESB 4.9.0. File always gets build before
> mediation.
>
> However use case mentioned can be handled with Schedule task -> File
> connector search -> File connector copy
>
> +1 for OOB solution.
>
> We can  do this for ESB 5.0. Also there is an option to do this as a
> custom inbound. So that this can be used by previous ESB versions as well.
>
> So default inbound can be used, if someone needs to do a mediation on file
> content and custom inbound for PT file use case. WDYT?
>
>
> On Mon, Feb 29, 2016 at 11:49 AM, Kasun Indrasiri  wrote:
>
>> Hi Malaka,
>>
>> Do we support the $subject? Basically we use an Inbound as the source and
>> use file connector as the destination. If this is not supported yet, we got
>> to add this to ESB 5.
>>
>> Thanks,
>> Kasun
>>
>> --
>> Kasun Indrasiri
>> Software Architect
>> WSO2, Inc.; http://wso2.com
>> lean.enterprise.middleware
>>
>> cell: +94 77 556 5206
>> Blog : http://kasunpanorama.blogspot.com/
>>
>
>
>
> --
>
> Best Regards,
>
> Malaka Silva
> Senior Tech Lead
> M: +94 777 219 791
> Tel : 94 11 214 5345
> Fax :94 11 2145300
> Skype : malaka.sampath.silva
> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
> Blog : http://mrmalakasilva.blogspot.com/
>
> WSO2, Inc.
> lean . enterprise . middleware
> http://www.wso2.com/
> http://www.wso2.com/about/team/malaka-silva/
> 
> https://store.wso2.com/store/
>
> Save a tree -Conserve nature & Save the world for your future. Print this
> email only if it is absolutely necessary.
>



-- 

Best Regards,

Malaka Silva
Senior Tech Lead
M: +94 777 219 791
Tel : 94 11 214 5345
Fax :94 11 2145300
Skype : malaka.sampath.silva
LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
Blog : http://mrmalakasilva.blogspot.com/

WSO2, Inc.
lean . enterprise . middleware
http://www.wso2.com/
http://www.wso2.com/about/team/malaka-silva/

https://store.wso2.com/store/

Save a tree -Conserve nature & Save the world for your future. Print this
email only if it is absolutely necessary.
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] Streaming File transfer with File Inbound/File Connector

[Malaka Silva]

Hi Kasun,

Currently no OOB solution with ESB 4.9.0. File always gets build before
mediation.

However use case mentioned can be handled with Schedule task -> File
connector search -> File connector copy

+1 for OOB solution.

We can  do this for ESB 5.0. Also there is an option to do this as a custom
inbound. So that this can be used by previous ESB versions as well.

So default inbound can be used, if someone needs to do a mediation on file
content and custom inbound for PT file use case. WDYT?


On Mon, Feb 29, 2016 at 11:49 AM, Kasun Indrasiri  wrote:

> Hi Malaka,
>
> Do we support the $subject? Basically we use an Inbound as the source and
> use file connector as the destination. If this is not supported yet, we got
> to add this to ESB 5.
>
> Thanks,
> Kasun
>
> --
> Kasun Indrasiri
> Software Architect
> WSO2, Inc.; http://wso2.com
> lean.enterprise.middleware
>
> cell: +94 77 556 5206
> Blog : http://kasunpanorama.blogspot.com/
>



-- 

Best Regards,

Malaka Silva
Senior Tech Lead
M: +94 777 219 791
Tel : 94 11 214 5345
Fax :94 11 2145300
Skype : malaka.sampath.silva
LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
Blog : http://mrmalakasilva.blogspot.com/

WSO2, Inc.
lean . enterprise . middleware
http://www.wso2.com/
http://www.wso2.com/about/team/malaka-silva/

https://store.wso2.com/store/

Save a tree -Conserve nature & Save the world for your future. Print this
email only if it is absolutely necessary.
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] Streaming File transfer with File Inbound/File Connector

[Kasun Indrasiri]

Hi Malaka,

Do we support the $subject? Basically we use an Inbound as the source and
use file connector as the destination. If this is not supported yet, we got
to add this to ESB 5.

Thanks,
Kasun

-- 
Kasun Indrasiri
Software Architect
WSO2, Inc.; http://wso2.com
lean.enterprise.middleware

cell: +94 77 556 5206
Blog : http://kasunpanorama.blogspot.com/
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev