Hi All,
Please ignore my previous comments. This can be done with current
implementation.
if (builder instanceof DataSourceMessageBuilder &&
"true".equals(streaming)) {
dataSource = ManagedDataSourceFactory.create(new
FileObjectDataSource(file, contentType));
in = null;
} else {
in = new
AutoCloseInputStream(file.getContent().getInputStream());
dataSource = null;
}
..............
OMElement documentElement;
if (in != null) {
documentElement = builder.processDocument(in, contentType,
axis2MsgCtx);
} else {
documentElement =
((DataSourceMessageBuilder)builder).processDocument(dataSource,
contentType, axis2MsgCtx);
}
...........
if(dataSource != null) {
dataSource.destroy();
}
This is
because
message builder able to build messages from
DataSource objects.
DataSource
by definition the data from a DataSource can be read
multiple times,
builders that implement
this interface
can
avoid storing the message content in memory.
If a message builder implements this interface and the
file/vfs
is able to provide the message payload as a data source, then the method
defined by this interface should be preferred over the method defined by
Builder.
This
helps optimizing
PT
with
vfs/file.
The builder will typically expose the data source directly or indirectly
through the returned OMElement, e.g. by adding to the tree an OMText or
OMDataSource node referencing the data source.
I have checked this with inbound but there is a fix we need to do. I have
done it in [1]. I have done several tests with and without streaming from
5mb to 1gb files.
Without streaming [2] and [3] will show the memory growth. With streaming
[4]. Without streaming most of the time ESB went OOM.
Find the related configs and axis2 changes in [5] and [6].
[1] https://wso2.org/jira/browse/ESBJAVA-4458
[2]
[image: Inline image 1]
[3]
[image: Inline image 2]
[4]
[image: Inline image 3]
[5]
<inboundEndpoint xmlns="http://ws.apache.org/ns/synapse"; name="load"
sequence="request" onError="fault" protocol="file" suspend="false">
<parameters>
<parameter name="interval">10000</parameter>
<parameter name="coordination">true</parameter>
<parameter
name="transport.vfs.ContentType">application/file</parameter>
<parameter name="transport.vfs.LockReleaseSameNode">false</parameter>
<parameter name="transport.vfs.AutoLockRelease">false</parameter>
<parameter name="transport.vfs.ActionAfterFailure">DELETE</parameter>
<parameter name="transport.vfs.CreateFolder">true</parameter>
<parameter name="sequential">true</parameter>
<parameter name="transport.vfs.ActionAfterProcess">DELETE</parameter>
<parameter
name="transport.vfs.FileURI">file:///home/wso2/work/tmp/file/in</parameter>
<parameter name="transport.vfs.DistributedLock">false</parameter>
<parameter name="transport.vfs.Streaming">true</parameter>
<parameter name="transport.vfs.Locking">enable</parameter>
<parameter name="transport.vfs.FileSortAscending">true</parameter>
<parameter name="transport.vfs.FileSortAttribute">NONE</parameter>
<parameter name="transport.vfs.Build">false</parameter>
</parameters>
</inboundEndpoint>
<endpoint name="fileout">
<address uri="vfs:file:///home/wso2/work/tmp/file/out"/>
</endpoint>
<sequence name="request" onError="fault">
<property name="OUT_ONLY" scope="default" type="STRING"
value="true"/>
<send>
<endpoint key="fileout"/>
</send>
</sequence>
[6]
<messageBuilder contentType="application/file"
class="org.apache.axis2.format.BinaryBuilder"/>
<messageFormatter contentType="application/file"
class="org.wso2.carbon.relay.ExpandingMessageFormatter"/>
On Mon, Feb 29, 2016 at 12:02 PM, Malaka Silva <[email protected]> wrote:
> Hi Kasun,
>
> Currently no OOB solution with ESB 4.9.0. File always gets build before
> mediation.
>
> However use case mentioned can be handled with Schedule task -> File
> connector search -> File connector copy
>
> +1 for OOB solution.
>
> We can do this for ESB 5.0. Also there is an option to do this as a
> custom inbound. So that this can be used by previous ESB versions as well.
>
> So default inbound can be used, if someone needs to do a mediation on file
> content and custom inbound for PT file use case. WDYT?
>
>
> On Mon, Feb 29, 2016 at 11:49 AM, Kasun Indrasiri <[email protected]> wrote:
>
>> Hi Malaka,
>>
>> Do we support the $subject? Basically we use an Inbound as the source and
>> use file connector as the destination. If this is not supported yet, we got
>> to add this to ESB 5.
>>
>> Thanks,
>> Kasun
>>
>> --
>> Kasun Indrasiri
>> Software Architect
>> WSO2, Inc.; http://wso2.com
>> lean.enterprise.middleware
>>
>> cell: +94 77 556 5206
>> Blog : http://kasunpanorama.blogspot.com/
>>
>
>
>
> --
>
> Best Regards,
>
> Malaka Silva
> Senior Tech Lead
> M: +94 777 219 791
> Tel : 94 11 214 5345
> Fax :94 11 2145300
> Skype : malaka.sampath.silva
> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
> Blog : http://mrmalakasilva.blogspot.com/
>
> WSO2, Inc.
> lean . enterprise . middleware
> http://www.wso2.com/
> http://www.wso2.com/about/team/malaka-silva/
> <http://wso2.com/about/team/malaka-silva/>
> https://store.wso2.com/store/
>
> Save a tree -Conserve nature & Save the world for your future. Print this
> email only if it is absolutely necessary.
>
--
Best Regards,
Malaka Silva
Senior Tech Lead
M: +94 777 219 791
Tel : 94 11 214 5345
Fax :94 11 2145300
Skype : malaka.sampath.silva
LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
Blog : http://mrmalakasilva.blogspot.com/
WSO2, Inc.
lean . enterprise . middleware
http://www.wso2.com/
http://www.wso2.com/about/team/malaka-silva/
<http://wso2.com/about/team/malaka-silva/>
https://store.wso2.com/store/
Save a tree -Conserve nature & Save the world for your future. Print this
email only if it is absolutely necessary.
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
