This is great. Thanks a lot Malaka. Also, if we use the file connector as the outbound channel, will it work in the same way?
On Tue, Mar 1, 2016 at 12:36 PM, Malaka Silva <[email protected]> wrote: > Hi All, > > Please ignore my previous comments. This can be done with current > implementation. > > if (builder instanceof DataSourceMessageBuilder && > "true".equals(streaming)) { > dataSource = ManagedDataSourceFactory.create(new > FileObjectDataSource(file, contentType)); > in = null; > } else { > in = new > AutoCloseInputStream(file.getContent().getInputStream()); > dataSource = null; > } > .............. > OMElement documentElement; > if (in != null) { > documentElement = builder.processDocument(in, contentType, > axis2MsgCtx); > } else { > documentElement = > ((DataSourceMessageBuilder)builder).processDocument(dataSource, > contentType, axis2MsgCtx); > } > ........... > if(dataSource != null) { > dataSource.destroy(); > } > > This is > because > message builder able to build messages from > > DataSource objects. > > > DataSource > by definition the data from a DataSource can be read > > multiple times, > builders that implement > this interface > can > avoid storing the message content in memory. > > If a message builder implements this interface and the > file/vfs > is able to provide the message payload as a data source, then the method > defined by this interface should be preferred over the method defined by > Builder. > This > helps optimizing > PT > with > vfs/file. > The builder will typically expose the data source directly or indirectly > through the returned OMElement, e.g. by adding to the tree an OMText or > OMDataSource node referencing the data source. > > I have checked this with inbound but there is a fix we need to do. I have > done it in [1]. I have done several tests with and without streaming from > 5mb to 1gb files. > > Without streaming [2] and [3] will show the memory growth. With streaming > [4]. Without streaming most of the time ESB went OOM. > > Find the related configs and axis2 changes in [5] and [6]. > > [1] https://wso2.org/jira/browse/ESBJAVA-4458 > > [2] > > [image: Inline image 1] > > [3] > > [image: Inline image 2] > > [4] > > [image: Inline image 3] > > [5] > > <inboundEndpoint xmlns="http://ws.apache.org/ns/synapse"; name="load" > sequence="request" onError="fault" protocol="file" suspend="false"> > <parameters> > <parameter name="interval">10000</parameter> > <parameter name="coordination">true</parameter> > <parameter > name="transport.vfs.ContentType">application/file</parameter> > <parameter name="transport.vfs.LockReleaseSameNode">false</parameter> > <parameter name="transport.vfs.AutoLockRelease">false</parameter> > <parameter name="transport.vfs.ActionAfterFailure">DELETE</parameter> > <parameter name="transport.vfs.CreateFolder">true</parameter> > <parameter name="sequential">true</parameter> > <parameter name="transport.vfs.ActionAfterProcess">DELETE</parameter> > <parameter > name="transport.vfs.FileURI">file:///home/wso2/work/tmp/file/in</parameter> > <parameter name="transport.vfs.DistributedLock">false</parameter> > <parameter name="transport.vfs.Streaming">true</parameter> > <parameter name="transport.vfs.Locking">enable</parameter> > <parameter name="transport.vfs.FileSortAscending">true</parameter> > <parameter name="transport.vfs.FileSortAttribute">NONE</parameter> > <parameter name="transport.vfs.Build">false</parameter> > </parameters> > </inboundEndpoint> > > <endpoint name="fileout"> > <address uri="vfs:file:///home/wso2/work/tmp/file/out"/> > </endpoint> > <sequence name="request" onError="fault"> > <property name="OUT_ONLY" scope="default" type="STRING" > value="true"/> > <send> > <endpoint key="fileout"/> > </send> > </sequence> > > [6] > <messageBuilder contentType="application/file" > class="org.apache.axis2.format.BinaryBuilder"/> > > <messageFormatter contentType="application/file" > > class="org.wso2.carbon.relay.ExpandingMessageFormatter"/> > > On Mon, Feb 29, 2016 at 12:02 PM, Malaka Silva <[email protected]> wrote: > >> Hi Kasun, >> >> Currently no OOB solution with ESB 4.9.0. File always gets build before >> mediation. >> >> However use case mentioned can be handled with Schedule task -> File >> connector search -> File connector copy >> >> +1 for OOB solution. >> >> We can do this for ESB 5.0. Also there is an option to do this as a >> custom inbound. So that this can be used by previous ESB versions as well. >> >> So default inbound can be used, if someone needs to do a mediation on >> file content and custom inbound for PT file use case. WDYT? >> >> >> On Mon, Feb 29, 2016 at 11:49 AM, Kasun Indrasiri <[email protected]> wrote: >> >>> Hi Malaka, >>> >>> Do we support the $subject? Basically we use an Inbound as the source >>> and use file connector as the destination. If this is not supported yet, we >>> got to add this to ESB 5. >>> >>> Thanks, >>> Kasun >>> >>> -- >>> Kasun Indrasiri >>> Software Architect >>> WSO2, Inc.; http://wso2.com >>> lean.enterprise.middleware >>> >>> cell: +94 77 556 5206 >>> Blog : http://kasunpanorama.blogspot.com/ >>> >> >> >> >> -- >> >> Best Regards, >> >> Malaka Silva >> Senior Tech Lead >> M: +94 777 219 791 >> Tel : 94 11 214 5345 >> Fax :94 11 2145300 >> Skype : malaka.sampath.silva >> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77 >> Blog : http://mrmalakasilva.blogspot.com/ >> >> WSO2, Inc. >> lean . enterprise . middleware >> http://www.wso2.com/ >> http://www.wso2.com/about/team/malaka-silva/ >> <http://wso2.com/about/team/malaka-silva/> >> https://store.wso2.com/store/ >> >> Save a tree -Conserve nature & Save the world for your future. Print this >> email only if it is absolutely necessary. >> > > > > -- > > Best Regards, > > Malaka Silva > Senior Tech Lead > M: +94 777 219 791 > Tel : 94 11 214 5345 > Fax :94 11 2145300 > Skype : malaka.sampath.silva > LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77 > Blog : http://mrmalakasilva.blogspot.com/ > > WSO2, Inc. > lean . enterprise . middleware > http://www.wso2.com/ > http://www.wso2.com/about/team/malaka-silva/ > <http://wso2.com/about/team/malaka-silva/> > https://store.wso2.com/store/ > > Save a tree -Conserve nature & Save the world for your future. Print this > email only if it is absolutely necessary. > -- Kasun Indrasiri Software Architect WSO2, Inc.; http://wso2.com lean.enterprise.middleware cell: +94 77 556 5206 Blog : http://kasunpanorama.blogspot.com/
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
