I agree and can understand the time factor and also +1 for reusing stuff than trying to invent the wheel again :-)
Thanks, Ruwan On Sun, Mar 8, 2009 at 4:08 PM, Andreas Veithen <[email protected]>wrote: > Ruwan, > > It's not a question of possibility, it is a question of available time :-) > > Also note that some of the features that we might want to implement > have some similarities with what is done for attachments in Axiom > (except that an attachment is only available once, while a file over > VFS can be read several times). I think there is also some existing > code in Axis2 that might be useful. We should not reimplement these > things but try to make the existing code reusable. This however is > only realistic for the next release after 1.3. > > Andreas > > On Sun, Mar 8, 2009 at 03:47, Ruwan Linton <[email protected]> wrote: > > Andreas, > > > > Can we have the caching at the file system as a property to support the > > multiple layers touching the full message and is it possible make it to > > specify a threshold for streaming? For example if the message is touched > > several time we might still need streaming but not for the 100KB or > lesser > > files. > > > > Thanks, > > Ruwan > > > > On Sun, Mar 8, 2009 at 1:12 AM, Andreas Veithen < > [email protected]> > > wrote: > >> > >> I've done an initial implementation of this feature. It is available > >> in trunk and should be included in the next nightly build. In order to > >> enable this in your configuration, you need to add the following > >> property to the proxy: > >> > >> <parameter name="transport.vfs.Streaming">true</parameter> > >> > >> You also need to add the following mediators just before the <send> > >> mediator: > >> > >> <property action="remove" name="transportNonBlocking" scope="axis2"/> > >> <property action="set" name="OUT_ONLY" value="true"/> > >> > >> With this configuration Synapse will stream the data directly from the > >> incoming to the outgoing transport without storing it in memory or in > >> a temporary file. Note that this has two other side effects: > >> * The incoming file (or connection in case of a remote file) will only > >> be opened on demand. In this case this happens during execution of the > >> <send> mediator. > >> * If during the mediation the content of the file is needed several > >> time (which is not the case in your example), it will be read several > >> times. The reason is of course that the content is not cached. > >> > >> I tested the solution with a 2GB file and it worked fine. The > >> performance of the implementation is not yet optimal, but at least the > >> memory consumption is constant. > >> > >> Some additional comments: > >> * The transport.vfs.Streaming property has no impact on XML and SOAP > >> processing: this type of content is processed exactly as before. > >> * With the changes described here, we have now two different policies > >> for plain text and binary content processing: in-memory caching + no > >> streaming (transport.vfs.Streaming=false) and no caching + deferred > >> connection + streaming (transport.vfs.Streaming=true). Probably we > >> should define a wider range of policies in the future, including file > >> system caching + streaming. > >> * It is necessary to remove the transportNonBlocking property > >> (MessageContext.TRANSPORT_NON_BLOCKING) to prevent the <send> mediator > >> (more precisely the OperationClient) from executing the outgoing > >> transport in a separate thread. This property is set by the incoming > >> transport. I think this is a bug since I don't see any valid reason > >> why the transport that handles the incoming request should determine > >> the threading behavior of the transport that sends the outgoing > >> request to the target service. Maybe Asankha can comment on this? > >> > >> Andreas > >> > >> On Thu, Mar 5, 2009 at 07:21, kimhorn <[email protected]> wrote: > >> > > >> > Thats good; as this stops us using Synapse. > >> > > >> > > >> > > >> > Asankha C. Perera wrote: > >> >> > >> >> > >> >>> Exception in thread "vfs-Worker-4" java.lang.OutOfMemoryError: Java > >> >>> heap > >> >>> space > >> >>> at > >> >>> > >> >>> > java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:99) > >> >>> at > >> >>> > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:518) > >> >>> at java.lang.StringBuffer.append(StringBuffer.java:307) > >> >>> at java.io.StringWriter.write(StringWriter.java:72) > >> >>> at > org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1129) > >> >>> at org.apache.commons.io.IOUtils.copy(IOUtils.java:1104) > >> >>> at org.apache.commons.io.IOUtils.copy(IOUtils.java:1078) > >> >>> at org.apache.commons.io.IOUtils.toString(IOUtils.java:382) > >> >>> at > >> >>> > >> >>> > org.apache.synapse.format.PlainTextBuilder.processDocument(PlainTextBuilder.java:68) > >> >>> > >> >> Since the content type is text, the plain text formatter is trying to > >> >> use a String to parse as I see.. which is a problem for large > content.. > >> >> > >> >> A definite bug we need to fix .. > >> >> > >> >> cheers > >> >> asankha > >> >> > >> >> -- > >> >> Asankha C. Perera > >> >> AdroitLogic, http://adroitlogic.org > >> >> > >> >> http://esbmagic.blogspot.com > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> --------------------------------------------------------------------- > >> >> To unsubscribe, e-mail: [email protected] > >> >> For additional commands, e-mail: [email protected] > >> >> > >> >> > >> >> > >> > > >> > -- > >> > View this message in context: > >> > > http://www.nabble.com/VFS---Synapse-Memory-Leak-tp22344176p22345904.html > >> > Sent from the Synapse - Dev mailing list archive at Nabble.com. > >> > > >> > > >> > --------------------------------------------------------------------- > >> > To unsubscribe, e-mail: [email protected] > >> > For additional commands, e-mail: [email protected] > >> > > >> > > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > > > > > > > > -- > > Ruwan Linton > > http://wso2.org - "Oxygenating the Web Services Platform" > > http://ruwansblog.blogspot.com/ > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- Ruwan Linton http://wso2.org - "Oxygenating the Web Services Platform" http://ruwansblog.blogspot.com/
