Hi, On Wed, Jul 27, 2011 at 4:12 PM, Andreas Veithen <[email protected]>wrote:
> On Wed, Jul 27, 2011 at 04:45, Sadeep Jayasumana <[email protected]> > wrote: > > Hi, > >> Whether streaming is possible or not depends on how the > >> MIME parts are accessed, but you always need to support buffering if > >> necessary. > > Yes, existing functionality will not be broken. In fact, building the > Part > > in memory will be the default behavior. Streaming will kick in only if it > is > > explicitly enabled. > >> The Attachments object is one of the first things that will be created > >> when a new message is received. Therefore nobody will be able to set a > >> property in the message context, i.e. the setting would necessarily be > >> a global property in axis2.xml. This however is a problem if you have > >> services with diverging requirements. The decision to buffer or not > >> the content of the MIME part can't be taken at this stage in the > >> processing. It can only be taken when the MIME part is actually > >> accessed (which includes serializing the message to forward it) > > I agree with your comment. This functionality could be provided by > > introducing a new method, setAttachmentStreaming(), to Attachments class > or > > MessageContext class. However, I'm wondering whether it is an elegant way > of > > doling this. Other ways of doing the same involves significant > modifications > > to Axiom API. > > I don't think so. The desired way of doing this would be to have an > Axiom specific subclass of DataHandler (or a DataHandler backed by an > Axiom specific implementation of DataSource?) with a method such as: > > InputStream getInputStream(boolean preserve) > Coming back to our original problem, we need SOAPMessageFormatter to stream attachments when attachment streaming is explicitly enabled (via a configuration parameter or a message context property). If we introduce such a new API method, calling it from the SOAPMessageFormatter would involve adding number of new methods to different classes. Thanks, Sadeep > Probably we would have to introduce a couple of new APIs to make this > work behind the scene, but it would not imply modification of existing > APIs. Existing application code would continue to work as usual > (except maybe in the case of a missing MIME part) and the decision to > stream or not is taken just in time. > > > Thanks, > > Sadeep > > On Tue, Jul 26, 2011 at 12:03 AM, Andreas Veithen > > <[email protected]> wrote: > >> > >> Some time ago I was thinking about this issue. It is highly non > >> trivial to solve. Consider the following variant of your example: > >> > >> public void useBinaryData(DataHandler dh1, DataHandler dh2) { > >> try { > >> OutputStream out = new FileOutputStream(new > >> File("output1.zip")); > >> dh1.writeTo(out); > >> out = new FileOutputStream(new File("output2.zip")); > >> dh2.writeTo(out); > >> } catch (IOException e) { > >> throw new RuntimeException(e); > >> } > >> } > >> > >> Taking into account that the client is not required to send the MIME > >> parts in any particular order and that the service is not required to > >> consume them in any particular order, you will always end up with > >> scenarios where you have no other choice than to buffer some of the > >> MIME parts. Whether streaming is possible or not depends on how the > >> MIME parts are accessed, but you always need to support buffering if > >> necessary. > >> > >> What this example also shows is that you need to defer the calls to > >> Attachments#getDataHandler(String) until the very last moment. When > >> working with an Axiom tree, this is already the case: the call to > >> Attachments#getDataHandler(String) occurs when OMText#getDataHandler() > >> is called (and not when the OMText node is built). However, when using > >> a data binding, one would have to create some sort of DataHandler > >> proxy that defers the call to Attachments#getDataHandler(String). > >> > >> Andreas > >> > >> On Mon, Jul 25, 2011 at 09:21, Sadeep Jayasumana <[email protected] > > > >> wrote: > >> > Hi Devs, > >> > Here is the benefit of this feature from Axis2's perspective. > >> > > >> > Currently, when I use Axis2 to deploy a service class as follows, > >> > public class MTOMService { > >> > public void useBinaryData(String username, DataHandler > dataHandler) > >> > { > >> > try { > >> > System.out.println("Name : " + username); // line1 > >> > OutputStream out = new FileOutputStream(new > >> > File("output.zip")); > >> > dataHandler.writeTo(out); // line2 > >> > System.out.println("Saving done!"); > >> > } catch (IOException e) { > >> > throw new RuntimeException(e); > >> > } > >> > } > >> > } > >> > the entire attachment is loaded before useBinaryData() method is > called. > >> > Therefore, when a large attachment is used execution of line 1 will > >> > be significantly delayed. > >> > However, when the suggested feature is implemented, stream will be > read > >> > only > >> > when it is absolutely needed (i.e., in line 2). Therefore, execution > of > >> > line1 will happen right after receiving the client request. > >> > Enabling attachment streaming will be similar to enabling file caching > >> > of attachments [1], an axis2.xml parameter or MessageContext property > >> > could > >> > be used. > >> > [1] http://axis.apache.org/axis2/java/core/docs/mtom-guide.html#a41 > >> > Thanks, > >> > Sadeep > >> > > >> > On Mon, Jul 25, 2011 at 11:46 AM, Sadeep Jayasumana > >> > <[email protected]> > >> > wrote: > >> >> > >> >> Hi Devs, > >> >> In Apache Syanpse, we have a requirement to proxy an MTOM enabled web > >> >> service with minimum overhead. Large files (even in GB range) should > be > >> >> able > >> >> to go through Synapse without running it OOM. > >> >> To satisfy this requirement, Synapse should be able to forward an > >> >> incoming > >> >> SOAP message with MTOM attachments to the backend service without > >> >> building > >> >> the attachments. Synapse might read/modify the SOAP envelop but not > the > >> >> attachments. Therefore, it should be possible to stream attachments > >> >> directly > >> >> from the Synpase's client to the backend service. > >> >> However, in the current implementation of AXIOM and Axis2, MTOM > >> >> attachments are built (in memory or in a file) by > SOAPMessageFormatter. > >> >> This > >> >> caused Synapse to run OOM when in the above mentioned scenario. > >> >> I have come up with a fix for this. It is to introduce a > >> >> new org.apache.axiom.attachments.impl.AbstractPart implementation > which > >> >> streams non-soap MIME parts without building them. > >> >> To introduce this new feature without breaking existing stuff, I'm > >> >> planning to introduce a new message context property which enables > MTOM > >> >> streaming. org.apache.axis2.builder.BuilderUtils class will check > this > >> >> property in the message context and > >> >> create org.apache.axiom.attachments.Attachments > >> >> object accordingly. Does > >> >> this sound like the correct way of introducing this feature? > >> >> Appreciate your feedback. > >> >> Thanks, > >> >> -- > >> >> Sadeep Jayasumana > >> >> Software Engineer, > >> >> WSO2 Inc. > >> > > >> > -- > >> > Sadeep Jayasumana > >> > Software Engineer, > >> > WSO2 Inc. > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > > > > > > > > -- > > Sadeep > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
