Hi, On Wed, Jul 27, 2011 at 5:39 PM, Andreas Veithen <[email protected]>wrote:
> On Wed, Jul 27, 2011 at 13:22, Sadeep Jayasumana <[email protected]> > wrote: > > Hi, > > > > On Wed, Jul 27, 2011 at 4:12 PM, Andreas Veithen < > [email protected]> > > wrote: > >> > >> On Wed, Jul 27, 2011 at 04:45, Sadeep Jayasumana <[email protected] > > > >> wrote: > >> > Hi, > >> >> Whether streaming is possible or not depends on how the > >> >> MIME parts are accessed, but you always need to support buffering if > >> >> necessary. > >> > Yes, existing functionality will not be broken. In fact, building the > >> > Part > >> > in memory will be the default behavior. Streaming will kick in only if > >> > it is > >> > explicitly enabled. > >> >> The Attachments object is one of the first things that will be > created > >> >> when a new message is received. Therefore nobody will be able to set > a > >> >> property in the message context, i.e. the setting would necessarily > be > >> >> a global property in axis2.xml. This however is a problem if you have > >> >> services with diverging requirements. The decision to buffer or not > >> >> the content of the MIME part can't be taken at this stage in the > >> >> processing. It can only be taken when the MIME part is actually > >> >> accessed (which includes serializing the message to forward it) > >> > I agree with your comment. This functionality could be provided by > >> > introducing a new method, setAttachmentStreaming(), to Attachments > class > >> > or > >> > MessageContext class. However, I'm wondering whether it is > >> > an elegant way of > >> > doling this. Other ways of doing the same involves significant > >> > modifications > >> > to Axiom API. > >> > >> I don't think so. The desired way of doing this would be to have an > >> Axiom specific subclass of DataHandler (or a DataHandler backed by an > >> Axiom specific implementation of DataSource?) with a method such as: > >> > >> InputStream getInputStream(boolean preserve) > > > > Coming back to our original problem, we need SOAPMessageFormatter to > stream > > attachments when attachment streaming is explicitly enabled (via a > > configuration parameter or a message context property). If we introduce > such > > a new API method, calling it from the SOAPMessageFormatter would involve > > adding number of new methods to different classes. > > Thanks, > > Sadeep > > Nope. The code in SOAPMessageFormatter that is responsible for the > MTOM case is as follows: > > OMElement element = msgCtxt.getEnvelope(); > if (preserve) { > element.serialize(out, format); > } else { > element.serializeAndConsume(out, format); > } > > One would simply extend the concept of "consume" to the MIME parts and > the magic would happen behind the scenes inside Axiom. The SwA case is > a bit different because Axis2 will manipulate the attachments > directly. However, I don't think that covering that case requires > introducing new methods in lots of different places. > > serializeAndConsume() method ultimately retrieves data handlers to parts of the message. This results in Attachments#getDataHandler() method call. At this point Axiom has three options; building the part in memory (existing PartOnMemory implementation), building the part in a File (existing PartOnFile implementation) or streaming the attachment directly (introducing StreamingPart implementation). This decision is based on the state of the Attachments object that is set elsewhere. This is the existing behavior of the code. I introduced StreamingPart implementation without changing this behavior such that there is a minimal impact on the existing behavior and the API. However, as you suggest, if we are to organize things such that serializeAndConsume() always means attachment streaming, we will have to the refactor entire MIME message handling logic of Axiom while being careful not to break existing stuff or other components using Axiom. This is a significant effort as coming up with a perfect attachment streaming implementation that works in all the scenarios in not trivial as you have initially mentioned in this mail thread. But my point is, till the complete refactoring of MIME handling logic is done, this proposed solution adds some new feature to the existing code and enables Axis2 to work efficiently in a common use case: calling a web service with a large attachment. And this feature will come into play only if it is explicit enabled and hence will not break the existing stuff. Thanks, Sadeep >> > >> Probably we would have to introduce a couple of new APIs to make this > >> work behind the scene, but it would not imply modification of existing > >> APIs. Existing application code would continue to work as usual > >> (except maybe in the case of a missing MIME part) and the decision to > >> stream or not is taken just in time. > >> > >> > Thanks, > >> > Sadeep > >> > On Tue, Jul 26, 2011 at 12:03 AM, Andreas Veithen > >> > <[email protected]> wrote: > >> >> > >> >> Some time ago I was thinking about this issue. It is highly non > >> >> trivial to solve. Consider the following variant of your example: > >> >> > >> >> public void useBinaryData(DataHandler dh1, DataHandler dh2) { > >> >> try { > >> >> OutputStream out = new FileOutputStream(new > >> >> File("output1.zip")); > >> >> dh1.writeTo(out); > >> >> out = new FileOutputStream(new File("output2.zip")); > >> >> dh2.writeTo(out); > >> >> } catch (IOException e) { > >> >> throw new RuntimeException(e); > >> >> } > >> >> } > >> >> > >> >> Taking into account that the client is not required to send the MIME > >> >> parts in any particular order and that the service is not required to > >> >> consume them in any particular order, you will always end up with > >> >> scenarios where you have no other choice than to buffer some of the > >> >> MIME parts. Whether streaming is possible or not depends on how the > >> >> MIME parts are accessed, but you always need to support buffering if > >> >> necessary. > >> >> > >> >> What this example also shows is that you need to defer the calls to > >> >> Attachments#getDataHandler(String) until the very last moment. When > >> >> working with an Axiom tree, this is already the case: the call to > >> >> Attachments#getDataHandler(String) occurs when > OMText#getDataHandler() > >> >> is called (and not when the OMText node is built). However, when > using > >> >> a data binding, one would have to create some sort of DataHandler > >> >> proxy that defers the call to Attachments#getDataHandler(String). > >> >> > >> >> Andreas > >> >> > >> >> On Mon, Jul 25, 2011 at 09:21, Sadeep Jayasumana > >> >> <[email protected]> > >> >> wrote: > >> >> > Hi Devs, > >> >> > Here is the benefit of this feature from Axis2's perspective. > >> >> > > >> >> > Currently, when I use Axis2 to deploy a service class as follows, > >> >> > public class MTOMService { > >> >> > public void useBinaryData(String username, DataHandler > >> >> > dataHandler) > >> >> > { > >> >> > try { > >> >> > System.out.println("Name : " + username); // line1 > >> >> > OutputStream out = new FileOutputStream(new > >> >> > File("output.zip")); > >> >> > dataHandler.writeTo(out); // line2 > >> >> > System.out.println("Saving done!"); > >> >> > } catch (IOException e) { > >> >> > throw new RuntimeException(e); > >> >> > } > >> >> > } > >> >> > } > >> >> > the entire attachment is loaded before useBinaryData() method is > >> >> > called. > >> >> > Therefore, when a large attachment is used execution of line 1 will > >> >> > be significantly delayed. > >> >> > However, when the suggested feature is implemented, stream will be > >> >> > read > >> >> > only > >> >> > when it is absolutely needed (i.e., in line 2). Therefore, > execution > >> >> > of > >> >> > line1 will happen right after receiving the client request. > >> >> > Enabling attachment streaming will be similar to enabling file > >> >> > caching > >> >> > of attachments [1], an axis2.xml parameter or MessageContext > property > >> >> > could > >> >> > be used. > >> >> > [1] > http://axis.apache.org/axis2/java/core/docs/mtom-guide.html#a41 > >> >> > Thanks, > >> >> > Sadeep > >> >> > > >> >> > On Mon, Jul 25, 2011 at 11:46 AM, Sadeep Jayasumana > >> >> > <[email protected]> > >> >> > wrote: > >> >> >> > >> >> >> Hi Devs, > >> >> >> In Apache Syanpse, we have a requirement to proxy an MTOM enabled > >> >> >> web > >> >> >> service with minimum overhead. Large files (even in GB range) > should > >> >> >> be > >> >> >> able > >> >> >> to go through Synapse without running it OOM. > >> >> >> To satisfy this requirement, Synapse should be able to forward an > >> >> >> incoming > >> >> >> SOAP message with MTOM attachments to the backend service without > >> >> >> building > >> >> >> the attachments. Synapse might read/modify the SOAP envelop but > not > >> >> >> the > >> >> >> attachments. Therefore, it should be possible to stream > attachments > >> >> >> directly > >> >> >> from the Synpase's client to the backend service. > >> >> >> However, in the current implementation of AXIOM and Axis2, MTOM > >> >> >> attachments are built (in memory or in a file) by > >> >> >> SOAPMessageFormatter. > >> >> >> This > >> >> >> caused Synapse to run OOM when in the above mentioned scenario. > >> >> >> I have come up with a fix for this. It is to introduce a > >> >> >> new org.apache.axiom.attachments.impl.AbstractPart implementation > >> >> >> which > >> >> >> streams non-soap MIME parts without building them. > >> >> >> To introduce this new feature without breaking existing stuff, I'm > >> >> >> planning to introduce a new message context property which enables > >> >> >> MTOM > >> >> >> streaming. org.apache.axis2.builder.BuilderUtils class will check > >> >> >> this > >> >> >> property in the message context and > >> >> >> create org.apache.axiom.attachments.Attachments > >> >> >> object accordingly. Does > >> >> >> this sound like the correct way of introducing this feature? > >> >> >> Appreciate your feedback. > >> >> >> Thanks, > >> >> >> -- > >> >> >> Sadeep Jayasumana > >> >> >> Software Engineer, > >> >> >> WSO2 Inc. > >> >> > > >> >> > -- > >> >> > Sadeep Jayasumana > >> >> > Software Engineer, > >> >> > WSO2 Inc. > >> >> > >> >> --------------------------------------------------------------------- > >> >> To unsubscribe, e-mail: [email protected] > >> >> For additional commands, e-mail: [email protected] > >> >> > >> > > >> > > >> > > >> > -- > >> > Sadeep > >> > > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
