Re: Streaming RPC calls

[Martin Jericho]

Graham,   Could you let me know which are the vendors that tout a pure streaming approach?  (I realise you probably don't want to promote competitors too much, don't tell the boss!)   The requirement to stream millions of records is quite appropriate, the client isn't just a dumb user interface, and has to send as well as receive large amounts of data.  Any attempt to chop it up would be purely an implementation workaround.   If I send it as an attachment I may as well abandon SOAP, since I have to invent some way of encoding it in a cross-platform way.   Is there really no-one else who has faced this issue?  No comment from anyone else?   Martin   ----- Original Message ----- From: graham glass To: [EMAIL PROTECTED] Sent: Monday, September 23, 2002 5:54 PM Subject: RE: Streaming RPC calls hi martin,   glue currently creates a dom representation of a soap message before streaming. funnily enough, when doing benchmarking of soap messages that have an on-the-wire representation > 1Mbyte, this approach seems to be faster than vendors that tout a pure streaming approach.   i question whether your requirement of streaming millions of records in a single message is appropriate. is there not a better way in which the client requests a subset during a particular operation? alternatively, passing the records as a SOAP attachment bypasses this problem completely.   the main drawback that we've found regarding a pure streaming architecture (we have considered this many times in the past) is the extra complexity it injects at the API level for those who want to add interceptors, XSLT transforms, etc.   cheers, graham -----Original Message----- From: Martin Jericho [mailto:[EMAIL PROTECTED]] Sent: Monday, September 23, 2002 1:29 AM To: [EMAIL PROTECTED] Subject: Re: Streaming RPC calls Hi Graham   Thanks for your interest.  I have included some source files to show you how I did the streaming.  It's not a complete set so it doesn't compile, but it should give you an idea how the client side works at least.  (Sorry for posting an attachment everyone!  It's only 10K)   The original BatchItemStreamArray class as generated by WSDL2Java simply contains an array of BatchItem objects called batchItems.  Either the client or server can use this default implementation, at the risk of running into memory problems if the array is large.  The new BatchItemStreamArray class emulates the original class, but extends the StreamArray class, which contains all the hooks necessary to serialize/deserialize on the fly without having to actually create the array.  It's all a bit unrefined at the moment, since I'm just doing a proof of concept as a first step.   The application I am designing must be able to cope with hundreds of thousands or even millions of records.  The size of each record is dynamic, so I can't put an exact figure on the number of MB.   Does Glue have the same limitations as axis in this regard?   ----- Original Message ----- From: graham glass To: [EMAIL PROTECTED] Sent: Monday, September 23, 2002 2:38 PM Subject: RE: Streaming RPC calls hi martin,   out of interest, how would you write an interceptor to process inputs and outputs during the streaming process? one advantage of having a DOM represention is that it's easy to manipulate during inbound/outbound messaging using tools like xalan. would you provide some kind of streaming interceptor interface?   also, out of interest, how large is the data you are sending (on the wire)?   cheers, graham -----Original Message----- From: Martin Jericho [mailto:[EMAIL PROTECTED]] Sent: Sunday, September 22, 2002 11:31 PM To: [EMAIL PROTECTED] Subject: Streaming RPC calls I have implemented a special class which allows streaming of complex bean types in an RPC call, but have found that AXIS stores the whole message in memory before sending / deserializing it anyway, which defeats the whole purpose of what I am trying to do.   The first problem is that AXIS doesn't support chunked http messages, so the content length must be known before sending it.   QUESTION 1:  Are there any plans to implement chunked transfer encoding soon?   The SOAPPart class stores the entire message as a byte array before it is sent across the wire, which apart from the reason above is unnecessary.   QUESTION 2:  Are there any plans to clean this up to allow direct streaming from the serializers onto the wire?   I have only researched the client side to see how and why the entire message is stored in memory before sending it.  I have also noticed that the server stores it all in memory before deserializing it, but I haven't investigated whether there is any good reason for this.   QUESTION 3:  If AXIS won't support this any time soon, which I assume is the case, does anyone know whether the Sun JAXRPC reference implementation suffers the same problem?  I would also like to know whether the MS .NET implementation has the same problem if anyone knows about it.   QUESTION 4:  All I really want to do is send a large array of structured data in a platform independent way.  I would like to use the standard RPC encoding of SOAP to avoid having to define my own XML schemas for the data, but I'm not sure whether today's SOAP implementations are mature enough to use for this purpose.  What do other people do in this situation?  I can't imagine I'm the only one.   I would be more than happy to share my streaming serialization/deserialization code if anyone is interested, although it is not of much use with the current version of axis.   Thanks Martin Jericho