Martin, I have a similar design problem. I need to transfer huge amounts of data in a fast, efficient, scalable, reliable manner. I am using SOAP to communicate between the two systems and I would like to use SOAP to do the data transfer also, rather than look at something else.
Looking at where SOAP & Axis currently are, I have decided to chunk the data at application level myself and then then send that as SOAP Attachments (ugly but should work). In order to make efficient use of bandwidth, I am also planning to use the zlib library to compress the attachment. Ideally, I would have liked for SOAP & Axis to have built in support to do this. I think when DIME is accepted as a standard, that may be the right solution. Currently you can get more information on DIME at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnglobspec/html/dimeindex.asp. I am certainly interested in any alternative solutions that other people can recommend. Also would love to hear from you on what you finally decide to go with. thx, Vikas. --- Martin Jericho <[EMAIL PROTECTED]> wrote: > > ----- Original Message ----- > From: "Rogerio Saran" <[EMAIL PROTECTED]> > To: <[EMAIL PROTECTED]> > Sent: Tuesday, September 24, 2002 8:44 AM > Subject: Re: Streaming RPC calls > > > > Martin, forget webservices. Nowadays they are all > about "pull" or > > "submit", and you want to "push" data. > > I'm not too sure why you say I'm trying to "push" > data. All interaction is > initiated by the client. Do I have an outdated > notion of the terms push and > pull? > > > Did you consider a messaging system? > > No I haven't. I thought SOAP would be the most > appropriate because of the > cross-platform data encoding standard. I haven't > had much experience with > messaging. Are there messaging systems out there > that are cross-platform > and language independent? I thought they were all > tied to a particular > platform/language. A built-in data encoding > standard would also be very > nice, but not essential. Do you have one in > particular in mind? > > > > > First, you probably want to chunk it to make each > "delivery unit" more > > manageable. Next, you want them to be delivered as > chunks, so a broken > > connection will not compromise the whole transfer. > Finally you want it > > fast so you need to open several sockets at once. > > I thought about this as well. It was a compromise I > was willing to make to > gain the benefits of soap. I would certainly want > the chunking to be > abstracted out of the API though. I don't want to > have to worry about it on > the application level. > > > > > If you need to exchange a lot of data it does not > makes sense to force > > it through a single TCP connection. Most servers > have a limited ability > > to reach high speed in a single connection due > to TCP handshaking > > limitations. > > > > A quick and dirty recipe to transfer a zillion > records of structured > > data as a stream: > > > > a) Again, forget webservices and RPC. > > This is a raw data transfer problem. > > That is exactly what it isn't. It's not raw data, > it's structured data. It > has to be understood by different systems. If I > could borrrow the XML > encoding of SOAP and use it in a messaging system I > would be all for it, but > that sounds suspiciously like SOAP again, only using > a different transport > protocol. > > > > > b) Chunk your data and pack it in well formed XML > documents, making them > > suitable to be handled in lots of platforms. > > > > c) Send them through a messaging system. The > simpler, the better. Is > > SMTP good enough? Go for it. If you want a > sophisticated, "dog wags the > > tail" solution, there are also plenty of > message-queue servers around. > > > > d) Need to ensure chunk sequence? Here you will > cover your hands with > > some dirt to write a transport handshake > mechanism, like windows in TCP. > > Sequence doesn't matter too much, but the client > needs to get an answer back > when the entire message has been processed > successfully. I did not want to > have the client acting as a server as well because > of firewall issues, and > polling for the response is unresponsive and klunky, > so SMTP is definitely > out of the question. > > I'm certainly not committed to SOAP, but I would > appreciate it if you could > give me a pointer to even one example of an > appropriate messaging system. > > Thanks > Martin > > > > > > > *Saran > > > > Martin Jericho wrote: > > > > > > QUESTION 4: All I really want > to do is send a large > > > array of structured data in a > platform independent way. > > > I would like to use the standard > RPC encoding of SOAP to > > > avoid having to define my own > XML schemas for the data, > > > but I'm not sure whether today's > SOAP implementations > > > are mature enough to use for > this purpose. What do > > > other people do in this > situation? I can't imagine I'm > > > the only one. > > > > > > > > > Thanks > > > Martin Jericho > > > > > > > > > > > > > > > > > > > __________________________________________________ Do you Yahoo!? New DSL Internet Access from SBC & Yahoo! http://sbc.yahoo.com
