Re: Is SOAP appropriate for large data transmission broken pipe exception

Fri, 23 Jun 2006 00:55:30 -0700 

Hi,
it seems that brocken pipe exception (transfering large files axis2 version 1) is a kind of a bug. We have installed the latest nightly build version, and broken pipe exception has disapeared. 

German

Rodrigo Ruiz schrieb:

Hi Betsy, German,


When you say FTP is faster than HTTP do you mean in equality conditions, 
or do you mean "a native FTP server is faster than a Java HTTP server"? 
AFAIK, both protocols introduce a little overhead for the connection 
establishment, and after that, just the raw data is sent over the wire. 
Perhaps the difference lies on the packet sizes? What happens if you 
need to secure the access to the file? Is it possible to use a common 
shared AAA mechanism for both servers?



The limits you mention for large attachments are, in part, related to 
the Axis implementation, and not to the SOAP protocol. I think there is 
a size limit for MIME attachments from the MIME multipart encoding 
specification. The document I read (a long time ago) stated that the 
size of the parts must be a 32bit integer. This leads to a 2Gb limit if 
you use a Java "int" to represent this value. I really don't know if 
this limit is still there, or if the specification has been updated 
since I read it, nor if Axis respects it. As MTOM is compatible with 
MIME attachments, I guess it shares the same limitations. I think it is 
important to note that this limit is only for a single MIME part, that 
is, a "smart" transfer service could split larger files into several 
smaller MIME parts and send them as separate attachments into a single 
response. DIME attachments, if streamed, do not require their size to be 
specified, and so, do not have such a limitation.



I have successfully transferred 10GByte files with Axis 1.4, using DIME 
attachments and chunked transfer (yes, this one do introduce an extra 
overhead :-D), so I guess broken pipe exceptions should be considered 
bugs to be solved, rather than technology limitations ;-)


Best Regards,
Rodrigo Ruiz

Betsy Frey wrote:


Hi German,

I have not tried sending attachments with Axis2.

Betsy


-----Original Message-----

From: German Sakaryan [mailto:[EMAIL PROTECTED] Sent: 
Thursday, June 22, 2006 4:43 AM

To: [email protected]
Subject: Re: Is SOAP appropriate for large data transmission

Hi,Betsey
have tried to send large files with axis2 ?

I get fast always broken pipe exception.

German Sakaryan
Betsy Frey schrieb:


SOAP without an attachment is inappropriate for sending large amounts


of binary data, because the XML requires it to be converted to Base64,
which is inefficient.


With an attachment, SOAP over http has the overhead of the protocol,


which is not as fast as ftp, for instance.


I have used DIME attachments in Axis 1.1.  They worked fine up to some


limit, 2gb or so as I recall; 2 ^ 31 or 32.


Betsy

-----Original Message-----

From: Rodrigo Ruiz [mailto:[EMAIL PROTECTED] Sent: Thursday, 
June 22, 2006 2:51 AM

To: [email protected]
Subject: Is SOAP appropriate for large data transmission

Hi all,


I usually read in this forum that web services and SOAP are not 
appropriate for large data transmission, but I have never read about


any


good technical reasons behind these sentences.


Most of the issues I have seen in this matter are related to the SOAP 
stack implementation, rather than to SOAP itself:


- Progress bar for file download: You can split the download in


smaller


chunks and transmit them using several calls. It should be possible to




modify the attachments implementation of SOAP and add the logic for 
asynchronous status notification, so it is technically possible to


track


the progress of attachments download/upload.

- SOAP XML overhead: In fact, the overhead of a SOAP envelope


containing


just a reference to the attachment part is more important for small 
attachments than for large ones. Just one or two kilobytes of XML are 
not too much when you are trying to transfer an 8GB stream.


- Partial/broken transmissions: Again, if the attachments


implementation


is able to track the progress of a download, it may be possible to 
resume it after a transmission error. We would just need to be able to




specify the range we are interested in, just like with old plain HTML.



Could somebody shed some light on the reasons that make SOAP 
inappropriate for this task?


Regards,
Rodrigo Ruiz




---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


**********************************************************************

This email and any files transmitted with it are confidential and 
intended solely for the use of the individual or entity to whom they 
are addressed. Any unauthorized review, use, disclosure or 
distribution is prohibited. If you are not the intended recipient, 
please contact the sender by reply e-mail and destroy all copies of 
the original message.



---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]









---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]















    











					Reply via email to