You know, one of these days I'm going to learn to press the right
button. I replied on this message only to the original author rather
than the mailing list so I am forwarding it here as well.
Mark Berryman
--- Begin Message ---
Craig A. Berry wrote:
>
> At 12:07 PM -0700 11/5/03, Michael Downey wrote:
> >"Carl Braesicke" <[EMAIL PROTECTED]> wrote in message
> >news:[EMAIL PROTECTED]
> >This script fragment hangs on OpenVMS until I trim "piece.out" below 64K
> >bytes; the same code works fine on a linux box with an untrimmed
> >"piece.out":
> >
> >Is there a 64K limit in the OpenVMS socket support?
> >
> >Regards,
> >
> >Carl B., x2796
> >
> >
> >Yes, there is a 64K limit on socket writes on VMS at least when VMS 7.3 came
> >out. Both writev and write to a socket can only accept a 64K message. I
> >believe that the perl abstraction layer should handle this but since your
> >seeing this problem it likely isn't and would need to be fixed.
>
> I've done a bit of digging on this issue, and it looks to me like
> there are plenty of bad assumptions to go around. Perhaps TCP/IP
> services should provide a way to send packets larger than 64K, though
> I'm not sure if that's even possible using a QIO interface (upon
> which the CRTL socket interface is clearly based) since the spot
> where it writes the number of bytes transferred in the IOSB is a
> 16-bit word. It is hard to find where, if anywhere, the 64K limit
> for sockets is documented.
The 64K limit for QIO is documented and applies to all devices that use
QIO (which includes sockets). To issue I/O requests larger than 64K
requires the use of FastIO, which is not currently supported by TCP/IP
services (although I hear rumors that it may one day be).
> Even on linux or other operating systems, the maximum buffer sizes
> are not guaranteed to be greater than 64K and are generally tuneable
> by the system manager. So unless you control all the pieces,
> including networking hardware, you can't assume very large packets
> won't cause problems.
>
> IO::Socket should probably check the maximum chunk it can send and
> return an error if it's passed something too large.
I wouldn't think so. IP programming MANDATES that the write function
return the number of bytes actually written (on a successful write) and
the program MUST check this value and, if it is not the same as what was
submitted to the write function, adjust the buffer and issue another
write. (Well, not strictly true. For datagram sockets failure to write
the entire buffer is an error but for stream sockets (i.e. TCP
connections) this method holds true).
If the write function of IO::Socket is not returning the number of bytes
written then it is broken and needs to be fixed. If it is and the
user's script is not checking and acting on that value, then it is the
user's script that is broken.
Now, maybe it was the intent of IO::Socket to do this effort on behalf
of the user. In that case, IO::Socket needs to accept the user's write
request and perform the above loop on behalf of the user, returning the
total bytes written.
However, that is not the way I read the documentation. As I read it,
IO::Socket simply provides an interface to the C send routine and does,
indeed, return the actual number of bytes written.
> Net::MySQL should probably not assume that an entire query will be
> sent as one packet, though the same assumption may exist on the
> server side.
It doesn't. Nothing can ever assume that anything will be sent as a
single packet. Even a single write by the client cannot be assumed to
reach the server as a single packet.
> The application should definitely not assume that it can suck an
> entire file into memory and send it to the database in one insert
> statement. Even if you manage to lift the 64K limit, you may
> eventually run into a 256K or 1MB or whatever limit. Pretty trivial
> changes to the script could do the inserts in more manageable pieces,
> say 1000 rows at a time. This is by far the easiest thing to change
> and should be done regardless of whether changes are possible in the
> underlying layers.
Most definitely.
Mark Berryman
--- End Message ---
