Re: send(), sendmsg(), sendto() not thread-safe
David et al, I think you may be missing the point. David S. Miller wrote: I don't understand why the desire is so high to ensure that individual threads get atomic writes, you can't even ensure that in the general case. I think Mark's point isn't about atomic writes; instead, he was pointing out that when a programmer reads that a call is thread-safe he usually understands that to mean that it does not need to be protected by a lock when used by multiple threads. With the current implementation of sendmsg it takes some imagination to come up with scenario where it is okay for concurrent sendmsg calls to mix their data. Only sloppy programs that don't do their own internal locking hit into issues in this area. You are looking at it backwards. If sendmsg were indeed thread-safe (two concurrent calls to sendmsg would not mix their data), then it would be stupid for a programmer to put a lock around the sendmsg call. It would also be stupid for a programmer to protect a call to socket() with a lock. This thread-safe issue is an even bigger deal when you are sharing the socket amoung many independent processes since locking in that context is a bit harder and more inefficient than in a pthread context. So, if it isn't going to be fixed. It would be nice to at least document the issue. A cursory examination of the sendmsg kernel code leads you to believe that it is thread-safe in the sense that Mark is talking about. ben - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send(), sendmsg(), sendto() not thread-safe
Benjamin Reed wrote: I think Mark's point isn't about atomic writes; instead, he was pointing out that when a programmer reads that a call is thread-safe he usually understands that to mean that it does not need to be protected by a lock when used by multiple threads. snip This thread-safe issue is an even bigger deal when you are sharing the socket amoung many independent processes since locking in that context is a bit harder and more inefficient than in a pthread context. There have always been possible issues with concurrent access to sockets/fds. Consider what happens if you share a socket between processes/threads, you call select() in both of them, then do a blocking recv() in both to receive the message. One of them gets the message, and the other blocks unexpectedly. Or consider calling lseek() from one task, while doing successive read() calls in the other. Or imagine one thread doing recvmsg() with MSG_PEEK, then calling recvmsg() to get the data, but meanwhile another thread has already read the message. The calls themselves are thread-safe, but the side-effects of them may lead to unexpected behaviour unless the designer enforces appropriate synchronization. Chris - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send(), sendmsg(), sendto() not thread-safe
You are using the wrong examples, which may be why you don't understand the problem Mark identified. --- Christopher Friesen [EMAIL PROTECTED] wrote: There have always been possible issues with concurrent access to sockets/fds. snip Or consider calling lseek() from one task, while doing successive read() calls in the other. I think all but the most clueless programmer would know that lseek() and read() need to be in a critical section. That is not the point. We are not talking about thread-safe across calls. We are talking about thread-safety for a given call. In the case of lseek() and read(), you can use pread(). pread() is cool because you don't need to protect it with a critical section. Two threads can do a pread() to different offsets on the same file handle. If you protect the pread() with a critical section, you are just wasting code, which is sloppy. When reading the doc for sendmsg, it appears you can avoid the critical section for sendmsg as well, but as Mark points out, you need the critical section unless you are running on Windows. ben - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send(), sendmsg(), sendto() not thread-safe
Benjamin Reed wrote: In the case of lseek() and read(), you can use pread(). What is the meaning of the offset parameter of pread() for TCP or UDP etc? rick jones - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send(), sendmsg(), sendto() not thread-safe
On Mon, May 15, 2006 at 06:50:36PM -0700, Mark A Smith wrote: I cannot think of another possible definition for thread-safe in the context of these functions. Certainly they should not cause a crash when called from multiple threads, but that's a requirement independent of thread-safety. ... I think the question really boils down to: What does thread-safe mean with respect to send()? It might be more easily answered by asking, How would a non-thread-safe send() behave? I think it would behave the way we're seeing it behave. A non-thread-safe send() could use a global buffer to hold copied data or results. When called from multiple threads it could lose data or mix data from send()s to different sockets. Or it could return a result in one thread that reflects the result of an overlapping send() in another thread. Or it could sometimes core dump. The behavior of a non-thread-safe function called from multiple threads is undefined. There is really no requirement independent of thread-safety on the behavior of a function called from multiple unsyncronized threads. As Rick noted, this is not really a threading test but a multiprocess test. A requirement to be thread-safe shouldn't imply anything about the behavior of functions in single-threaded processes. The posix standard has a limit of PIPE_BUF bytes for atomic writes to a pipe. That might extend to expecting sends of under PIPE_BUF bytes to be atomic on a socket. But I don't actually find any guarantee of atomic writes for sockets, even for sizes below PIPE_BUF. -- Mike Stroyan, [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send(), sendmsg(), sendto() not thread-safe
From: Mark A Smith [EMAIL PROTECTED] Date: Mon, 15 May 2006 14:39:06 -0700 I discovered that in some cases, send(), sendmsg(), and sendto() are not thread-safe. Although the man page for these functions does not specify whether these functions are supposed to be thread-safe, my reading of the POSIX/SUSv3 specification tells me that they should be. I traced the problem to tcp_sendmsg(). I was very curious about this issue, so I wrote up a small page to describe in more detail my findings. You can find it at: http://www.almaden.ibm.com/cs/people/marksmith/sendmsg.html . I don't understand why the desire is so high to ensure that individual threads get atomic writes, you can't even ensure that in the general case. Only sloppy programs that don't do their own internal locking hit into issues in this area. From your findings, the vast majority of systems you investigated do not provide atomic thread safe write semantics over TCP sockets. And frankly, BSD defines BSD socket semantics here not some wording in the POSIX standards. Finally, this discussion belongs on the networking development mailing list, netdev@vger.kernel.org, not linux-kernel. - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send(), sendmsg(), sendto() not thread-safe
On Mon, 15 May 2006 16:17:48 -0700 Rick Jones [EMAIL PROTECTED] wrote: David S. Miller wrote: From: Mark A Smith [EMAIL PROTECTED] Date: Mon, 15 May 2006 14:39:06 -0700 I discovered that in some cases, send(), sendmsg(), and sendto() are not thread-safe. Although the man page for these functions does not specify whether these functions are supposed to be thread-safe, my reading of the POSIX/SUSv3 specification tells me that they should be. I traced the problem to tcp_sendmsg(). I was very curious about this issue, so I wrote up a small page to describe in more detail my findings. You can find it at: http://www.almaden.ibm.com/cs/people/marksmith/sendmsg.html . # ./sendmsgclient localhost ERROR! We should have all 0! We don't! buff[16384]=1 buff[16385]=1 buff[16386]=1 buff[16387]=1 buff[16388]=1 buff[16389]=1 buff[16390]=1 buff[16391]=1 buff[16392]=1 buff[16393]=1 That's 10/32768 bad bytes # uname -a HP-UX tarry B.11.23 U ia64 2397028692 unlimited-user license Given that the URL above asserts that HP-UX claims atomicity, either there is a bug in the UX stack, or perhaps the test? I took a quick look at the HP-UX 11iv2 (aka 11.23) manpage for sendmsg and didn't see anything about atomicity there - on which manpage(s) or docs was the assertion of HP-UX atomicity made? I presume this is only for blocking sockets? I cannot at least off the top of my head see how a stack could offer it on non-blocking sockets. The test seems to be based on sending a big message. In this case, on non-blocking sockets, the send call will return partial status. The return from the system call will be less than the number of bytes requested. And frankly, BSD defines BSD socket semantics here not some wording in the POSIX standards. Have BSD socket semantics ever been updated/clarified any any quasi-official manner since the popular presence of threads? Or are/were Posix/Xopen filling a gap? rick jones - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send(), sendmsg(), sendto() not thread-safe
I presume this is only for blocking sockets? I cannot at least off the top of my head see how a stack could offer it on non-blocking sockets. The test seems to be based on sending a big message. In this case, on non-blocking sockets, the send call will return partial status. The return from the system call will be less than the number of bytes requested. Right, and at that point it is already too late to do anything about keeping some other thread of execution from shoving _its_ bytes into the socket before the rest of the first sends' can be put there. (Perhaps I'm preaching to the choir) rick jones - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send(), sendmsg(), sendto() not thread-safe
I cannot think of another possible definition for thread-safe in the
context of these functions. Certainly they should not cause a crash when
called from multiple threads, but that's a requirement independent of
thread-safety. The POSIX specification doesn't mention atomicity with
respect to these functions. In the context of read(), it defines atomicity
as follows: Atomic means that all the bytes from a single operation that
started out together end up together, without interleaving from other I/O
operations. This seems to be the semantics I'm expecting from a
thread-safe sendmsg(). What else could thread-safe mean for send(),
sendmsg()?
As for fork() vs. pthreads, my original post indicated that this occurs
with or without pthreads, the same problem occurs if you use pthreads. I
tried to indicate that I was using the term thread to be independent of
whether the address space was shared or not.
If you do not see any send() sent 10 bytes messages in your server
output, it means that the problem occured the very first time the 10-byte
send was attempted. If you watch the data going over the wire, you'll see
those 10 bytes of 1's.
That write() document is excellent, and reads exactly like the POSIX
specification.
As for limits, I modified my test to use very small buffers, below any
reasonable limit. By removing the printf's, which slow the server
considerably, I can still get the problem to produce.
I think the question really boils down to: What does thread-safe mean with
respect to send()? It might be more easily answered by asking, How would
a non-thread-safe send() behave? I think it would behave the way we're
seeing it behave.
-Mark
That's excellent documentation, and reads exactly to the write()
specification in POSIX.
|-+
| | Rick Jones |
| | [EMAIL PROTECTED]|
| | om |
| ||
| | 05/15/2006 05:43 |
| | PM |
|-+
-|
|
|
| To: Mark A Smith/Almaden/[EMAIL PROTECTED]
|
| cc: [EMAIL PROTECTED], Linux Network Development list
netdev@vger.kernel.org|
| Subject: Re: send(), sendmsg(), sendto() not thread-safe
|
-|
Mark A Smith wrote:
Hi Rick, Stephen,
The thread-safe claim is at:
http://devrsrc1.external.hp.com/STKS/cgi-bin/man2html?manpage=/usr/share/man/man2.Z/send.2
Specifically,
MULTITHREAD USAGE
The send(), sendmsg(), and sendto() system calls are thread-safe.
They each have a cancellation point; and they are async-cancel
safe,
async-signal safe, and fork-safe.
That looks to be the 11iv1 manpage (aka 11.11).I wonder if perhaps
there is a distinction made between thread-safety and an atomicity
semantic?
Also, _strictly_ speaking, since your test is calling fork() rather than
pthread_create(), it isn't really testing thread safty but multiple
process atomicity right?
I noticed that you were thinking that the problem may be with my test and
that the send call is returning partial status.
Either the test, or the stack.
Actually, that's exactly
the issue. On the systems I tested on, and I assume HP-UX, send is _not_
returning partial status, it is returning that the entire buffer has been
written, and yet is interleaving data from packets in the other thread.
Ostensibly, I should see some ten byte send messages in the output of
sendmsgserver yes? I just ran a test where it was all 32768's, no 10's
but the client still reported an error. Is that supposed to be possible?
# sendmsgserver sendmsgserver.log
# wc sendmsgserver.log
165888 663552 3981312 sendmsgserver.log
# grep -v 32768 sendmsgserver.log
#
# ./sendmsgclient localhost
ERROR! We should have all 0! We don't!
buff[16384]=1
buff[16385]=1
buff[16386]=1
buff[16387]=1
buff[16388]=1
buff[16389]=1
buff[16390]=1
buff[16391]=1
buff[16392]=1
buff[16393]=1
That's 10/32768 bad bytes
I've also seen it fail at 12288 rather than 16384. I wonder if perhaps
there are unstated limits to the size of the write that can be atomic?
Looking at the 11.11 manpage for write(2) in the discussion of writes to
a pipe or FIFO it says:
+ Write requests of {PIPE_BUF} bytes or less will not be
interleaved with data from other processes doing writes on the
same pipe. Writes of greater than {PIPE_BUF} bytes may have
data interleaved, on arbitrary boundaries
