[Python-Dev] Socket timeout and completion based sockets
Regarding the recent discussion on python-ideas about asyncronous IO, I'd like to ask a question about python socket's Timeout feature. Specifically this: Is it a documented or a guaranteed feature that a send/receive operation that times out with a socket.timeout error is re-startable on that socket? The reason I ask is that depending on the implementation, a timeout may leave a socket in an undefined state. As it is implemented in the standard cPython implementation, the timeout feature is done with an internal select() call. Thus, if the select() call times out, socket send/receive is not even called, so a retry is possible without issue. However, other implementations of python sockets, e.g. ones that rely on IO completion, may not have the luxury of using select. For example, on Windows, there is no way to abort an IOCP socket call, so a timeout must be implemented by aborting the wait. Dealing with the resulting race can be an interesting challenge. K ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] type vs. class terminology
Hm. None of the distinctions brought up so far really hit true with me (though they all are part of the picture). For example, I think the distinction between type(x) and x.__class__ is rarely significant -- I bet that if anyone were to rely on this they'd first have to change a lot of code that used one or the other without much thinking about the difference. So we might as well consider these equivalent. Part of the distinction is probably just in historical usage -- many idioms hark back to when there *was* a difference. If I had to invent an artificial difference, I'd say that I use "type" when talking about the type as a fairly abstract concept, such as the type of a variable (which may be required to be in a given set, for example), whereas I'll say "class" when I'm interested in the class as an object, or its definition (through a class statement). So, if I'm going to ask for the name, the phrase "the name of the class" rolls easier off my tongue than "the name of the type". OTOH if I'm going to just assert set membership, I might slightly prefer "x's type must be int or str". It's clear that we ought to at least cross-link the two glossary entries and point out that they are the same concept (type(x) vs. x.__class__) notwithstanding. I don't think it's important to try and eradicate the word type from our conversation or our docs -- so no sweeping global changes, please. However, if you come across a use of either one in the docs that seems odd given modern usage, feel free to clean it up, perhaps after checking with someone else to make sure your intuition matches that of others. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Socket timeout and completion based sockets
If you're talking about the standard socket module, I'm not aware that it uses IOCP on Windows. Are you asking this just in the abstract, or do you know of a Python implementation that uses IOCP to implement the standard socket type? As to the design of the async I/O library (which I am still working on!), I cannot guarantee anything, and the issue will probably be moot -- the API won't have the same kind of timeout as the current socket object (it will have other ways to set deadlines though). Confusedly yours, --Guido On Mon, Nov 26, 2012 at 3:49 AM, Kristján Valur Jónsson wrote: > Regarding the recent discussion on python-ideas about asyncronous IO, I‘d > like to ask a question about python socket‘s Timeout feature. > > Specifically this: Is it a documented or a guaranteed feature that a > send/receive operation that times out with a socket.timeout error is > re-startable on that socket? > > > > The reason I ask is that depending on the implementation, a timeout may > leave a socket in an undefined state. > > As it is implemented in the standard cPython implementation, the timeout > feature is done with an internal select() call. Thus, if the select() call > times out, socket send/receive is not even called, so a retry is possible > without issue. > > However, other implementations of python sockets, e.g. ones that rely on IO > completion, may not have the luxury of using select. For example, on > Windows, there is no way to abort an IOCP socket call, so a timeout must be > implemented by aborting the wait. Dealing with the resulting race can be an > interesting challenge. > > > > K > > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Socket timeout and completion based sockets
On 26/11/2012 11:49am, Kristján Valur Jónsson wrote: However, other implementations of python sockets, e.g. ones that rely on IO completion, may not have the luxury of using select. For example, on Windows, there is no way to abort an IOCP socket call, so a timeout must be implemented by aborting the wait. Dealing with the resulting race can be an interesting challenge. I am not quite sure what you mean by "aborting the wait". But you can abort an overlapped operation using CancelIo()/CancelIoEx(). I have just done some experimenting. Using CancelIo()/CancelIoEx() to abort an operation started with WSARecv() does not seem to cause a problem -- you just call GetOverlappedResult() afterwards to check whether the operation completed before it could be aborted. But aborting an operation started with WSASend() sometimes seems to "break" the connection: a subsequent WSARecv()/WSASend() will fail with WSAECONNABORTED or WSAECONNRESET depending on which end of the connection you are on. So, as you say, if you abort a send then you cannot expect to successfully resend the data later. -- Richard ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Socket timeout and completion based sockets
On Nov 26, 2012, at 11:05 AM, Richard Oudkerk wrote: > Using CancelIo()/CancelIoEx() to abort an operation started with WSARecv() > does not seem to cause a problem (emphasis mine) Little command-line experiments are not the right way to verify the behavior of high-performance I/O APIs. You need to do careful reading of the documentation, significant testing under load and experiments on a huge variety of platforms. Windows runs in _lots_ of horrible, horrible places. I think that the safest option would really be to better document the somewhat mangled state that a timeout may leave a socket in. I don't believe the feature was intended for pipelined protocols; if you receive a timeout by using the stdlib timeout functionality you have generally burned the socket. And, generally, things that care enough about performance or scalability enough to use IOCP operations should never use timeout-sockets anyway; it may do a select() internally (and on Windows, where poll() is never available, it _will_ do a select() internally), which limits the number of file descriptors you might even have in your process before you start encountering spurious errors. The use case it supports is when you have a little tool that just needs to fetch a URL or something really simple, but wants to be able to get on with things if that doesn't work or takes too long. -glyph ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
