Re: [Python-Dev] thread safe SMTP module
Gordon Messmer [EMAIL PROTECTED] wrote: After some discussion, Aahz suggested that I discuss the problem here, on python-dev. He seemed to think that the problem I saw may have been an indication of a bug in python. Could anyone take a look at that thread and say whether it looks like a bug, or working with non-blocking sockets was the right thing to do? Well, not having to faff around with non-blocking IO is one of the major advantages of threading, so if blocking IO is indeed the cause of your problem then it certainly sounds like a bug in Python, or possibly your operating system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thread safe SMTP module
On Wed, Mar 14, 2007, Jon Ribbens wrote: Gordon Messmer [EMAIL PROTECTED] wrote: After some discussion, Aahz suggested that I discuss the problem here, on python-dev. He seemed to think that the problem I saw may have been an indication of a bug in python. Could anyone take a look at that thread and say whether it looks like a bug, or working with non-blocking sockets was the right thing to do? Well, not having to faff around with non-blocking IO is one of the major advantages of threading, so if blocking IO is indeed the cause of your problem then it certainly sounds like a bug in Python, or possibly your operating system. One small wrinkle (and the reason I suggested bringing this to python-dev): I suspect that the problem is not a bug, but simply the occasional failure of sockets. When that happens in a threaded app without timeouts, eventually threads die (block forever). But you apparently can't use timeouts with file-wrapped sockets, so if that's the problem, we need to pick one of: * switch to non-wrapped sockets * switch to non-blocking I/O for smtplib * make file-wrapped sockets work with timeouts (And please note that if my analysis is correct, we need to make the fix for non-threaded apps, it's just more unusual to encounter the problem there.) This level of design decision is beyond my capabilities. -- Aahz ([EMAIL PROTECTED]) * http://www.pythoncraft.com/ I disrespectfully agree. --SJM ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thread safe SMTP module
Gordon Messmer [EMAIL PROTECTED] wrote: After some discussion, Aahz suggested that I discuss the problem here, on python-dev. He seemed to think that the problem I saw may have been an indication of a bug in python. Could anyone take a look at that thread and say whether it looks like a bug, or working with non-blocking sockets was the right thing to do? His most recent message in that thread actually said, No, at this point I think this is neither bug nor about thread blocking on I/O. I think it's about sockets dying and the inability of sockets in blocking mode to recover. I have seen this kind of behavior in single-threaded systems. I would concur with Aahz, more or less. I have also seen this behavior on single-threaded blocking systems. The real issue may be related to blocking sockets breaking coupled with socket.makefile(...).readline() not being able to handle the breakage. If you can, try to dig into the file object implementation returned by socket.makefile(). If you don't want to (I would understand), try replacing smtplib's use of 'self.file.readline()' with a Python variant that handles failures more gracefully. Heck, you could even add timeouts (which are being added to httplib and either urllib or urllib2). - Josiah As an aside, for even minimally loaded systems, I've noticed that file.readline() isn't any faster than just using a while loop with socket.recv() and checking for '\n'. I believe that it would make sense to replace certain libraries' (poplib, smtplib, etc.) use of file.readline() with that of a _readline() method that has a default implementation using either a file.readline() or sock.recv() in a loop. By pulling it out into a single method, the user could easily override it if it isn't doing what they want/expect. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thread safe SMTP module
Aahz [EMAIL PROTECTED] wrote: One small wrinkle (and the reason I suggested bringing this to python-dev): I suspect that the problem is not a bug, but simply the occasional failure of sockets. When that happens in a threaded app without timeouts, eventually threads die (block forever). But you apparently can't use timeouts with file-wrapped sockets, so if that's the problem, we need to pick one of: * switch to non-wrapped sockets * switch to non-blocking I/O for smtplib * make file-wrapped sockets work with timeouts In case it's helpful, I've put the file-like-sockets-with-timeouts code that I wrote up here: http://www.unequivocal.co.uk/dl/socketutils.py This is a pure-Python implementation that provides a file-like interface to a socket, with timeouts, that definitely works with threaded programs. Note it was written in 2002, so any part of the Python 'file' interface which was added since then is not in there. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thread safe SMTP module
Aahz wrote: One small wrinkle (and the reason I suggested bringing this to python-dev): I suspect that the problem is not a bug, but simply the occasional failure of sockets. When that happens in a threaded app without timeouts, eventually threads die (block forever). IIRC, my whole process was locking up, not just individual threads. Tonight I should have time to pull an old copy of the code out of CVS and recreate the test script that I used. Once I have, it should be a matter of feeding a big list of email addresses to the script and waiting a couple of minutes for the script to lock up. The last time I looked, the problem was very easy to observe. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thread safe SMTP module
Gordon Messmer [EMAIL PROTECTED] wrote: Tonight I should have time to pull an old copy of the code out of CVS and recreate the test script that I used. Once I have, it should be a matter of feeding a big list of email addresses to the script and waiting a couple of minutes for the script to lock up. The last time I looked, the problem was very easy to observe. Try it using the alternative makefile code I posted and see if it makes a difference... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
