[issue13354] tcpserver should document non-threaded long-living connections
Bas Wijnen added the comment: Thank you for your fast response as well. I overlooked that paragraph indeed. It doesn't mention anything about avoiding a socket shutdown however. Keeping a list of requests isn't very useful if all the sockets in the list are closed. ;-) So I would indeed suggest an addition: I would change this paragraph: These four classes process requests synchronously; each request must be completed before the next request can be started. This isn’t suitable if each request takes a long time to complete, because it requires a lot of computation, or because it returns a lot of data which the client is slow to process. The solution is to create a separate process or thread to handle each request; the ForkingMixIn and ThreadingMixIn mix-in classes can be used to support asynchronous behaviour. into: By default, these four classes process requests synchronously; each request must be completed before the next request can be started. This isn’t suitable if each request takes a long time to complete, because it requires a lot of computation, or because it returns a lot of data which the client is slow to process, or because the information that should be sent to the client isn't available yet when the request is made. One possible solution is to create a separate process or thread to handle each request; the ForkingMixIn and ThreadingMixIn mix-in classes can be used to support asynchronous behaviour. Another option is to store the socket for later use, and disable the shutting down of the socket by overriding process_request with an function that only calls finish_request, and not shutdown_request. In that case, the socket must be shut down by the program when it is done with it. At the end of the last paragraph you refer to, it should also be mentioned that the program must do something to prevent the socket from being shut down. In the description of process_request, it would probably also be useful to add that the default implementation (as opposed to *MixIn) calls shutdown_request() after finish_request(). -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue13354> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13354] tcpserver should document non-threaded long-living connections
Bas Wijnen added the comment: For example, I have some programs which are HTTP servers that implement WebSockets. For regular web page requests, it is acceptable if the connection is closed after the page is sent. But for a WebSocket it isn't: the whole point of that protocol is to allow the server to send unsolicited messages to the browser. If the connection is closed, it cannot do that. The documentation currently suggests that the only way to avoid handle() closing the connection is to not return. That means that the only way to do other things is by using multi-processing or (shiver) multi-threading. My suggestion is to add a short explanation about how connections can be kept open after handle() returns, so that a single threaded event loop can be used. Of course the socket must then be manually closed when the program is done with it. If I understand you correctly, overriding process_request would allow handle() to return without closing the socket. That does sound better than overriding shutdown_request. In the current documentation (same link as before, now for version 3.5.1), there is no mention at all about close() or shutdown() of the accepted sockets. The only information on the subject is that if you want asynchronous handling of data, you must start a new thread or process. This is not true, and in many cases it is not what I would recommend. I think event loops are much easier to maintain and debug than multi-process applications, and infinitely easier than multi-threading applications. I don't mind that other people disagree, and I'm not suggesting that those ways of handling should be removed (multi-process has its place, and I can't convince everyone that multi-threading is evil). What I'm saying is that the ability to use an event loop to handle asynchronous data with this class deserves to be mentioned as well. Obviously, it does then need to have the explanation about what to override to make it work. My suggestion is simply what I ended up with after seeing it fail and reading the code. If what you describe is the recommended way, please say that instead. My point is that the scenario should presented as an option, I don't have an opinion on what it should say. -- status: pending -> open ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue13354> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21430] Document ssl.pending()
Bas Wijnen added the comment: Alexey: please be more civil. Antoine: In that case, can you please explain how you would recommend me to implement my use case, where most of my calls are master-initiated and blocking, but some slave-initiated events must be non-blocking? Should I make a lot of calls to sslsocket.setblocking() to switch it on and off all the time? AFAIK that is a system call (or isn't it?); while that won't make any real difference in performance in Python, it doesn't feel right to make system calls when there's technically no need for it. Also, as I suggested previously, if you don't document the method, could you please add the word pending somewhere in the text? This ensures people looking for documentation of what they see in the source will find this explanation. It may also be good to add a note to the source code that this function should not be used. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21430 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21430] Document ssl.pending()
Bas Wijnen added the comment: After trying to use this, I think ssl.pending is a valuable function that should be supported and documented. My use case is a half-synchronous system, where I want most calls to block but asynchronous events are possible as well. Switching the socket between blocking and non-blocking all the time is annoying and results in code that is harder to read. With ssl.pending, I can simply repeat the read as long as data is pending. Setting my buffer size high might technically work, but that seems too fragile. I don't like using an undocumented function either, but I'm hoping that gets fixed. ;-) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21430 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21430] Document ssl.pending()
Bas Wijnen added the comment: The documentation about non-blocking is clear enough, thank you for pointing it out. However, I would ask to write anything in there that contains the word pending. The reason is that I didn't find anything in the documentation, looked in the source, found the pending() method and searched the documentation to see how it was defined. If I would have found an explanation that I shouldn't be using it, even if that wasn't the documentation of the method, I would have done the right thing in my program. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21430 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21430] Document ssl.pending()
New submission from Bas Wijnen: In order to use ssl sockets asynchronously, it is important to use the pending() method, otherwise the internal buffer will be ignored, and select may block for new data while it's already waiting. See bug #16976 and http://stackoverflow.com/questions/21663705/how-to-use-select-with-python-ssl-socket-buffering Using pending() works fine, but is entirely undocumented. https://docs.python.org/2.7/library/ssl.html (and all other versions) don't even mention the existence of the method. I hope this can be changed; using an undocumented feature isn't nice, but in this case there is no other good solution. -- assignee: docs@python components: Documentation messages: 217884 nosy: docs@python, shevek priority: normal severity: normal status: open title: Document ssl.pending() versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21430 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13354] tcpserver should document non-threaded long-living connections
New submission from Bas Wijnen wij...@debian.org: http://docs.python.org/py3k/library/socketserver.html says: The solution is to create a separate process or thread to handle each request; the ForkingMixIn and ThreadingMixIn mix-in classes can be used to support asynchronous behaviour. There is another way, which doesn't bring multi-threading hell over you: keep a copy of the file descriptor and use it when events occur. I recall that this suggestion used to be in the documentation as well, but I can't find it anymore. It would be good to add this suggestion to the documentation. However, there is a thing you must know before you can use this approach: tcpserver calls shutdown() on the socket when handle() returns. This means that the network connection is closed. Even dup2() doesn't keep it open (it lets you keep a file descriptor, but it returns EOF). The solution for this is to override shutdown_request of your handler to only call close_request (or not call anything at all) for sockets which must remain open. That way, as long as there is a reference to the socket, the network connection will not be shut down. Optionally the socket can be shutdown() explicitly when you're done with the connection. Something like the paragraph above would be useful in the documentation IMO. -- assignee: docs@python components: Documentation messages: 147147 nosy: docs@python, shevek priority: normal severity: normal status: open title: tcpserver should document non-threaded long-living connections type: feature request versions: Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13354 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
