[issue32608] Incompatibilities with the socketserver and multiprocessing packages
Michael Durso <rbprogram...@gmail.com> added the comment: I added more commits to the PR based on your comments. Although I cannot figure out why on the new Process objects create Threads that do not get cleaned up. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32608] Incompatibilities with the socketserver and multiprocessing packages
Michael Durso <rbprogram...@gmail.com> added the comment: Hi Antoine, were you able to check out the pull request? I'd like to get the issues resolved so the code can be merged into the next version. Thanks! -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32608] Incompatibilities with the socketserver and multiprocessing packages
Michael Durso <rbprogram...@gmail.com> added the comment: Hi Antoine, no worries for the delay. I've been looking into why the AppVeyor check is failing. I believe it to be an issue with the test harness in that it changes the environment during the tests, but only in the Windows case. I have been thinking about ignoring the failing tests in Windows and create new tests just for the Windows case. But I'm not sure that's the best approach. Ideally the test harness should be able to either not modify the environment of the process or revert the changes it has made. I'm not a strong Windows developer, so with my crude Windows development environment I'm struggling to see why the tests work in my Linux dev environment but not my Windows VM one. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32608] Incompatibilities with the socketserver and multiprocessing packages
Change by Michael Durso <rbprogram...@gmail.com>: -- keywords: +patch pull_requests: +5104 stage: -> patch review ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32608] Incompatibilities with the socketserver and multiprocessing packages
New submission from Michael Durso <rbprogram...@gmail.com>: This is more of an issue with socketserver rather than multiprocessing. I could not find socketserver experts to add to the nosy list, but I've included the multiprocessing experts for completeness. Under specific conditions, multiprocessing queues cease to work when modules like socketserver is used, more specifically when fork() is used. This makes sense that things can break when fork() is used since a simple forked process has things like file descriptors and threads invalidated. The socketserver.ForkingMixIn class, which has very reasonable use-cases, causes multiprocessing queues and threads to break. A script that replicates and simulates the issue is attached. The attached script sets up a multiprocessing queue used for logging across all sub-processes, then uses a socketserver.ForkingTCPServer to handle new client connection. What's happening is the fork() is invalidating the thread in the QueueHandler in each new sub-process (for each new client conncetion). It turns the QueueHandler into a dead object. In this scenario, the only indication the problem exists is the fact that log messages are not actually logged. But I think it's plausible that more than just log data could be lost. When a non-modified version of CPython, the script should run and produce a log file. However, logs from the client connection handler (MyClientHandler) will be lost and never sent to the logging handler that performs the actual file writing. To exercise the proposed fix to CPython, build the changes to socketserver.py and then change the script's "USE_FIX" flag to "True" and run the script. -- components: Library (Lib) files: multiprocessing_socketserver.py messages: 310371 nosy: davin, pitrou, rbprogrammer priority: normal severity: normal status: open title: Incompatibilities with the socketserver and multiprocessing packages type: behavior versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8 Added file: https://bugs.python.org/file47398/multiprocessing_socketserver.py ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
