New submission from Michael Durso <rbprogram...@gmail.com>:

This is more of an issue with socketserver rather than multiprocessing.  I 
could not find socketserver experts to add to the nosy list, but I've included 
the multiprocessing experts for completeness.

Under specific conditions, multiprocessing queues cease to work when modules 
like socketserver is used, more specifically when fork() is used.  This makes 
sense that things can break when fork() is used since a simple forked process 
has things like file descriptors and threads invalidated.  The 
socketserver.ForkingMixIn class, which has very reasonable use-cases, causes 
multiprocessing queues and threads to break.  A script that replicates and 
simulates the issue is attached.

The attached script sets up a multiprocessing queue used for logging across all 
sub-processes, then uses a socketserver.ForkingTCPServer to handle new client 
connection.  What's happening is the fork() is invalidating the thread in the 
QueueHandler in each new sub-process (for each new client conncetion).  It 
turns the QueueHandler into a dead object.

In this scenario, the only indication the problem exists is the fact that log 
messages are not actually logged.  But I think it's plausible that more than 
just log data could be lost.

When a non-modified version of CPython, the script should run and produce a log 
file.  However, logs from the client connection handler (MyClientHandler) will 
be lost and never sent to the logging handler that performs the actual file 
writing.  To exercise the proposed fix to CPython, build the changes to 
socketserver.py and then change the script's "USE_FIX" flag to "True" and run 
the script.

----------
components: Library (Lib)
files: multiprocessing_socketserver.py
messages: 310371
nosy: davin, pitrou, rbprogrammer
priority: normal
severity: normal
status: open
title: Incompatibilities with the socketserver and multiprocessing packages
type: behavior
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8
Added file: https://bugs.python.org/file47398/multiprocessing_socketserver.py

_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32608>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Reply via email to