[issue6721] Locks in the standard library should be sanitized on fork

[Kyle Evans]

Change by Kyle Evans :


--
nosy: +kevans
nosy_count: 28.0 -> 29.0
pull_requests: +21627
pull_request: https://github.com/python/cpython/pull/22651

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Andrés Delfino]

Change by Andrés Delfino :


--
nosy:  -adelfino

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Stefan Behnel]

Change by Stefan Behnel :


--
nosy:  -scoder

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Stefan Behnel]

Change by Stefan Behnel :


--
pull_requests:  -21519

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Stefan Behnel]

Change by Stefan Behnel :


--
nosy: +scoder
nosy_count: 29.0 -> 30.0
pull_requests: +21519
pull_request: https://github.com/python/cpython/pull/22474

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Andrés Delfino]

Change by Andrés Delfino :


--
nosy: +adelfino
nosy_count: 28.0 -> 29.0
pull_requests: +21315
pull_request: https://github.com/python/cpython/pull/22205

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[mohamed koubaa]

Change by mohamed koubaa :


--
nosy: +koubaa
nosy_count: 27.0 -> 28.0
pull_requests: +21218
pull_request: https://github.com/python/cpython/pull/21986

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Jakub Kulik]

Change by Jakub Kulik :


--
pull_requests:  -21197

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Jakub Kulik]

Change by Jakub Kulik :


--
nosy:  -kulikjak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Jakub Kulik]

Change by Jakub Kulik :


--
nosy: +kulikjak
nosy_count: 27.0 -> 28.0
pull_requests: +21197
pull_request: https://github.com/python/cpython/pull/22040

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Gregory P. Smith]

Change by Gregory P. Smith :


--
pull_requests:  -21110

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Raymond Hettinger]

Change by Raymond Hettinger :


--
nosy: +rhettinger
nosy_count: 26.0 -> 27.0
pull_requests: +21110
pull_request: https://github.com/python/cpython/pull/21994

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Jesse Farnham]

Change by Jesse Farnham :


--
nosy: +jesse.farnham

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[STINNER Victor]

STINNER Victor  added the comment:

See also bpo-25920: PyOS_AfterFork should reset socketmodule's lock.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Deomid Ryabkov]

Deomid Ryabkov  added the comment:

https://bugs.python.org/issue40442 is a fresh instance of this, entirely 
self-inflicted.

--
nosy: +rojer

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Related issue:
https://bugs.python.org/issue40399
"""
IO streams locking can be broken after fork() with threads
"""

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[STINNER Victor]

STINNER Victor  added the comment:

I created bpo-40089: Add _at_fork_reinit() method to locks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Hugh Redelmeier]

Change by Hugh Redelmeier :


--
nosy: +hugh

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Gregory P. Smith]

Gregory P. Smith  added the comment:

Thanks for the debugging details!  I've filed 
https://bugs.python.org/issue36533 to specifically track this potential 
regression in the 3.7 stable branch.  lets carry on there where the discussion 
thread isn't too long for bug tracker sanity.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[A. Jesse Jiryu Davis]

Change by A. Jesse Jiryu Davis :


--
nosy:  -emptysquare

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[cagney]

cagney  added the comment:

Below is a backtrace from the deadlock.

It happens because the logging code is trying to acquire two per-logger locks; 
and in an order different to the random order used by the fork() handler.

The code in question has a custom class DebugHandler(logging.Handler).  The 
default logging.Handler.handle() method grabs its lock and calls .emit() vis:

if rv:
self.acquire()
try:
self.emit(record)
finally:
self.release()

the custom .emit() then sends the record to a sub-logger stream vis:

def emit(self, record):
for stream_handler in self.stream_handlers:
stream_handler.emit(record)
if _DEBUG_STREAM:
_DEBUG_STREAM.emit(record)

and one of these emit() functions calls flush() which tries to acquire a 
further lock.

Thread 0x7f976b7fe700 (most recent call first):
  File "/usr/lib64/python3.7/logging/__init__.py", line 854 in acquire
  File "/usr/lib64/python3.7/logging/__init__.py", line 1015 in flush

def flush(self):
"""
Flushes the stream.
"""
self.acquire() <
try:
if self.stream and hasattr(self.stream, "flush"):
self.stream.flush()
finally:
self.release()

  File "/usr/lib64/python3.7/logging/__init__.py", line 1038 in emit

self.flush() <

  File "/home/build/libreswan-web/master/testing/utils/fab/logutil.py", line 
163 in emit

def emit(self, record):
for stream_handler in self.stream_handlers:
stream_handler.emit(record) <---
if _DEBUG_STREAM:
_DEBUG_STREAM.emit(record)

  File "/usr/lib64/python3.7/logging/__init__.py", line 905 in handle

def handle(self, record):
"""
Conditionally emit the specified logging record.

Emission depends on filters which may have been added to the handler.
Wrap the actual emission of the record with acquisition/release of
the I/O thread lock. Returns whether the filter passed the record for
emission.
"""
rv = self.filter(record)
if rv:
self.acquire()
try:
self.emit(record) <---
finally:
self.release()
return rv

  File "/usr/lib64/python3.7/logging/__init__.py", line 1591 in callHandlers

hdlr.handle(record)

  File "/usr/lib64/python3.7/logging/__init__.py", line 1529 in handle

self.callHandlers(record)

  File "/usr/lib64/python3.7/logging/__init__.py", line 1519 in _log

self.handle(record)

  File "/usr/lib64/python3.7/logging/__init__.py", line 1449 in log

self._log(level, msg, args, **kwargs)

  File "/usr/lib64/python3.7/logging/__init__.py", line 1768 in log

self.logger.log(level, msg, *args, **kwargs)

  File "/usr/lib64/python3.7/logging/__init__.py", line 1724 in debug

self.log(DEBUG, msg, *args, **kwargs)

  File "/home/build/libreswan-web/master/testing/utils/fab/shell.py", line 110 
in write

self.logger.debug(self.message, ascii(text))

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[cagney]

cagney  added the comment:

> acquiring locks before fork in the thread doing the forking and releasing 
> them afterwards is always the safe thing to do.

It's also an easy way to cause a deadlock:

- register_at_fork() et.al. will cause per-logger locks to be acquired before 
the global lock (this isn't immediately obvious from the code)

If a thread were to grab its logging lock before the global lock then it would 
deadlock.  I'm guessing this isn't allowed - however I didn't see any comments 
to this effect?

Can I suggest documenting this, and also merging the two callbacks so that the 
ordering of these two acquires is made explicit.

- the per-logger locks are acquired in a random order

If a thread were to acquire two per-logger locks in a different order then 
things would deadlock.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Gregory P. Smith]

Gregory P. Smith  added the comment:

I'd start with faulthandler.register with all_threads=True and see if that 
gives you what you need.

https://docs.python.org/3/library/faulthandler.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[cagney]

cagney  added the comment:

> Does your code use any C code that forks on its own without properly calling 
> the C Python PyOS_BeforeFork(), PyOS_AfterFork_Parent(), and 
> PyOS_AfterFork_Child() APIs?

No.

Is there a web page explaining how to pull a python backtrace from all the 
threads running within a daemon?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Gregory P. Smith]

Gregory P. Smith  added the comment:

We need a small test case that can reproduce your problem.  I believe 
https://github.com/python/cpython/commit/3b699932e5ac3e76031bbb6d700fbea07492641d
 to be correct.

acquiring locks before fork in the thread doing the forking and releasing them 
afterwards is always the safe thing to do.

Example possibility: Does your code use any C code that forks on its own 
without properly calling the C Python PyOS_BeforeFork(), 
PyOS_AfterFork_Parent(), and PyOS_AfterFork_Child() APIs?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[cagney]

cagney  added the comment:

I suspect 3b699932e5ac3e7 is causing a hang in libreswan's kvmrunner.py on 
Fedora.

Looking at the Fedora RPMs:

python3-3.7.0-9.fc29.x86_64 didn't contain the fix and worked
python3-3.7.1-4.fc29.x86_64 reverted the fix (for anaconda) and worked
python3-3.7.2-4.fc29.x86_64 included the fix; eventually hangs

I believe the hang looks like:

Traceback (most recent call last):
  File "/home/build/libreswan-web/master/testing/utils/fab/runner.py", line 
389, in _process_test
test_domains = _boot_test_domains(logger, test, domain_prefix, 
boot_executor)
  File "/home/build/libreswan-web/master/testing/utils/fab/runner.py", line 
203, in _boot_test_domains
TestDomain.boot_and_login)
  File "/home/build/libreswan-web/master/testing/utils/fab/runner.py", line 
150, in submit_job_for_domain
logger.debug("scheduled %s on %s", job, domain)
  File "/usr/lib64/python3.7/logging/__init__.py", line 1724, in debug

  File "/usr/lib64/python3.7/logging/__init__.py", line 1768, in log
def __repr__(self):
  File "/usr/lib64/python3.7/logging/__init__.py", line 1449, in log
"""
  File "/usr/lib64/python3.7/logging/__init__.py", line 1519, in _log
break
  File "/usr/lib64/python3.7/logging/__init__.py", line 1529, in handle
logger hierarchy. If no handler was found, output a one-off error
  File "/usr/lib64/python3.7/logging/__init__.py", line 1591, in callHandlers

  File "/usr/lib64/python3.7/logging/__init__.py", line 905, in handle
try:
  File "/home/build/libreswan-web/master/testing/utils/fab/logutil.py", line 
163, in emit
stream_handler.emit(record)
  File "/usr/lib64/python3.7/logging/__init__.py", line 1038, in emit
Handler.__init__(self)
  File "/usr/lib64/python3.7/logging/__init__.py", line 1015, in flush
name += ' '
  File "/usr/lib64/python3.7/logging/__init__.py", line 854, in acquire
self.emit(record)
KeyboardInterrupt

--
nosy: +cagney

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[STINNER Victor]

STINNER Victor  added the comment:

> New changeset 3b699932e5ac3e76031bbb6d700fbea07492641d by Gregory P. Smith 
> (Miss Islington (bot)) in branch '3.7':
> bpo-6721: Hold logging locks across fork() (GH-4071) (#9291)

It seems like this change caused a regression in the Anaconda installer of 
Fedora:
https://bugzilla.redhat.com/show_bug.cgi?id=1644936

But we are not sure at this point. I have to investigate to understand exactly 
what is happening.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset 3b699932e5ac3e76031bbb6d700fbea07492641d by Gregory P. Smith 
(Miss Islington (bot)) in branch '3.7':
bpo-6721: Hold logging locks across fork() (GH-4071) (#9291)
https://github.com/python/cpython/commit/3b699932e5ac3e76031bbb6d700fbea07492641d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[miss-islington]

Change by miss-islington :


--
pull_requests: +8722

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset 19003841e965bbf56fd06824d6093620c1b66f9e by Gregory P. Smith in 
branch 'master':
bpo-6721: Hold logging locks across fork() (GH-4071)
https://github.com/python/cpython/commit/19003841e965bbf56fd06824d6093620c1b66f9e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Olivier Chédru]

Olivier Chédru  added the comment:

FWIW, I encountered the same kind of issue when using the mkstemp() function: 
under the hood, it calls gettempdir() and this one is protected by a lock too.

Current thread 0x7ff10231f700 (most recent call first):
   File "/usr/lib/python3.5/tempfile.py", line 432 in gettempdir
   File "/usr/lib/python3.5/tempfile.py", line 269 in _sanitize_params
   File "/usr/lib/python3.5/tempfile.py", line 474 in mkstemp

--
nosy: +ochedru

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Actually, we already have a doubly-linked list of buffered IO objects
(for another purpose), so we can reuse that and register a single set of
global callbacks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Gregory P. Smith]

Gregory P. Smith  added the comment:

logging is pretty easy to deal with so I created a PR.

bufferedio.c is a little more work as we either need to use the posixmodule.c 
os.register_at_fork API or expose it as an internal C API to be able to call it 
to add acquires and releases around the buffer's self->lock member when 
non-NULL.  either way, that needs to be written safely so that it doesn't crash 
if fork happens after a buffered io struct is freed.  (unregister at fork 
handlers when freeing it? messy)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Gregory P. Smith]

Change by Gregory P. Smith :


--
pull_requests: +4042

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Oh, I forgot that IO buffered objects also have a lock.  So we would have to 
special-case those as well, unless we take the generic approach...

A problem with the generic approach is that it would leave higher-level 
synchronization objects such as RLock, Event etc. in an inconsistent state.  
Not to mention the case where the lock is taken by the thread calling fork()...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I think we should somehow move forward on this, at least for logging locks 
which can be quite an annoyance.

There are two possible approaches:
- either a generic mechanism as posted by sbt in reinit_locks_2.diff
- or a logging-specific fix using os.register_at_fork()

What do you think?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Daniel Birnstiel]

Daniel Birnstiel added the comment:

While having to deal with this bug for a while I have written a small library 
using `pthread_atfork`: https://github.com/Birne94/python-atfork-lock-release

It allows registering atfork-hooks (similar to the ones available by now) and 
frees the stdout/stderr as well as manually provided io locks. I guess it uses 
some hacky ways to get the job done, but resolved the issue for me and has been 
working without problems for some weeks now.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Gregory P. Smith]

Gregory P. Smith added the comment:

http://bugs.python.org/issue16500 added the os.register_at_fork() API which may 
be usable for this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Gregory P. Smith]

Changes by Gregory P. Smith :


--
versions: +Python 3.7 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Daniel Birnstiel]

Daniel Birnstiel added the comment:

Currently using
Python 3.6.0 (default, Mar  4 2017, 12:32:34) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin

> So, somehow the print() statement is blocking, which I have /no/ idea how to 
> go about debugging. I assume there's a lock /in/ the print statement function 
> call, and I'm probably going to look into wrapping both the print() call and 
> the multiprocessing.Process() call  execution in a single, shared 
> multiprocessing lock, but that
> seems like a very patchwork solution to something that should just work.

I am currently having a similar issue where I get a deadlock to a stdout.flush 
call (which I assume is called when printing). Flush internally appears to 
acquire a lock which is getting stuck in the subprocess. 

This is the backtrace I was able to obtain through lldb, showing the waiting 
for a lock after the flush-call:

* thread #1: tid = 0x77c27d, 0x7fffe33c9c86 
libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
frame #0: 0x7fffe33c9c86 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x7fffe34b396a libsystem_pthread.dylib`_pthread_cond_wait + 
712
frame #2: 0x0001021ecad8 Python`PyThread_acquire_lock_timed + 256
frame #3: 0x00010221cc2f Python`_enter_buffered_busy + 169
frame #4: 0x00010221ed36 Python`_io_BufferedWriter_write + 203
frame #5: 0x00010215448b Python`_PyCFunction_FastCallDict + 529
frame #6: 0x00010211b3f0 Python`_PyObject_FastCallDict + 237
frame #7: 0x00010211be9e Python`PyObject_CallMethodObjArgs + 240
frame #8: 0x00010222171a Python`_textiowrapper_writeflush + 150
  * frame #9: 0x000104a8 Python`_io_TextIOWrapper_flush + 239


Is there any update on this issue or any solution to avoid deadlocking without 
wrapping every fork/print/logging call with a multiprocessing (or billiard in 
my case) lock?

--
nosy: +Birne94

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Camilla Montonen]

Changes by Camilla Montonen :


--
nosy: +Winterflower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Davin Potts]

Changes by Davin Potts :


--
nosy: +davin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[STINNER Victor]

STINNER Victor added the comment:

I suggest to close the issue as WONT FIX. Python code base is huge and
Python depends on a lot of external code. We cannot warranty anything.

It might be possible to track all kinds of locks with an infinite time. But
I'm not sure that it's worth it.

It is possible to use fork() with threads. The problem is more to execute
non trivial code after the fork. In short, the POSIX advices to only call
exec() syscall after fork and nothing else. The list of functions which are
safe after fork() is very short.

You can still use the multiprocessing module using the fork server for
example.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Connor Wolf]

Connor Wolf added the comment:

> Python 3.5.1+ (default, Mar 30 2016, 22:46:26)

Whatever the stock 3.5 on ubuntu 16.04 x64 is.

I've actually been running into a whole horde of really bizarre issues related 
to what I /think/ is locking in stdout. 

Basically, I have a context where I have thousands and thousands of (relatively 
short lived) `multiprocessing.Process()` processes, and over time they all get 
wedged (basically, I have ~4-32 processes alive at any time, but they all get 
recycled every few minutes).

After doing some horrible 
(https://github.com/fake-name/ReadableWebProxy/blob/master/logSetup.py#L21-L78) 
hackery in the logging module, I'm not seeing processes get wedged there, but I 
do still encounter issues with what I can only assume is a lock in the print 
statement. I'm hooking into a wedged process using 
[pystuck](https://github.com/alonho/pystuck)

durr@rwpscrape:/media/Storage/Scripts/ReadableWebProxy⟫ pystuck --port 6675
Welcome to the pystuck interactive shell.
Use the 'modules' dictionary to access remote modules (like 'os', or '__main__')
Use the `%show threads` magic to display all thread stack traces.

In [1]: show threads
<_MainThread(MainThread, started 140574012434176)>
  File "runScrape.py", line 74, in 
go()
  File "runScrape.py", line 57, in go
runner.run()
  File "/media/Storage/Scripts/ReadableWebProxy/WebMirror/Runner.py", line 453, 
in run
living = sum([manager.check_run_jobs() for manager in managers])
  File "/media/Storage/Scripts/ReadableWebProxy/WebMirror/Runner.py", line 453, 
in 
living = sum([manager.check_run_jobs() for manager in managers])
  File "/media/Storage/Scripts/ReadableWebProxy/WebMirror/Runner.py", line 364, 
in check_run_jobs
proc.start()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
  File "/usr/lib/python3.5/multiprocessing/context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/lib/python3.5/multiprocessing/context.py", line 267, in _Popen
return Popen(process_obj)
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 74, in _launch
code = process_obj._bootstrap()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
  File "/media/Storage/Scripts/ReadableWebProxy/WebMirror/Runner.py", line 145, 
in run
run.go()
  File "/media/Storage/Scripts/ReadableWebProxy/WebMirror/Runner.py", line 101, 
in go
self.log.info("RunInstance starting!")
  File "/usr/lib/python3.5/logging/__init__.py", line 1279, in info
self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.5/logging/__init__.py", line 1415, in _log
self.handle(record)
  File "/usr/lib/python3.5/logging/__init__.py", line 1425, in handle
self.callHandlers(record)
  File "/usr/lib/python3.5/logging/__init__.py", line 1487, in callHandlers
hdlr.handle(record)
  File "/usr/lib/python3.5/logging/__init__.py", line 855, in handle
self.emit(record)
  File "/media/Storage/Scripts/ReadableWebProxy/logSetup.py", line 134, in emit
print(outstr)


  File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
self._bootstrap_inner()
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.5/dist-packages/rpyc/utils/server.py", line 241, 
in start
self.accept()
  File "/usr/local/lib/python3.5/dist-packages/rpyc/utils/server.py", line 128, 
in accept
sock, addrinfo = self.listener.accept()
  File "/usr/lib/python3.5/socket.py", line 195, in accept
fd, addr = self._accept()


  File "/usr/local/lib/python3.5/dist-packages/pystuck/thread_probe.py", line 
15, in thread_frame_generator
yield (thread_, frame)


So, somehow the print() statement is blocking, which I have /no/ idea how to go 
about debugging. I assume there's a lock /in/ the print statement function 
call, and I'm probably going to look into wrapping both the print() call and 
the multiprocessing.Process() call  execution in a single, shared 
multiprocessing lock, but that
seems like a very patchwork solution to something that should just work.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Gregory P. Smith]

Gregory P. Smith added the comment:

My intent is not to block anything.  I'm Just explaining why I'm not motivated 
to spend much time on this issue myself.  Others are welcome to.

subprocess is not related to this issue, it has been fixed for use with threads 
(in 3.2 and higher) with an extremely widely used drop in replacement back-port 
for 2.7 https://pypi.python.org/pypi/subprocess32.  But even 2.7's poor 
subprocess implementation never triggered this specific issue in the first 
place (unless someone logged from a pre_exec_fn which would be a laughable 
thing to do anyways).

multiprocessing: It has spawn (as of 3.4) and forkserver methods both of which 
can help avoid this issue.  Caveats: spawn understandably has negative 
performance implications and forkserver requires the forkserver to be forked 
before threads potentially holding locks have been started.

As for the gross hacky monkey patching workaround: That was the approach I took 
in 
https://github.com/google/python-atfork/blob/master/atfork/stdlib_fixer.py#L51

Definitely a hack, but one that does work on existing interpreters.

Conner & lesha: Which Python version(s) are you using?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[lesha]

lesha added the comment:

On a moment's reflection, a lot of my prior comment is wrong. Sorry about that. 

- glog does not, that I know of, sanitize locks on fork. You just shouldn't log 
after fork but before exec.

- Using `pthread_atfork` to clean up the `logging` lock might be enough to make 
it safe from the "just forked" context, but without adding fairly exhaustive 
tests around this logic, it would be fragile with respect to further 
improvements to `logging`. So even just making this one library safe is a 
considerable amount of work.

So I retract most of my previous opinion. Sorry.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[lesha]

lesha added the comment:

I work on a multi-million-line C++ codebase that uses fork() from multithreaded 
programs all over the place. We use `glog` ubiquitously.  

This bug here that spans 6 years and has affected dozens of people 
(conservatively) simply does not exist for us. That is because glog takes the 
pragmatic approach of sanitizing its mutex on fork:

https://github.com/google/glog/blob/4d391fe692ae6b9e0105f473945c415a3ce5a401/src/base/mutex.h#L249

In my opinion, "thou shalt never fork() in a threaded program" is impractical 
purism. It is good to be aware of the dangers that lie therein, but it is 
completely possible to safely spawn **subprocesses** from multithreaded 
programs on modern OSes like Linux.

Python's subprocess **ought** to be safe to use in threaded programs. Any 
issues with this (aside from `pre_exec_fn`, obviously) are bugs in 
`subprocess`. 

Here is a C++ implementation of the concept that can be safely used in threaded 
programs:

https://github.com/facebook/folly/blob/master/folly/Subprocess.cpp

Note that unlike Python's subprocess `pre_exec_fn`, the C++ analog is very loud 
in warning you about the scary world in which your closure will execute:

https://github.com/facebook/folly/blob/master/folly/Subprocess.h#L252

The point to my message is simple: there is a pragmatic way to save hundreds of 
debugging hours for users of Python. People are going to find it necessary to 
do such things from time to time, so instead of telling them to redesign their 
programs, it is better to give them a safer tool.

Taking the glog approach in `logging` has no cost to the standard library, but 
it does have real world benefits. 

Please don't block shipping such a fix.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Connor Wolf]

Connor Wolf added the comment:

Arrrgh, s/threading/multiprocessing/g in my last message.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Connor Wolf]

Connor Wolf added the comment:

> IMNSHO, working on "fixes" for this issue while ignoring the larger 
> application design flaw elephant in the room doesn't make a lot of sense.

I understand the desire for a canonically "correct" fix, but it seems the issue 
with fixing it "correctly" has lead to the /actual/ implementation being broken 
for at least 6 years now.

As it is, my options are:
A. Rewrite the many, many libraries I use that internally spawn threads.
B. Not use multiprocessing.

(A) is prohibitive from a time perspective (I don't even know how many 
libraries I'd have to rewrite!), and (B) means I'd get 1/24-th of my VMs 
performance, so it's somewhat prohibitive.

At the moment, I've thrown together a horrible, horrible fix where I reach into 
the logging library (which is where I'm seeing deadlocks), and manually iterate 
over all attached log managers, resetting the locks in each immediately when 
each process spawns. 
This is, I think it can be agreed, a horrible, horrible hack, but in my 
particular case it works (the worst case result is garbled console output for a 
line or two). 

---

If a canonical fix is not possible, at least add a facility to the threading 
fork() call that lets the user decide what to do. In my case, my program is 
wedging in the logging system, and I am entirely OK with having transiently 
garbled logs, if it means I don't wind up deadlocking and having to force kill 
the interpreter (which is, I think, far /more/ destructive an action).

If I could basically do `multiprocessing.Process(*args, *kwargs, 
_clear_locks=True)`, that would be entirely sufficient, and not change existing 
behaviour at all.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Gregory P. Smith]

Gregory P. Smith added the comment:

For me the momentum on fixing these things has stalled because I no longer work 
on any code that runs into this.  There is a fundamental problem: You cannot 
safely use threading and os.fork() in the same application per POSIX rules.  So 
even if the standard library and interpreter to tried to force its locks into 
some sort of consistent state post os.fork(), the much more fundamental POSIX 
problem remains.

IMNSHO, working on "fixes" for this issue while ignoring the larger application 
design flaw elephant in the room doesn't make a lot of sense.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Connor Wolf]

Connor Wolf added the comment:

Is anything happening with these fixes? This is still an issue (I'm running 
into it now)?

--
nosy: +Connor.Wolf

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[A. Jesse Jiryu Davis]

Changes by A. Jesse Jiryu Davis :


--
nosy: +emptysquare

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Nir Soffer]

Changes by Nir Soffer :


--
nosy: +nirs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Maries Ionel Cristian]

Changes by Maries Ionel Cristian :


--
nosy: +ionel.mc

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Dan O'Reilly]

Changes by Dan O'Reilly :


--
nosy: +dan.oreilly

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Tshepang Lekhonkhobe]

Changes by Tshepang Lekhonkhobe :


--
nosy: +tshepang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Tshepang Lekhonkhobe]

Changes by Tshepang Lekhonkhobe :


--
title: Locks in python standard library should be sanitized on fork -> Locks in 
the standard library should be sanitized on fork
versions: +Python 3.5 -Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com