[issue6721] Locks in the standard library should be sanitized on fork
Daniel Birnstiel added the comment: While having to deal with this bug for a while I have written a small library using `pthread_atfork`: https://github.com/Birne94/python-atfork-lock-release It allows registering atfork-hooks (similar to the ones available by now) and frees the stdout/stderr as well as manually provided io locks. I guess it uses some hacky ways to get the job done, but resolved the issue for me and has been working without problems for some weeks now. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue6721> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30385] Segfault on OSX with 3.6.1
Daniel Birnstiel added the comment: I am currently (occasionally) running into the same problem, using Python 3.6.0 on OSX 10.12.3. I cannot reproduce this segmentation fault reliably, it happens after running for a while in production. This appears to be similar to issue 13829. python traceback (trimmed): Fatal Python error: Segmentation fault Current thread 0x765fb000 (most recent call first): File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 2592 in proxy_bypass_macosx_sysconf File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 2616 in proxy_bypass File "/Users/daniel/Workspace/venv/lib/python3.6/site-packages/requests/utils.py", line 677 in should_bypass_proxies File "/Users/daniel/Workspace/venv/lib/python3.6/site-packages/requests/utils.py", line 693 in get_environ_proxies File "/Users/daniel/Workspace/venv/lib/python3.6/site-packages/requests/sessions.py", line 686 in merge_environment_settings File "/Users/daniel/Workspace/venv/lib/python3.6/site-packages/requests/sessions.py", line 509 in request File "/Users/daniel/Workspace/venv/lib/python3.6/site-packages/requests/api.py", line 58 in request crash dump traceback (trimmed): * thread #1: tid = 0x, 0x7fffc1ef0dd6 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP * frame #0: 0x7fffc1ef0dd6 libsystem_kernel.dylib`__pthread_kill + 10 frame #1: 0x7fffc1fdc787 libsystem_pthread.dylib`pthread_kill + 90 frame #2: 0x7fffc1e07497 libsystem_c.dylib`raise + 26 frame #3: 0x7fffc1fcfbba libsystem_platform.dylib`_sigtramp + 26 frame #4: 0x7fffc1da188f libdispatch.dylib`_dispatch_queue_push + 172 frame #5: 0x7fffc1d91572 libdispatch.dylib`_dispatch_mach_msg_send + 657 frame #6: 0x7fffc1d924ba libdispatch.dylib`_dispatch_mach_send_drain + 280 frame #7: 0x7fffc1da91ea libdispatch.dylib`_dispatch_mach_send_push_and_trydrain + 487 frame #8: 0x7fffc1da6745 libdispatch.dylib`_dispatch_mach_send_msg + 282 frame #9: 0x7fffc1da6804 libdispatch.dylib`dispatch_mach_send_with_result + 50 frame #10: 0x7fffc201f25e libxpc.dylib`_xpc_connection_enqueue + 104 frame #11: 0x7fffc201eca3 libxpc.dylib`xpc_connection_send_message_with_reply + 152 frame #12: 0x7fffac99b8e5 CoreFoundation`__80-[CFPrefsSearchListSource alreadylocked_generationCountFromListOfSources:count:]_block_invoke_2 + 133 frame #13: 0x7fffac9cb4e4 CoreFoundation`-[_CFXPreferences withConnectionForRole:performBlock:] + 36 frame #14: 0x7fffac99b817 CoreFoundation`__80-[CFPrefsSearchListSource alreadylocked_generationCountFromListOfSources:count:]_block_invoke + 199 frame #15: 0x7fffac99b662 CoreFoundation`-[CFPrefsSearchListSource alreadylocked_generationCountFromListOfSources:count:] + 226 frame #16: 0x7fffac81ee80 CoreFoundation`-[CFPrefsSearchListSource alreadylocked_copyDictionary] + 336 frame #17: 0x7fffac81eb5c CoreFoundation`-[CFPrefsSearchListSource alreadylocked_copyValueForKey:] + 60 frame #18: 0x7fffac94dd75 CoreFoundation`-[CFPrefsSource copyValueForKey:] + 53 frame #19: 0x7fffac9c9b00 CoreFoundation`__76-[_CFXPreferences copyAppValueForKey:identifier:container:configurationURL:]_block_invoke + 32 frame #20: 0x7fffac99ca32 CoreFoundation`__108-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:]_block_invoke + 290 frame #21: 0x7fffac99c8a9 CoreFoundation`-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:] + 345 frame #22: 0x7fffac9c9a66 CoreFoundation`-[_CFXPreferences copyAppValueForKey:identifier:container:configurationURL:] + 310 frame #23: 0x7fffb343a56f SystemConfiguration`SCDynamicStoreCopyProxiesWithOptions + 164 frame #24: 0x00010b36f926 _scproxy.cpython-36m-darwin.so`get_proxy_settings + 24 frame #25: 0x00010942148b Python`_PyCFunction_FastCallDict + 529 frame #26: 0x000109488959 Python`call_function + 562 frame #27: 0x000109485d10 Python`_PyEval_EvalFrameDefault + 26835 -- nosy: +Birne94 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30385> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29859] Return code of pthread_* in thread_pthread.h is not used for perror
Changes by Daniel Birnstiel <dan...@birne.me>: -- resolution: -> fixed stage: -> resolved status: open -> closed ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29859> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29859] Return code of pthread_* in thread_pthread.h is not used for perror
Daniel Birnstiel added the comment: While I agree, fprintf it not really nice, I looked through other parts of the python source where information is printed to stderr and fprintf was used there as well, so I fell back to it myself. % grep -rnw . -e "fprintf(stderr," | wc -l 178 Using threading and multiprocessing is insane, I know that. Nevertheless the error codes returned by the pthread_* calls are processed incorrectly, so I would consider this a bug worth fixing. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29859> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29859] Return code of pthread_* in thread_pthread.h is not used for perror
Daniel Birnstiel added the comment: While you might scold me for the following code, it is just some fiddling with the cpython functions from python side. Backstory is issue 6721 or related to it. I am working on a multi-process service which uses multiprocessing (or rather the billiard fork). While forking I run into deadlocks as described in the issue when I fork while the io lock is acquired. In order to find a solution I experimented with the cpython module and tried to release the lock manually after a fork. The code can be found attached (tested with python 3.6/3.7 on OSX Sierra). While it does not really do what it is supposed to do, it shows the problem described in this issue (#29859): If the internal calls to the pthread_* functions fail, I will only get a generic error message: pthread_mutex_lock[3]: Undefined error: 0 pthread_cond_signal: Undefined error: 0 pthread_mutex_unlock[3]: Undefined error: 0 In reality the produced error messages should be: pthread_mutex_lock[3]: Invalid argument pthread_cond_signal: Invalid argument pthread_mutex_unlock[3]: Invalid argument This happens because the pthread_* functions do not set the erno variable as described in the issue description. Please note that the issue is not only related to my code, but might affect all code using locks. While I suppose there shouldn't be any errors when calling the low level functions, the attached patch/pr will make debugging a lot easier when it actually happens by displaying the correct error message. -- Added file: http://bugs.python.org/file46749/test.py ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29859> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29859] Return code of pthread_* in thread_pthread.h is not used for perror
Changes by Daniel Birnstiel <dan...@birne.me>: -- pull_requests: +654 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29859> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29859] Return code of pthread_* in thread_pthread.h is not used for perror
Daniel Birnstiel added the comment: I have attached a diff adding a new macro for handling pthread_* status codes. Will submit PR as soon as my CLA is approved. -- keywords: +patch versions: +Python 3.7 Added file: http://bugs.python.org/file46746/patch.diff ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29859> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29859] Return code of pthread_* in thread_pthread.h is not used for perror
New submission from Daniel Birnstiel: Python/thread_pthread.h:145 defines the CHECK_STATUS macro used for printing error messages in case any of the calls fail. CHECK_STATUS uses perror for formatting an error message, which relies on the global erno being set (see man perror). Since the pthread functions return their status code instead of setting erno (which might not even work in threaded environments), no additional information is displayed. See for example produced by PyThread_release_lock: pthread_mutex_lock[3]: Undefined error: 0 pthread_cond_signal: Undefined error: 0 pthread_mutex_unlock[3]: Undefined error: 0 The correct solution would be to use strerror(status) in order to show the proper message. -- components: Interpreter Core messages: 289884 nosy: Birne94 priority: normal severity: normal status: open title: Return code of pthread_* in thread_pthread.h is not used for perror type: behavior versions: Python 3.6 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29859> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in the standard library should be sanitized on fork
Daniel Birnstiel added the comment: Currently using Python 3.6.0 (default, Mar 4 2017, 12:32:34) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin > So, somehow the print() statement is blocking, which I have /no/ idea how to > go about debugging. I assume there's a lock /in/ the print statement function > call, and I'm probably going to look into wrapping both the print() call and > the multiprocessing.Process() call execution in a single, shared > multiprocessing lock, but that > seems like a very patchwork solution to something that should just work. I am currently having a similar issue where I get a deadlock to a stdout.flush call (which I assume is called when printing). Flush internally appears to acquire a lock which is getting stuck in the subprocess. This is the backtrace I was able to obtain through lldb, showing the waiting for a lock after the flush-call: * thread #1: tid = 0x77c27d, 0x7fffe33c9c86 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP frame #0: 0x7fffe33c9c86 libsystem_kernel.dylib`__psynch_cvwait + 10 frame #1: 0x7fffe34b396a libsystem_pthread.dylib`_pthread_cond_wait + 712 frame #2: 0x0001021ecad8 Python`PyThread_acquire_lock_timed + 256 frame #3: 0x00010221cc2f Python`_enter_buffered_busy + 169 frame #4: 0x00010221ed36 Python`_io_BufferedWriter_write + 203 frame #5: 0x00010215448b Python`_PyCFunction_FastCallDict + 529 frame #6: 0x00010211b3f0 Python`_PyObject_FastCallDict + 237 frame #7: 0x00010211be9e Python`PyObject_CallMethodObjArgs + 240 frame #8: 0x00010222171a Python`_textiowrapper_writeflush + 150 * frame #9: 0x000104a8 Python`_io_TextIOWrapper_flush + 239 Is there any update on this issue or any solution to avoid deadlocking without wrapping every fork/print/logging call with a multiprocessing (or billiard in my case) lock? -- nosy: +Birne94 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue6721> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
