[issue38707] Multiprocessing: bug with Native ID for threading.mainthread()
Jake Tesler added the comment: PR was updated with tests and is ready for core developer review and then the merge to cpython:master. After that (if I understand correctly) a backport will automatically get picked into the 3.8 branch if there aren't any conflicts. -- ___ Python tracker <https://bugs.python.org/issue38707> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38707] Multiprocessing: bug with Native ID for threading.mainthread()
Jake Tesler added the comment: @vstinner PR created :) https://github.com/python/cpython/pull/17088 -- ___ Python tracker <https://bugs.python.org/issue38707> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38707] Multiprocessing: bug with Native ID for threading.mainthread()
Change by Jake Tesler : -- keywords: +patch pull_requests: +16596 stage: -> patch review pull_request: https://github.com/python/cpython/pull/17088 ___ Python tracker <https://bugs.python.org/issue38707> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38707] Multiprocessing: bug with Native ID for threading.mainthread()
Change by Jake Tesler : -- nosy: +pitrou ___ Python tracker <https://bugs.python.org/issue38707> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38707] Multiprocessing: bug with Native ID for threading.mainthread()
New submission from Jake Tesler :
I have encountered a minor bug with the new `threading.get_native_id()`
featureset in Python 3.8. The bug occurs when creating a new
multiprocessing.Process object on Unix (or on any platform where the
multiprocessing start_method is 'fork' or 'forkserver').
When creating a new process via fork, the Native ID in the new MainThread is
incorrect. The new forked process' threading.MainThread object inherits the
Native ID from the parent process' MainThread instead of capturing/updating its
own (new) Native ID.
See the following snippet:
>>> import threading, multiprocessing
>>> multiprocessing.set_start_method('fork') # or 'forkserver'
>>> def proc(): print(threading.get_native_id(),
>>> threading.main_thread().native_id) # get_native_id(), mainthread.native_id
>>> proc()
22605 22605 # get_native_id(), mainthread.native_id
>>> p = multiprocessing.Process(target=proc)
>>> p.start()
22648 22605 # get_native_id(), mainthread.native_id
>>>
>>> def update(): threading.main_thread()._set_native_id()
>>> def print_and_update(): proc(); update(); proc()
>>> print_and_update()
22605 22605 # get_native_id(), mainthread.native_id
22605 22605
>>> p2=multiprocessing.Process(target=print_and_update); p2.start()
22724 22605 # get_native_id(), mainthread.native_id
22724 22724
>>> print_and_update()
22605 22605 # get_native_id(), mainthread.native_id
22605 22605
As you can see, the new Process object's MainThread.native_id attribute matches
that of the MainThread of its parent process.
Unfortunately, I'm not too familiar with the underlying mechanisms that
Multiprocessing uses to create forked processes.
I believe this behavior occurs because (AFAIK) a forked multiprocessing.Process
copies the MainThread object from its parent process, rather than
reinitializing a new one. Looking further into the multiprocessing code, it
appears the right spot to fix this would be in the
multiprocessing.Process.bootstrap() function.
I've created a branch containing a working fix - I'm also open to suggestions
of how a fix might otherwise be implemented.
If it looks correct I'll create a PR against the CPython 3.8 branch.
See the branch here: https://github.com/jaketesler/cpython/tree/fix-mp-native-id
Thanks all!
-Jake
--
components: Library (Lib)
messages: 356070
nosy: jaketesler, vstinner
priority: normal
severity: normal
status: open
title: Multiprocessing: bug with Native ID for threading.mainthread()
type: behavior
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue38707>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36084] Threading: add builtin TID attribute to Thread objects
Change by Jake Tesler : -- status: open -> closed ___ Python tracker <https://bugs.python.org/issue36084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36084] Threading: add builtin TID attribute to Thread objects
Jake Tesler added the comment:
I have encountered a minor bug with this new feature. The bug occurs when
creating a new multiprocessing.Process object on Unix (or on any platform where
the multiprocessing start_method is 'fork' or 'forkserver').
When creating a new process via fork, the Native ID in the new MainThread is
incorrect. The new forked process' threading.MainThread object inherits the
Native ID from the parent process' MainThread instead of capturing/updating its
own (new) Native ID.
See the following snippet:
>>> import threading, multiprocessing
>>> multiprocessing.set_start_method('fork') # or 'forkserver'
>>> def proc(): print(threading.get_native_id(),
>>> threading.main_thread().native_id) # get_native_id(), mainthread.native_id
>>> proc()
22605 22605 # get_native_id(), mainthread.native_id
>>> p = multiprocessing.Process(target=proc)
>>> p.start()
22648 22605 # get_native_id(), mainthread.native_id
>>>
>>> def update(): threading.main_thread()._set_native_id()
>>> def print_and_update(): proc(); update(); proc()
>>> print_and_update()
22605 22605 # get_native_id(), mainthread.native_id
22605 22605
>>> p2=multiprocessing.Process(target=print_and_update); p2.start()
22724 22605 # get_native_id(), mainthread.native_id
22724 22724
>>> print_and_update()
22605 22605 # get_native_id(), mainthread.native_id
22605 22605
As you can see, the new Process object's MainThread.native_id attribute matches
that of the MainThread of its parent process.
Unfortunately, I'm not too familiar with the underlying mechanisms that
Multiprocessing uses to create forked processes.
I believe this behavior occurs because (AFAIK) a forked multiprocessing.Process
copies the MainThread object from its parent process, rather than
reinitializing a new one. Looking further into the multiprocessing code, it
appears the right spot to fix this would be in the
multiprocessing.Process.bootstrap() function.
I've created a branch containing a working fix - I'm also open to suggestions
of how a fix might otherwise be implemented.
If it looks correct I'll create a PR against the CPython 3.8 branch.
See the branch here: https://github.com/jaketesler/cpython/tree/fix-mp-native-id
Thanks all!
-Jake
--
status: closed -> open
___
Python tracker
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36084] Threading: add builtin TID attribute to Thread objects
Change by Jake Tesler : -- pull_requests: +14636 pull_request: https://github.com/python/cpython/pull/14845 ___ Python tracker <https://bugs.python.org/issue36084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36084] Threading: add builtin TID attribute to Thread objects
Jake Tesler added the comment: Michael Felt - If you would like some help with adding/building AIX support for this functionality, tag me, I'd be glad to help out! :) -- ___ Python tracker <https://bugs.python.org/issue36084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36084] Threading: add builtin TID attribute to Thread objects
Jake Tesler added the comment: Victor – the return value of _start_new_thread is the the `ident` parameter, and its not the same as the native id. See here: https://github.com/python/cpython/pull/11993#issuecomment-491544908 -- ___ Python tracker <https://bugs.python.org/issue36084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36084] Threading: add builtin TID attribute to Thread objects
Jake Tesler added the comment: I will look into whether adding thread_self() for AIX would be simple enough for this PR. -- ___ Python tracker <https://bugs.python.org/issue36084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36084] Threading: add builtin TID attribute to Thread objects
Jake Tesler added the comment: In general, I’ve concluded most ‘editions’ of pthread_self() are not the same value as this feature aims to implement. I’m not familiar enough with AIX to be certain about that platform, though. If there’s an equivalent function in AIX to capture the actual integral thread ID in the same 3-line way as Linux, FreeBSD, et al., are implemented (the big block in PyThread_get_thread_native_id), then it’s worth including. If the mechanism required is a more complex one than just adding a few lines, then it might be best left for a new PR. :) -- ___ Python tracker <https://bugs.python.org/issue36084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36084] Threading: add builtin TID attribute to Thread objects
Jake Tesler added the comment: Victor Stinner: would you mind taking a look at the new PR? Is this more along the lines of what you had in mind? -- ___ Python tracker <https://bugs.python.org/issue36084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36084] Threading: add builtin TID attribute to Thread objects
Jake Tesler added the comment: New PR created with requested edits. :) -- ___ Python tracker <https://bugs.python.org/issue36084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36084] Threading: add builtin TID attribute to Thread objects
Change by Jake Tesler : -- pull_requests: +13373 ___ Python tracker <https://bugs.python.org/issue36084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36084] Threading: add builtin TID attribute to Thread objects
Jake Tesler added the comment: I will implement these changes - let’s try to hit that end-of-month target! -- ___ Python tracker <https://bugs.python.org/issue36084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36084] Threading: add builtin TID attribute to Thread objects
Jake Tesler added the comment: So where do we go from here? -- ___ Python tracker <https://bugs.python.org/issue36084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36084] Threading: add builtin TID attribute to Thread objects
Jake Tesler added the comment: The feature is supported on Windows: the file supporting Windows threading is `thread_nt.h`, not `thread_pthread.h` since Windows doesn't use POSIX-style threads. Also it is different from threading.get_ident() - ident is a Python-issued unique identifier, TID is issued by the OS and is tracable system-wide. -- ___ Python tracker <https://bugs.python.org/issue36084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36084] Threading: add builtin TID attribute to Thread objects
Jake Tesler added the comment: *bump* Could someone look into reviewing this bug/PR? Thanks! -- ___ Python tracker <https://bugs.python.org/issue36084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36084] Threading: add builtin TID attribute to Thread objects
New submission from Jake Tesler : This functionality adds a native Thread ID to threading.Thread objects. This ID (TID), similar to the PID of a process, is assigned by the OS (kernel) and is generally used for externally monitoring resources consumed by the running thread (or process). This does not replace the `ident` attribute within Thread objects, which is assigned by the Python interpreter and is guaranteed as unique for the lifetime of the Python instance. -- messages: 336348 nosy: Jake Tesler priority: normal severity: normal status: open title: Threading: add builtin TID attribute to Thread objects type: enhancement versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue36084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
