[issue44751] crypt.h should be in _cryptmodule.c, not in public header
STINNER Victor added the comment: New changeset 80285ecc8deaa2b0e7351bf4be863d1a0ad3c188 by Miss Islington (bot) in branch '3.10': closes bpo-44751: Move crypt.h include from public header to _cryptmodule (GH-27394) (GH-28636) https://github.com/python/cpython/commit/80285ecc8deaa2b0e7351bf4be863d1a0ad3c188 -- nosy: +vstinner ___ Python tracker <https://bugs.python.org/issue44751> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43753] [C API] Add Py_Is(x, y) and Py_IsNone(x) functions
Change by STINNER Victor : -- pull_requests: +27009 pull_request: https://github.com/python/cpython/pull/28641 ___ Python tracker <https://bugs.python.org/issue43753> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25653] ctypes+callbacks+fork+selinux = crash
STINNER Victor added the comment: Another recent crash involving libffi, closure, fork and SELinux: https://bugzilla.redhat.com/show_bug.cgi?id=1977410 This bug comes from libffi, not from Python (but it can be easily reproducing using ctypes which uses libffi). -- nosy: +vstinner ___ Python tracker <https://bugs.python.org/issue25653> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45262] crash if asyncio is used before and after re-initialization if using python embedded in an application
STINNER Victor added the comment: > "Assertion failed: Py_IS_TYPE(rl, _Type), file > D:\a\1\s\Modules_asynciomodule.c, line 261" The _asyncio extension uses static types and doesn't implement the multiphase initialization API. It doesn't work well with your "deinitialize/initialize the interpreter" use case. The _asyncio extension should use heap types and the multiphase init API. -- nosy: +vstinner ___ Python tracker <https://bugs.python.org/issue45262> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45325] Allow "p" in Py_BuildValue
STINNER Victor added the comment: > but there is a catch -- the arguments should be a C int. If it is not, you > can break a stack. I never understood how "..." arguments work under the hood. What happens if you pass a double, it is stored as a double on the C stack, and then Py_BuildValue() will read junk data? -- ___ Python tracker <https://bugs.python.org/issue45325> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45325] Allow "p" in Py_BuildValue
STINNER Victor added the comment: I would be interested to see how the "p" format could be used in existing code. I found a few places would benefit of it. Either create a new draft PR, or put it in the same PR 28634. Modules/_itertoolsmodule.c: return Py_BuildValue("O(N)(OO)", Py_TYPE(lz), it, lz->saved, Py_True); return Py_BuildValue("O(O)(OO)", Py_TYPE(lz), lz->it, lz->saved, lz->firstpass ? Py_True : Py_False); Modules/_ssl.c: ok = RAND_bytes((unsigned char*)PyBytes_AS_STRING(bytes), len); if (ok == 0 || ok == 1) return Py_BuildValue("NO", bytes, ok == 1 ? Py_True : Py_False); return Py_BuildValue( "{sksssisi" "sO" "}", "id", cipher_id, "name", cipher_name, "protocol", cipher_protocol, "description", buf, "strength_bits", strength_bits, "alg_bits", alg_bits ,"aead", aead ? Py_True : Py_False, "symmetric", skcipher, "digest", digest, "kea", kx, "auth", auth ); Modules/main.c: runargs = PyTuple_Pack(2, module, set_argv0 ? Py_True : Py_False); Objects/fileobject.c: stream = _PyObject_CallMethodId(io, _open, "isisssO", fd, mode, buffering, encoding, errors, newline, closefd ? Py_True : Py_False); -- nosy: +vstinner ___ Python tracker <https://bugs.python.org/issue45325> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36207] robotsparser deny all with some rules
Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg402889 ___ Python tracker <https://bugs.python.org/issue36207> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41710] acquire(timeout) of threading.Lock and threading.Condition is affected by jumps in system time: Python should use sem_clockwait(CLOCK_MONOTONIC)
Change by STINNER Victor : -- keywords: +patch pull_requests: +26999 stage: -> patch review pull_request: https://github.com/python/cpython/pull/28629 ___ Python tracker <https://bugs.python.org/issue41710> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45322] [X86_64]test_concurrent_futures fail
STINNER Victor added the comment: > File > "/home/abuild/rpmbuild/BUILD/Python-3.7.9/Lib/multiprocessing/queues.py", > line 224 in _feed Python 3.7 no longer accept bugfixes: I close the issue. You can open a new issue if you can reproduce the issue on Python 3.9 or newer: https://devguide.python.org/#status-of-python-branches -- resolution: -> wont fix stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue45322> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45319] Possible regression in __annotations__ descr for heap type subclasses
Change by STINNER Victor : -- nosy: +petr.viktorin, vstinner ___ Python tracker <https://bugs.python.org/issue45319> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39026] Include/cpython/pystate.h contains non-relative of initconfig.h include causing macOS Framework include failure
STINNER Victor added the comment: I propose to only rename header files in Python 3.11. For Python 3.9 and 3.10, I suggest to modify Include/cpython/pystate.h by replacing: #include "cpython/initconfig.h" with: #include "initconfig.h" Like the PR 20181. -- ___ Python tracker <https://bugs.python.org/issue39026> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39026] Include/cpython/pystate.h contains non-relative of initconfig.h include causing macOS Framework include failure
STINNER Victor added the comment: I'm not sure why Include/cpython/ is part of the compiler include directories. I am not sure why Include/ is *not* part of the compiler include directory. Anyway, it's hard to control how people build Python and there are always legit use cases where you fall into such issue. That's why I suggest to fix any risk of confusion by renaming Include/cpython/ header files to add a "cpython_" prefix. For example, rename Include/cpython/pystate.h to Include/cpython/cpython_pystate.h. In the Include/internal/ directory, header files already have a prefix for the exact same reason. For example, there is a pycore_pystate.h file there. -- ___ Python tracker <https://bugs.python.org/issue39026> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45308] multiprocessing default fork child process will not free object, which is inherited from parent process
Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker <https://bugs.python.org/issue45308> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39026] Include/cpython/pystate.h contains non-relative of initconfig.h include causing macOS Framework include failure
Change by STINNER Victor : -- title: pystate.h contains non-relative of initconfig.h include causing macOS Framework include failure -> Include/cpython/pystate.h contains non-relative of initconfig.h include causing macOS Framework include failure ___ Python tracker <https://bugs.python.org/issue39026> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39026] pystate.h contains non-relative of initconfig.h include causing macOS Framework include failure
Change by STINNER Victor : -- pull_requests: +26982 pull_request: https://github.com/python/cpython/pull/28612 ___ Python tracker <https://bugs.python.org/issue39026> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39026] pystate.h contains non-relative of initconfig.h include causing macOS Framework include failure
STINNER Victor added the comment: I marked bpo-40642 as a duplicate of this issue. -- ___ Python tracker <https://bugs.python.org/issue39026> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40642] Cpython "pystate.h" subdirectory wrong
STINNER Victor added the comment: I mark this issue as a duplicate of bpo-39026. -- resolution: -> duplicate stage: patch review -> resolved status: open -> closed superseder: -> pystate.h contains non-relative of initconfig.h include causing macOS Framework include failure ___ Python tracker <https://bugs.python.org/issue40642> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40642] Cpython "pystate.h" subdirectory wrong
STINNER Victor added the comment: Having two header files with the same name in two directories which are part of the include directories is causing issues like this one :-( For the internal C API, I solve the issue by adding "pycore_" prefix to filenames. Maybe pystate.h should be renamed to cpython_pystate.h to avoid to fix the issue. It would include cpython_initconfig.h. -- nosy: +vstinner ___ Python tracker <https://bugs.python.org/issue40642> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45201] API function PySignal_SetWakeupFd is not exported and unusable
STINNER Victor added the comment: See also bpo-45316: [C API] Functions not exported with PyAPI_FUNC(). -- nosy: +vstinner ___ Python tracker <https://bugs.python.org/issue45201> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45316] [C API] Functions not exported with PyAPI_FUNC()
New submission from STINNER Victor : The Python C API contains multiple functions which are not exported with PyAPI_FUNC() and so are not usable. Public functions: --- void PyLineTable_InitAddressRange(const char *linetable, Py_ssize_t length, int firstlineno, PyCodeAddressRange *range); int PyLineTable_NextAddressRange(PyCodeAddressRange *range); int PyLineTable_PreviousAddressRange(PyCodeAddressRange *range); --- => Either make this functions private ("_Py" prefix) and move them to the internal C API, or add PyAPI_FUNC(). Private functions: --- int _PyCode_InitAddressRange(PyCodeObject* co, PyCodeAddressRange *bounds); int _PyCode_InitEndAddressRange(PyCodeObject* co, PyCodeAddressRange* bounds); PyDictKeysObject *_PyDict_NewKeysForClass(void); Py_ssize_t _PyDict_KeysSize(PyDictKeysObject *keys); uint32_t _PyDictKeys_GetVersionForCurrentState(PyDictKeysObject *dictkeys); Py_ssize_t _PyDictKeys_StringLookup(PyDictKeysObject* dictkeys, PyObject *key); PyObject *_PyDict_Pop_KnownHash(PyObject *, PyObject *, Py_hash_t, PyObject *); PyObject *_PyDict_FromKeys(PyObject *, PyObject *, PyObject *); int _PyObjectDict_SetItem(PyTypeObject *tp, PyObject **dictptr, PyObject *name, PyObject *value); PyObject *_PyDict_LoadGlobal(PyDictObject *, PyDictObject *, PyObject *); Py_ssize_t _PyDict_GetItemHint(PyDictObject *, PyObject *, Py_ssize_t, PyObject **); PyFrameObject* _PyFrame_New_NoTrack(struct _interpreter_frame *, int); int PySignal_SetWakeupFd(int fd); uint32_t _PyFunction_GetVersionForCurrentState(PyFunctionObject *func); PyObject *_PyGen_yf(PyGenObject *); PyObject *_PyCoro_GetAwaitableIter(PyObject *o); PyObject *_PyAsyncGenValueWrapperNew(PyObject *); void _PyArg_Fini(void); int _Py_CheckPython3(void); --- => I suggest to move all these declarations to the internal C API. Moreover, Include/moduleobject.h contains: --- #ifdef Py_BUILD_CORE extern int _PyModule_IsExtension(PyObject *obj); #endif --- IMO this declaration should be moved to the internal C API. See also bpo-45201 about PySignal_SetWakeupFd(). -- components: C API messages: 402827 nosy: vstinner priority: normal severity: normal status: open title: [C API] Functions not exported with PyAPI_FUNC() versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue45316> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36819] Crash during encoding using UTF-16/32 and custom error handler
STINNER Victor added the comment: > We could just forbid error handlers returning position not in the range > (start , end] Yeah, that sounds like a reasonable solution. I don't see the point of returning a position outside this range. What would be the use case? For me, the only corner case is the "ignore" error handler which returns an empty string, but it returns a position in this range, no? > it can break some code, so it is better to do this only in a new release. Implementing custom error handlers is a rare use case, so it should only affect a minority of users. Moreover, IMO returning a position outside the valid range is a bug. It's common that security fixes change the behavior, like rejecting values which were previously acceptd, to prevent a Python crash. -- ___ Python tracker <https://bugs.python.org/issue36819> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45308] multiprocessing default fork child process will not free object, which is inherited from parent process
STINNER Victor added the comment: > I still have some questions about is The Python bug tracker is not a forum for asking help how to use Python. Please stop reopening this issue. -- resolution: -> not a bug status: open -> closed ___ Python tracker <https://bugs.python.org/issue45308> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1170] shlex have problems with parsing unicode
STINNER Victor added the comment: > I would like to suggest making this change (or something similar) to the > official version of "shlex.py". Would sending an email to > "python-id...@python.org" be a good place to make this proposal? Yes, python-ideas is a good place to start discussion such idea. This issue is closed, if you discuss it here, you will get a limited audience. -- ___ Python tracker <https://bugs.python.org/issue1170> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42972] [C API] Heap types (PyType_FromSpec) must fully implement the GC protocol
Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg394832 ___ Python tracker <https://bugs.python.org/issue42972> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45307] Removal of _PyImport_FindExtensionObject() in 3.10 limits custom extension module loaders
STINNER Victor added the comment: > Until 3.10 makes equivalent functionality available or another workaround is > supported, I don't understand which long term solution do you propose. > a properly designed public API is probably a better solution Can you propose a public API? -- ___ Python tracker <https://bugs.python.org/issue45307> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45310] test_multiprocessing_forkserver: test_shared_memory_basics() failed with FileExistsError: [Errno 17] File exists: '/test01_tsmb'
New submission from STINNER Victor : AMD64 Fedora Stable LTO + PGO 3.10 (build 357): https://buildbot.python.org/all/#/builders/651/builds/357 First test_multiprocessing_forkserver failed, then test_multiprocessing_spawn, then test_multiprocessing_fork. I confirm that these tests fail if the /test01_tsmb shared memory exists. The test is supposed to unlink the shared memory once the test completes: def test_shared_memory_basics(self): sms = shared_memory.SharedMemory('test01_tsmb', create=True, size=512) self.addCleanup(sms.unlink) Maybe the whole process was killed while the test was running. I removed it manually: sudo ./python -c "import _posixshmem; _posixshmem.shm_unlink('/test01_tsmb')" Logs: Traceback (most recent call last): File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/resource_tracker.py", line 209, in main cache[rtype].remove(name) KeyError: '/psm_49a93592' (...) == ERROR: test_shared_memory_basics (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory) -- Traceback (most recent call last): File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/test/_test_multiprocessing.py", line 3777, in test_shared_memory_basics sms = shared_memory.SharedMemory('test01_tsmb', create=True, size=512) File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/shared_memory.py", line 103, in __init__ self._fd = _posixshmem.shm_open( FileExistsError: [Errno 17] File exists: '/test01_tsmb' -- components: Tests messages: 402770 nosy: vstinner priority: normal severity: normal status: open title: test_multiprocessing_forkserver: test_shared_memory_basics() failed with FileExistsError: [Errno 17] File exists: '/test01_tsmb' versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue45310> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45308] multiprocessing default fork child process will not free object, which is inherited from parent process
STINNER Victor added the comment: That's the principle of fork. You should use a different spawn method if you don't want to inherit resources inherited from the parent process, like spawn or forkserver: https://docs.python.org/dev/library/multiprocessing.html#contexts-and-start-methods -- nosy: +vstinner resolution: -> not a bug stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue45308> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1596321] KeyError at exit after 'import threading' in other thread
STINNER Victor added the comment: Better late than never. I only took 15 years to fix this old bug :-D -- versions: +Python 3.10, Python 3.11, Python 3.9 -Python 2.7, Python 3.2 ___ Python tracker <https://bugs.python.org/issue1596321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1596321] KeyError at exit after 'import threading' in other thread
STINNER Victor added the comment: New changeset 94d19f606fa18a1c4d2faca1caf2f470a8ce6d46 by Victor Stinner in branch '3.9': bpo-1596321: Fix threading._shutdown() for the main thread (GH-28549) (GH-28589) https://github.com/python/cpython/commit/94d19f606fa18a1c4d2faca1caf2f470a8ce6d46 -- ___ Python tracker <https://bugs.python.org/issue1596321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45301] pycore_condvar.h: remove Windows conditonal variable emulation, use Windows native conditional variable
STINNER Victor added the comment: Eryk: > SleepConditionVariableSRW() can't be interrupted in PyCOND_WAIT() and > PyCOND_TIMEDWAIT(). Oh. A comment on StackOverlow says: "The WaitForXxx functions accept parameters of the generic HANDLE type, which represents a handle to a kernel object. Condition variables are user-mode objects, not kernel objects, so you cannot use them with these functions, since they work only with kernel objects." https://stackoverflow.com/questions/37522108/can-you-really-wait-on-condition-variable-with-waitfor-objects I only created this issue because of this comment in pycore_condvar.h: --- /* non-emulated condition variables are provided for those that want * to target Windows Vista. Modify this macro to enable them. */ #ifndef _PY_EMULATED_WIN_CV #define _PY_EMULATED_WIN_CV 1 /* use emulated condition variables */ #endif --- -- title: pycore_condvar.h: remove Windows conditonal variable emulation -> pycore_condvar.h: remove Windows conditonal variable emulation, use Windows native conditional variable ___ Python tracker <https://bugs.python.org/issue45301> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1596321] KeyError at exit after 'import threading' in other thread
Change by STINNER Victor : -- pull_requests: +26971 pull_request: https://github.com/python/cpython/pull/28589 ___ Python tracker <https://bugs.python.org/issue1596321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1596321] KeyError at exit after 'import threading' in other thread
STINNER Victor added the comment: New changeset 95d31370829b7d729667588e0a9943217401ea5b by Victor Stinner in branch 'main': bpo-1596321: Fix threading._shutdown() for the main thread (GH-28549) https://github.com/python/cpython/commit/95d31370829b7d729667588e0a9943217401ea5b -- ___ Python tracker <https://bugs.python.org/issue1596321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29971] Lock.acquire() not interruptible on Windows
STINNER Victor added the comment: time.sleep() is now implemented with a waitable timer and WaitForMultipleObjects() on Windows. It uses _PyOS_SigintEvent() so it can be interrupted with CTRL+C. commit 58f8adfda3c2b42f654a55500e8e3a6433cb95f2 Author: Victor Stinner Date: Wed Sep 22 16:09:30 2021 +0200 bpo-21302: time.sleep() uses waitable timer on Windows (GH-28483) On Windows, time.sleep() now uses a waitable timer which has a resolution of 100 ns (10^-7 sec). Previously, it had a solution of 1 ms (10^-3 sec). * On Windows, time.sleep() now calls PyErr_CheckSignals() before resetting the SIGINT event. * Add _PyTime_As100Nanoseconds() function. * Complete and update time.sleep() documentation. Co-authored-by: Livius -- ___ Python tracker <https://bugs.python.org/issue29971> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45301] pycore_condvar.h: remove Windows conditonal variable emulation
STINNER Victor added the comment: See also bpo-29971 "Lock.acquire() not interruptible on Windows". -- ___ Python tracker <https://bugs.python.org/issue45301> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29971] Lock.acquire() not interruptible on Windows
STINNER Victor added the comment: See also bpo-45301 "pycore_condvar.h: remove Windows conditonal variable emulation". -- ___ Python tracker <https://bugs.python.org/issue29971> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45301] pycore_condvar.h: remove Windows conditonal variable emulation
New submission from STINNER Victor : I recently worked on time.sleep() enhancement (bpo-21302) and threading bugfixes (bpo-45274, bpo-1596321). I saw one more time that Python emulates conditional variables to support Windows XP and older. But Python 3.11 requires Windows 8.1 or newer. IMO it's time to remove _PY_EMULATED_WIN_CV code path from pycore_condvar.h. -- components: Library (Lib) messages: 402720 nosy: paul.moore, steve.dower, tim.golden, vstinner, zach.ware priority: normal severity: normal status: open title: pycore_condvar.h: remove Windows conditonal variable emulation versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue45301> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29971] Lock.acquire() not interruptible on Windows
STINNER Victor added the comment: See also bpo-21822: some comments are about this issue. It's a duplicate of bpo-45274 "Race condition in Thread._wait_for_tstate_lock()". -- ___ Python tracker <https://bugs.python.org/issue29971> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45274] Race condition in Thread._wait_for_tstate_lock()
STINNER Victor added the comment: > This is the same as bpo-21822, so I suppose it should be closed as well with > a reference to this issue. You're right. I marked bpo-21822 as a duplicate of this issue. -- ___ Python tracker <https://bugs.python.org/issue45274> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21822] [Windows] KeyboardInterrupt during Thread.join hangs that Thread
STINNER Victor added the comment: I mark this issue as a duplicate of bpo-45274. -- I fixed bpo-45274 with this change: New changeset a22be4943c119fecf5433d999227ff78fc2e5741 by Victor Stinner in branch 'main': bpo-45274: Fix Thread._wait_for_tstate_lock() race condition (GH-28532) https://github.com/python/cpython/commit/a22be4943c119fecf5433d999227ff78fc2e5741 I tested join.py with the fix. It nows displays: --- vstinner@DESKTOP-DK7VBIL C:\vstinner\python\main>python x.py Running Debug|x64 interpreter... started. Press Ctrl-C now Ctrl-C [2.99] done=True alive=False finish [2.99] done=True alive=False Terminate batch job (Y/N)? n --- The script no longer hangs. -- resolution: -> duplicate stage: -> resolved status: open -> closed superseder: -> Race condition in Thread._wait_for_tstate_lock() ___ Python tracker <https://bugs.python.org/issue21822> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21822] [Windows] KeyboardInterrupt during Thread.join hangs that Thread
STINNER Victor added the comment: > I am having the same blocked signal issue on Windows when using Thread.join. > This program does not print "interrupted" after pressing Ctrl+C: This is a different issue: bpo-29971. Currently, threading.Lock.acquire() cannot be interrupted by CTRL+C. -- nosy: +vstinner ___ Python tracker <https://bugs.python.org/issue21822> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29971] Lock.acquire() not interruptible on Windows
STINNER Victor added the comment: Copy of Antoine Pitrou's msg316024 (bpo-21822): multiprocessing semaphores support Ctrl-C under Windows, so it should be doable for regular locks as well (notice the `sigint_event`): https://github.com/python/cpython/blob/master/Modules/_multiprocessing/semaphore.c#L109-L146 -- nosy: +vstinner ___ Python tracker <https://bugs.python.org/issue29971> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45300] Building Python documentation with doctest logs a ResourceWarning in Doc/library/nntplib.rst
New submission from STINNER Victor : Build Python documentatioin with: make -C Doc/ PYTHON=../python venv LANG= PYTHONTRACEMALLOC=10 make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W -j10" doctest 2>&1|tee log See the logs: /home/vstinner/python/main/Lib/socket.py:776: ResourceWarning: unclosed self._sock = None Object allocated at (most recent call last): File "/home/vstinner/python/main/Doc/venv/lib/python3.11/site-packages/sphinx/builders/__init__.py", lineno 361 self.write(docnames, list(updated_docnames), method) File "/home/vstinner/python/main/Doc/venv/lib/python3.11/site-packages/sphinx/ext/doctest.py", lineno 366 self.test_doc(docname, doctree) File "/home/vstinner/python/main/Doc/venv/lib/python3.11/site-packages/sphinx/ext/doctest.py", lineno 470 self.test_group(group) File "/home/vstinner/python/main/Doc/venv/lib/python3.11/site-packages/sphinx/ext/doctest.py", lineno 554 self.test_runner.run(test, out=self._warn_out, clear_globs=False) File "/home/vstinner/python/main/Lib/doctest.py", lineno 1495 return self.__run(test, compileflags, out) File "/home/vstinner/python/main/Lib/doctest.py", lineno 1348 exec(compile(example.source, filename, "single", File "", lineno 1 File "/home/vstinner/python/main/Lib/nntplib.py", lineno 334 self.sock = self._create_socket(timeout) File "/home/vstinner/python/main/Lib/nntplib.py", lineno 399 return socket.create_connection((self.host, self.port), timeout) File "/home/vstinner/python/main/Lib/socket.py", lineno 828 sock = socket(af, socktype, proto) -- assignee: docs@python components: Documentation messages: 402709 nosy: docs@python, vstinner priority: normal severity: normal status: open title: Building Python documentation with doctest logs a ResourceWarning in Doc/library/nntplib.rst versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue45300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45274] Race condition in Thread._wait_for_tstate_lock()
Change by STINNER Victor : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed versions: +Python 3.10, Python 3.9 ___ Python tracker <https://bugs.python.org/issue45274> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45274] Race condition in Thread._wait_for_tstate_lock()
STINNER Victor added the comment: New changeset fae2694bea5e9e5a114af8cb40b60e7131a6340c by Miss Islington (bot) in branch '3.10': bpo-45274: Fix Thread._wait_for_tstate_lock() race condition (GH-28532) (GH-28580) https://github.com/python/cpython/commit/fae2694bea5e9e5a114af8cb40b60e7131a6340c -- ___ Python tracker <https://bugs.python.org/issue45274> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45274] Race condition in Thread._wait_for_tstate_lock()
STINNER Victor added the comment: New changeset a22be4943c119fecf5433d999227ff78fc2e5741 by Victor Stinner in branch 'main': bpo-45274: Fix Thread._wait_for_tstate_lock() race condition (GH-28532) https://github.com/python/cpython/commit/a22be4943c119fecf5433d999227ff78fc2e5741 -- ___ Python tracker <https://bugs.python.org/issue45274> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45261] Unreliable (?) results from timeit (cache issue?)
Change by STINNER Victor : -- resolution: -> not a bug ___ Python tracker <https://bugs.python.org/issue45261> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36717] Allow retrieval of return value from the target of a threading.Thread
STINNER Victor added the comment: Storing the result of Thread.run() can keep Python objects alive longer than expected and it may require a lock on it. I don't think that the additional complexity is worth it. I suggest to reorganize your code to pass the result differently to the caller thread. You can use a queue, a variable, whatever works. -- nosy: +vstinner resolution: -> wont fix stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue36717> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27978] concurrent.futures.threading: Add a timeout to Executor shutdown() method
Change by STINNER Victor : -- title: [threading] Executor#shutdown with timeout -> concurrent.futures.threading: Add a timeout to Executor shutdown() method ___ Python tracker <https://bugs.python.org/issue27978> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1736792] dict reentrant/threading request
STINNER Victor added the comment: > As I first mentioned in my post[1] on comp.lang.python, it seems possible to > modify a dict while a lookup is ongoing, without causing the lookup to > restart. That's a bad practice. Python dict raises an exception in some cases: RuntimeError("dict mutated during update"). Detecting any change during a lookup would slow down, whereas dict performance is key in Python performance in general, since dict is used everywhere for Python namespaces. I close the issue as "wont fix". -- nosy: +vstinner resolution: -> wont fix stage: test needed -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue1736792> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45292] Implement PEP 654: Exception Groups
Change by STINNER Victor : -- title: Implement PEP 654 -> Implement PEP 654: Exception Groups ___ Python tracker <https://bugs.python.org/issue45292> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45020] Freeze all modules imported during startup.
STINNER Victor added the comment: Marc-Andre: I suppose that you're talking about LANDMARK in Modules/getpath.c and PC/getpathp.c. -- ___ Python tracker <https://bugs.python.org/issue45020> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux
STINNER Victor added the comment: > Do you have any information about when will be it released in 3.11? Here is a schedule of Python 3.11 releases: https://www.python.org/dev/peps/pep-0664/ In the meanwhile, you can develop a C extension to get the feature. -- ___ Python tracker <https://bugs.python.org/issue21302> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux
STINNER Victor added the comment: Thanks Livius for all these nice enhancements! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue21302> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux
STINNER Victor added the comment: New changeset 7834ff26cbcd4d8394d64d80d9f51a364d38b1c6 by Victor Stinner in branch 'main': bpo-21302: Add nanosleep() implementation for time.sleep() in Unix (GH-28545) https://github.com/python/cpython/commit/7834ff26cbcd4d8394d64d80d9f51a364d38b1c6 -- ___ Python tracker <https://bugs.python.org/issue21302> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45274] Race condition in Thread._wait_for_tstate_lock()
STINNER Victor added the comment: > I am not sure that it can be solved at Python level. Right. In pure Python, we cannot write code which works in all cases. My PR 28532 fix the most common case: application interrupted by a single CTRL+C. > with suppress_interrupt(): (...) It's important to be able to interrupt acquire() which can be called in blocking mode with no timeout: it's exactly what tox does, and users expect to be able to interrupt tox in this case. > * Add a Lock method (or just a builtin function) which acquires and > immediately releases the lock, without possibility to interrupt. The acquire()+release() sequence can be made atomic in C, but it doesn't solve the problem of _stop() which can be interrupted by a second exception. This bug is likely as old as Python. I don't think that we should attempt to design a perfect solution. I only propose to make the race condition (way) less likely. -- ___ Python tracker <https://bugs.python.org/issue45274> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41299] Python3 threading.Event().wait time is twice as large as Python27
STINNER Victor added the comment: I close the issue. > bpo-41299: Fix EnterNonRecursiveMutex() (GH-28548) I don't think that it's worth it to backport this change to 3.9 and 3.10. In practice, Py_FatalError() will never be called, and _PyTime_GetMonotonicClock() result must always be greater than 0. But I wanted to remove it in the main branch to reduce the number of functions calling Py_FatalError(). -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue41299> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41299] Python3 threading.Event().wait time is twice as large as Python27
STINNER Victor added the comment: New changeset 6bc89116cb121c176453b9745e3daa2cc5df43ab by Victor Stinner in branch 'main': bpo-41299: Mark private thread_nt.h functions as static (GH-28553) https://github.com/python/cpython/commit/6bc89116cb121c176453b9745e3daa2cc5df43ab -- ___ Python tracker <https://bugs.python.org/issue41299> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing)
STINNER Victor added the comment: Analysis use use_tracing usage in 3rd part code. I see two main ways to add C API functions covering these use cases: * Provide high-level functions like "call a trace function" (disable tracing, call trace function, reenable tracing, increment/decrement tstate->tracing) * Provide low-level functions just to control use_tracing: make PyThreadState structure opaque, but stil make the assumption that it is possible to disable temporarily tracing and profiling (in practice, it's implemented as use_tracing=0). (*) greenlet greenlet disables temporarily tracing in g_calltrace(), and then restore it, to call a "tracing" function: --- tstate->tracing++; TSTATE_USE_TRACING(tstate) = 0; retval = PyObject_CallFunction(tracefunc, "O(OO)", event, origin, target); tstate->tracing--; TSTATE_USE_TRACING(tstate) = (tstate->tracing <= 0 && ((tstate->c_tracefunc != NULL) || (tstate->c_profilefunc != NULL))); --- It also saves and then restores use_tracing value: --- ts__g_switchstack_use_tracing = tstate->cframe->use_tracing; (...) tstate->cframe->use_tracing = ts__g_switchstack_use_tracing; --- => it can use PyThreadState_IsTracing(), PyThreadState_DisableTracing() and PyThreadState_ResetTracing(). These functions don't handle "tstate->tracing++;" and "tstate->tracing--;" which is also used by greenlet. greenlet also saves and restores tstate->cframe: https://github.com/python-greenlet/greenlet/blob/master/src/greenlet/greenlet.c (*) dipy Code generated by Cython. (*) smartcols Code generated by Cython. (*) yappi yappi is Python profiler. yappi sets use_tracing to 1 when it sets its profile function: "ts->c_profilefunc = _yapp_callback;". It sets use_tracing to 0 when it clears the profile function: "ts->c_profilefunc = NULL;". That's wrong, it ignores the trace function. PyEval_SetProfile() cannot be used because yappi works on a PyThreadState (ts). Code: https://github.com/sumerc/yappi/blob/master/yappi/_yappi.c It can use PyThreadState_DisableTracing() and PyThreadState_ResetTracing(). Maybe a PyThreadState_SetProfile(tstate, func) function would fit better yappi's use case. (*) Cython Cython defines 2 compatibility functions: * __Pyx_IsTracing(tstate, check_tracing, check_funcs): it can check c_profilefunc and c_tracefunc * __Pyx_SetTracing(tstate, enable) Code: https://github.com/cython/cython/blob/0.29.x/Cython/Utility/Profile.c The code is quite complicated. In short, it checks if tracing and/or profiling is enabled. If it's enabled, it disables temporarily tracing (use_tracing=0) while calling trace and profile functions. => it requires PyThreadState_IsTracing(), PyThreadState_DisableTracing() and PyThreadState_ResetTracing(). -- ___ Python tracker <https://bugs.python.org/issue43760> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41299] Python3 threading.Event().wait time is twice as large as Python27
STINNER Victor added the comment: New changeset d639e3170506a1a80c4b2f1b3a6250bb95ff190a by Victor Stinner in branch 'main': bpo-41299: Fix EnterNonRecursiveMutex() (GH-28548) https://github.com/python/cpython/commit/d639e3170506a1a80c4b2f1b3a6250bb95ff190a -- ___ Python tracker <https://bugs.python.org/issue41299> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41299] Python3 threading.Event().wait time is twice as large as Python27
STINNER Victor added the comment: New changeset f35ddf24227e834c9b6b39ad23a0ec382b4de48b by Victor Stinner in branch 'main': bpo-41299: QueryPerformanceFrequency() cannot fail (GH-28552) https://github.com/python/cpython/commit/f35ddf24227e834c9b6b39ad23a0ec382b4de48b -- ___ Python tracker <https://bugs.python.org/issue41299> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41299] Python3 threading.Event().wait time is twice as large as Python27
Change by STINNER Victor : -- pull_requests: +26936 pull_request: https://github.com/python/cpython/pull/28553 ___ Python tracker <https://bugs.python.org/issue41299> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41299] Python3 threading.Event().wait time is twice as large as Python27
Change by STINNER Victor : -- pull_requests: +26935 pull_request: https://github.com/python/cpython/pull/28552 ___ Python tracker <https://bugs.python.org/issue41299> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40699] Memory leak in threading library with Python 3.6-3.8
STINNER Victor added the comment: I fail to reproduce the leak using attached script. I close the issue. I get constant memory usage on Linux with the main branch of Python (future 3.11): (...) VmRSS: 11344 kB VmRSS: 11344 kB VmRSS: 11344 kB VmRSS: 11344 kB VmRSS: 11344 kB VmRSS: 11344 kB VmRSS: 11344 kB VmRSS: 11344 kB VmRSS: 11344 kB VmRSS: 11344 kB (...) -- nosy: +vstinner resolution: -> not a bug stage: -> resolved status: open -> closed Added file: https://bugs.python.org/file50304/script.py ___ Python tracker <https://bugs.python.org/issue40699> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40657] Resource leaks with threading.Thread
STINNER Victor added the comment: You must join thread using thread.join(). If I add .join() calls, threading.active_count() stays around 2, it doesn't grow. It's not a bug in Python. -- resolution: -> not a bug stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue40657> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43050] threading timer memory leak
STINNER Victor added the comment: I cannot reproduce the issue. IMO it has been fixed. Moreover, you must join timers using timer.join(): a timer remains a thread. Code: --- import os import threading os.system(f"grep ^VmRSS /proc/{os.getpid()}/status") # warmup for n in range(10): timer = threading.Timer(5, None) timer.start() timer.cancel() timer.join() os.system(f"grep ^VmRSS /proc/{os.getpid()}/status") for n in range(1000): timer = threading.Timer(5, None) timer.start() timer.cancel() timer.join() os.system(f"grep ^VmRSS /proc/{os.getpid()}/status") --- Output on Linux with the main branch of Python (3.11): --- VmRSS: 10924 kB VmRSS: 11104 kB VmRSS: 11104 kB --- -- nosy: +vstinner resolution: duplicate -> fixed stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue43050> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1596321] KeyError at exit after 'import threading' in other thread
STINNER Victor added the comment: I proposed PR 28549 to fix this very old threading issue. A C extension can spawn threads without using the threading module and then then run Python code which imports the threading module. In this case, threading._main_thread is the thread which imported first the threading module. My PR changes so threading._shutdown() simply ignores _main_thread when it happens. -- ___ Python tracker <https://bugs.python.org/issue1596321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1596321] KeyError at exit after 'import threading' in other thread
Change by STINNER Victor : -- keywords: +patch nosy: +vstinner nosy_count: 14.0 -> 15.0 pull_requests: +26932 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/28549 ___ Python tracker <https://bugs.python.org/issue1596321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41299] Python3 threading.Event().wait time is twice as large as Python27
STINNER Victor added the comment: Calling Py_FatalError() must be avoided by any mean. I spent significant time to remove Py_FatalError() at Python startup in my PEP 587 implementation: https://www.python.org/dev/peps/pep-0587/ I also removed Py_FatalError() calls to report errors to the caller: https://bugs.python.org/issue38631 Calling Py_FatalError() is really bad when Python is embedded in an application. It kills the whole process, the caller has no way to catch the error. -- ___ Python tracker <https://bugs.python.org/issue41299> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41299] Python3 threading.Event().wait time is twice as large as Python27
STINNER Victor added the comment: I created PR #28548 to remove the Py_FatalError() code path. -- ___ Python tracker <https://bugs.python.org/issue41299> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41299] Python3 threading.Event().wait time is twice as large as Python27
Change by STINNER Victor : -- pull_requests: +26931 pull_request: https://github.com/python/cpython/pull/28548 ___ Python tracker <https://bugs.python.org/issue41299> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42969] pthread_exit & PyThread_exit_thread from PyEval_RestoreThread etc. are harmful
STINNER Victor added the comment: Jeremy Maitin-Shepard: "In general, I view hanging threads as the least bad thing to do when faced with either acquiring the GIL or not returning at all. There is a lot of existing usage of Python that currently poses a risk of random crashes and memory corruption while Python is exiting, and I would like to fix that." Showing warnings by default or not was discussed many times in Python. It was decided to *hide* DeprecationWarning by default. The PEP 565 is a minor trade-off to show them in the __main__ module. For me, more generally, Python default behavior is designed for *users* who don't want to be annoyed by warnings or anything which would make sense for *developers*. That's why I designed a new "Python Development Mode" (-X dev): https://docs.python.org/dev/library/devmode.html Maybe in development mode, the behavior could be changed to call abort(). But honestly, I'm not really excited by that. I'm not embedding Python in a C++ application. I'm almot only use Python directly: the Unix command "python3". For this use case, IMO it's fine to call pthread_exit() by default. -- ___ Python tracker <https://bugs.python.org/issue42969> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42969] pthread_exit & PyThread_exit_thread from PyEval_RestoreThread etc. are harmful
STINNER Victor added the comment: Another example where a developer asks to call abort() to notice bugs, whereas Python previously silently ignored it: bpo-36829. Calling abort() is a legit use case, but not really the best default behavior. Again, the problem was solved by letting developers setting their own callback: sys.unraisablehook. If I understood correctly, pytest doesn't override it but "took" into the default implementation: it chains its own code with the default implementation. It's possible because there is a way to "get" the current hook: just read sys.unraisablehook ;-) Another argument in favor of also adding Py_GetThreadExitCallback() ;-) -- ___ Python tracker <https://bugs.python.org/issue42969> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42969] pthread_exit & PyThread_exit_thread from PyEval_RestoreThread etc. are harmful
STINNER Victor added the comment: Gregory P. Smith: Python has many API using callbacks: PEP 445 added PyMem_SetAllocator() to set memory allocator. Adding PyMem_GetAllocator() also made possible to chain allocators and to "hook" into an existing allocator to execute code before and after it's called (the PEP contains an example). I'm not sure if it's important or useless to chain callbacks with Py_SetThreadExitCallback(). I suggest to always override the previously set callback. It would matter if library A sets a callback to emit log and library B sets a callback to hang threads. It maybe be nice to first emit a log and then hang the thread. But then the order in which callbacks are set starts to matter a lot :-) I'm fine with adding Py_GetThreadExitCallback() if you consider that it matters. > If someone passes nullptr does that undo it (please no!). I don't think that we should bother with adding a special case. I prefer to consider developers as adults and let them make their own mistakes if they consider that they understand the code well enough ;-) _PyEval_SetTrace() allows to remove the current trace function. It's a legit use case. If library C is annoyed by library A and library B installed annoying callbacks, IMO it's also ok to let it "remove" the previously set callback, no? IMO Py_SetThreadExitCallback(NULL) should simply set the callback to NULL, so restore the default behavior: call pthread_exit(). -- ___ Python tracker <https://bugs.python.org/issue42969> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42969] pthread_exit & PyThread_exit_thread from PyEval_RestoreThread etc. are harmful
STINNER Victor added the comment: > In this case that unexpected thread simply disappearing can lead to a > deadlock in our process. This problem also remains me the very complex case of bpo-6721: "Locks in the standard library should be sanitized on fork". The issue title looks simple, but 12 years after the issue was created, it's still open. This issue is being solved by adding atfork callbacks to modules which must do something at fork in the child process (sometimes also in the parent process). I added threading.Lock._at_fork_reinit() private method to simplify the implementation of these callbacks. Such problem has no silver bullet solution, so it's better to let developers design their own solution with their specific requirements. -- ___ Python tracker <https://bugs.python.org/issue42969> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux
Change by STINNER Victor : -- pull_requests: +26929 pull_request: https://github.com/python/cpython/pull/28545 ___ Python tracker <https://bugs.python.org/issue21302> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing)
Change by STINNER Victor : -- pull_requests: +26926 pull_request: https://github.com/python/cpython/pull/28542 ___ Python tracker <https://bugs.python.org/issue43760> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45274] Race condition in Thread._wait_for_tstate_lock()
Change by STINNER Victor : -- nosy: +pablogsal stage: patch review -> ___ Python tracker <https://bugs.python.org/issue45274> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45274] Race condition in Thread._wait_for_tstate_lock()
Change by STINNER Victor : -- pull_requests: +26920 stage: -> patch review pull_request: https://github.com/python/cpython/pull/28532 ___ Python tracker <https://bugs.python.org/issue45274> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45274] Race condition in Thread._wait_for_tstate_lock()
STINNER Victor added the comment: For curious people, see also bpo-44422 "Fix threading.enumerate() reentrant call": another example of race condition recently fixed in the threading module. But it's unrelated to this bug ;-) -- ___ Python tracker <https://bugs.python.org/issue45274> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45274] Race condition in Thread._wait_for_tstate_lock()
Change by STINNER Victor : Added file: https://bugs.python.org/file50300/threading_bug.py ___ Python tracker <https://bugs.python.org/issue45274> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45274] Race condition in Thread._wait_for_tstate_lock()
New submission from STINNER Victor : Bernát Gábor found an interesting bug on Windows. Sometimes, when a process is interrupted on Windows with CTRL+C, its parent process hangs in thread.join(): https://twitter.com/gjbernat/status/1440949682759426050 Reproducer: * Install https://github.com/gaborbernat/tox/tree/2159 * Make an empty folder and put the above tox.ini in it * Invoke python -m tox and once that command is sleeping press CTRL+C (the app should lock there indefinitely). tox.ini: --- [testenv] skip_install = true commands = python -c 'import os; print(f"start {os.getpid()}"); import time; time.sleep(100); print("end");' --- Source: https://gist.github.com/gaborbernat/f1e1aff0f2ee514b50a92a4d019d4d13 I tried to attach the Python process in Python: there is a single thread, the main thread which is blocked in thread.join(). You can also see it in the faulthandler traceback. I did a long analysis of the _tstate_lock and I checked that thread really completed. Raw debug traces: == thread 6200 exit == thread_run[pid=3984, thread_id=6200]: clear PyThreadState_Clear[pid=3984, thread_id=6200]: on_delete() release_sentinel[pid=3984, thread_id=6200]: enter release_sentinel[pid=3984, thread_id=6200]: release(obj=01C1122669C0, lock=01C110BBDA00) release_sentinel[pid=3984, thread_id=6200]: exit PyThreadState_Clear[pid=3984, thread_id=6200]: on_delete()-- == main thread is calling join() but gets a KeyboardInterrupt == [pid=3984, thread_id=8000] Lock.acquire() -> ACQUIRED Current thread 0x1f40 (most recent call first): File "C:\vstinner\python\3.10\lib\threading.py", line 1118 in _wait_for_tstate_lock File "C:\vstinner\python\3.10\lib\threading.py", line 1092 in join File "C:\vstinner\env\lib\site-packages\tox\session\cmthread_run[pid=3984, thread_id=6200]: exit d\run\common.py", line 203 in execute File "C:\vstinner\env\lib\site-packages\tox\session\cmd\run\sequential.py", line 20 in run_sequential File "C:\vstinner\env\lib\site-packages\tox\session\cmd\legacy.py", line 104 in legacy File "C:\vstinner\env\lib\site-packages\tox\run.py", line 49 in main File "C:\vstinner\env\lib\site-packages\tox\run.py", line 23 in run File "C:\vstinner\env\lib\site-packages\tox\__main__.py", line 4 in File "C:\vstinner\python\3.10\lib\runpy.py", line 86 in _run_code File "C:\vstinner\python\3.10\lib\runpy.py", line 196 in _run_module_as_main _wait_for_tstate_lock[pid=3984, current thread_id=8000, self thread_id=6200]: EXC: KeyboardInterrupt(); acquired? None == main thread calls repr(thread) == ROOT: [3984] KeyboardInterrupt - teardown started _wait_for_tstate_lock[pid=3984, current thread_id=8000, self thread_id=6200]: acquire(block=False, timeout=-1): lock obj= 0x1c1122669c0 File "C:\vstinner\python\3.10\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\vstinner\python\3.10\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\vstinner\env\lib\site-packages\tox\__main__.py", line 4, in run() File "C:\vstinner\env\lib\site-packages\tox\run.py", line 23, in run result = main(sys.argv[1:] if args is None else args) File "C:\vstinner\env\lib\site-packages\tox\run.py", line 49, in main result = handler(state) File "C:\vstinner\env\lib\site-packages\tox\session\cmd\legacy.py", line 104, in legacy return run_sequential(state) File "C:\vstinner\env\lib\site-packages\tox\session\cmd\run\sequential.py", line 20, in run_sequential return execute(state, max_workers=1, has_spinner=False, live=True) File "C:\vstinner\env\lib\site-packages\tox\session\cmd\run\common.py", line 217, in execute print(f'join {thread}') File "C:\vstinner\python\3.10\lib\threading.py", line 901, in __repr__ self.is_alive() # easy way to get ._is_stopped set when appropriate File "C:\vstinner\python\3.10\lib\threading.py", line 1181, in is_alive self._wait_for_tstate_lock(False) File "C:\vstinner\python\3.10\lib\threading.py", line 1113, in _wait_for_tstate_lock traceback.print_stack() _wait_for_tstate_lock[pid=3984, current thread_id=8000, self thread_id=6200]: failed to acquire 0x1c1122669c0 _wait_for_tstate_lock[pid=3984, current thread_id=8000, self thread_id=6200]: exit 0x1c1122669c0 join == main thread calls thread.join()... which hangs == _wait_for_tstate_lock[pid=3984, current thread_id=8000, self thread_id=6200]: acquire(block=True, timeout=-1): lock obj= 0x1c1122669c0 File "C:\vstinner\python\3.10\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\vstinner\python\3.10\lib\runpy.py", line 86, in _run_code exec(code, run_globals) F
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing)
STINNER Victor added the comment: New changeset f4ccb79d52ee726d58bbb038ea98b4deec52001d by Victor Stinner in branch 'main': bpo-43760: Document PyThreadState.use_tracing removal (GH-28527) https://github.com/python/cpython/commit/f4ccb79d52ee726d58bbb038ea98b4deec52001d -- ___ Python tracker <https://bugs.python.org/issue43760> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42969] pthread_exit & PyThread_exit_thread from PyEval_RestoreThread etc. are harmful
STINNER Victor added the comment: I'm not comfortable with PR 28525 which always hang threads which attempt to acquire the GIL after Python exited. I would prefer to keep the current behavior by default, but give the ability to applications embedding Python to decide what to do. With my Py_SetThreadExitCallback(func) idea, PyThread_exit_thread() would call func() and then pthread_exit(0). Applications can hang threads, log a message, call abort(), or whatever else. I'm not comfortable with writing a portable function to "hang a thread". For example, I don't understand why PR 28525 processes Windows messages on hang threads. Well, it's a complex problem :-( -- ___ Python tracker <https://bugs.python.org/issue42969> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41137] pdb uses the locale encoding for .pdbrc
STINNER Victor added the comment: New changeset ec7ffa4b5b262369f726a54e145e9c03eaeb4c1a by Victor Stinner in branch 'main': bpo-41137: Reorganize What's New in Python 3.11 (GH-28518) https://github.com/python/cpython/commit/ec7ffa4b5b262369f726a54e145e9c03eaeb4c1a -- ___ Python tracker <https://bugs.python.org/issue41137> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing)
STINNER Victor added the comment: I created PR 28527 to document PyThreadState.use_tracing removal and explain how to port existing code to Python 3.10. -- ___ Python tracker <https://bugs.python.org/issue43760> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing)
Change by STINNER Victor : -- pull_requests: +26918 pull_request: https://github.com/python/cpython/pull/28527 ___ Python tracker <https://bugs.python.org/issue43760> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45256] Remove the usage of the C stack in Python to Python calls
Change by STINNER Victor : -- title: Remove the usage of the cstack in Python to Python calls -> Remove the usage of the C stack in Python to Python calls ___ Python tracker <https://bugs.python.org/issue45256> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45261] Unreliable (?) results from timeit (cache issue?)
STINNER Victor added the comment: I suggest to close this issue. timeit works as expected. It has limitations, but that's ok. Previous attempts to enhance timeit were rejected. -- ___ Python tracker <https://bugs.python.org/issue45261> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45261] Unreliable (?) results from timeit (cache issue?)
STINNER Victor added the comment: > but you give no reason for why you should not use the minimum. See https://pyperf.readthedocs.io/en/latest/analyze.html#minimum-vs-average I'm not really interested to convince you. Use the minimum if you believe that it better fits your needs. But pyperf will stick to the mean to get more reproducible benchmark results ;-) There are many believes and assumptions made by people running benchmarks. As I wrote, do your own experiments ;-) Enjoy multimodal distributions ;-) -- ___ Python tracker <https://bugs.python.org/issue45261> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux
STINNER Victor added the comment: See also bpo-19007: "precise time.time() under Windows 8: use GetSystemTimePreciseAsFileTime". -- ___ Python tracker <https://bugs.python.org/issue21302> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux
STINNER Victor added the comment: Livius: do you care about using nanosleep(), or can I close the issue? -- ___ Python tracker <https://bugs.python.org/issue21302> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux
STINNER Victor added the comment: > On Windows with a Python 3.11 debug build, I get: > Mean +- std dev: 21.9 ms +- 7.8 ms (228 values) I wrote an optimization to cache the Windows timer handle between time.sleep() calls (don't close it). I don't think that it's needed because they shortest sleep is about 15.6 ms. CreateWaitableTimerW() is likely way more fast than 15.6 ms. So this optimization is basically useless. -- ___ Python tracker <https://bugs.python.org/issue21302> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux
STINNER Victor added the comment: bench.py: measure the shortest possible sleep. Use time.sleep(1e-10): 0.1 nanosecond. It should be rounded to the resolution of the used sleep function, like 1 ns on Linux. On Linux with Fedora 34 Python 3.10 executable, I get: Mean +- std dev: 60.5 us +- 12.9 us (80783 values) On Windows with a Python 3.11 debug build, I get: Mean +- std dev: 21.9 ms +- 7.8 ms (228 values) Sadly, it seems like on Windows 10, one of the following function still uses the infamous 15.6 ms resolution: * CreateWaitableTimerW() * SetWaitableTimer() * WaitForMultipleObjects() -- Added file: https://bugs.python.org/file50294/bench.py ___ Python tracker <https://bugs.python.org/issue21302> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux
STINNER Victor added the comment: Livius: your first PR modified Sleep() in Modules/_tkinter.c to use nanosleep(). I don't see the point since this function has a solution of 1 ms (10^-3). Using select() on Unix is enough: resolution of 1 us (10^-6). -- ___ Python tracker <https://bugs.python.org/issue21302> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux
STINNER Victor added the comment: New changeset 58f8adfda3c2b42f654a55500e8e3a6433cb95f2 by Victor Stinner in branch 'main': bpo-21302: time.sleep() uses waitable timer on Windows (GH-28483) https://github.com/python/cpython/commit/58f8adfda3c2b42f654a55500e8e3a6433cb95f2 -- ___ Python tracker <https://bugs.python.org/issue21302> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45256] Remove the usage of the cstack in Python to Python calls
Change by STINNER Victor : -- nosy: +vstinner ___ Python tracker <https://bugs.python.org/issue45256> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28307] Accelerate 'string' % (value, ...) by using formatted string literals
STINNER Victor added the comment: commit a0bd9e9c11f5f52c7ddd19144c8230da016b53c6 Author: Serhiy Storchaka Date: Sat May 8 22:33:10 2021 +0300 bpo-28307: Convert simple C-style formatting with literal format into f-string. (GH-5012) C-style formatting with literal format containing only format codes %s, %r and %a (with optional width, precision and alignment) will be converted to an equivalent f-string expression. It can speed up formatting more than 2 times by eliminating runtime parsing of the format string and creating temporary tuple. commit 8b010673185d36d13e69e5bf7d902a0b3fa63051 Author: Serhiy Storchaka Date: Sun May 23 19:06:48 2021 +0300 bpo-28307: Tests and fixes for optimization of C-style formatting (GH-26318) Fix errors: * "%10.s" should be equal to "%10.0s", not "%10s". * Tuples with starred expressions caused a SyntaxError. -- nosy: +vstinner ___ Python tracker <https://bugs.python.org/issue28307> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45261] Unreliable (?) results from timeit (cache issue?)
STINNER Victor added the comment: > But timing results are not like that, the measurement errors are one-sided, not two: (..) I suggest you to run tons of benchmarks and look at the distribution. The reality is more complex than what you may think. > measurement = true value + random error In my experience, there is no single "true value", they are multiple values. Concrete example where the Python randomized hash function gives different value each time you spawn a new Python process: https://vstinner.github.io/journey-to-stable-benchmark-average.html Each process has its own "true value", but pyperf spawns 20 Python processes :-) There are multiple sources of randomness, not only the Python randomized hash function. On Linux, the process address space is randomized by ASLR. I may give different timing at each run. Code placement, exact memory address, etc. Many things enter into the game when you look into functions which take less than 100 ns. Here is report is about a value lower than a single nanosecond: "0.08ns/element". -- I wrote articles about benchmarking: https://vstinner.readthedocs.io/benchmark.html#my-articles I gave a talk about it: * https://raw.githubusercontent.com/vstinner/talks/main/2017-FOSDEM-Brussels/howto_run_stable_benchmarks.pdf * https://archive.fosdem.org/2017/schedule/event/python_stable_benchmark/ Again, good luck with benchmarking, it's a hard problem ;-) -- Once you will consider that you know everything about benchmarking, you should read the following paper and cry: https://arxiv.org/abs/1602.00602 See also my analysis of PyPy performance: https://vstinner.readthedocs.io/pypy_warmups.html -- ___ Python tracker <https://bugs.python.org/issue45261> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41137] pdb uses the locale encoding for .pdbrc
Change by STINNER Victor : -- nosy: +vstinner nosy_count: 4.0 -> 5.0 pull_requests: +26909 pull_request: https://github.com/python/cpython/pull/28518 ___ Python tracker <https://bugs.python.org/issue41137> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45261] Unreliable (?) results from timeit (cache issue?)
STINNER Victor added the comment: PyPy emits a warning when the timeit module is used, suggesting to use pyperf. timeit uses the minimum, whereas pyperf uses the average (arithmetic mean). timeit uses a single process, pyperf spawns 21 processes: 1 just for the loop calibration, 20 to compute values. timeit computes 5 values, pyperf computes 60 values. timeit uses all computed values, pyperf ignores the first value considered as a "warmup value" (the number of warmup values can be configured). timeit doesn't compute the standard deviation, pyperf does. The standard deviation gives an idea if the benchmark looks reliable or not. IMO results without standard deviation should not be trusted. pyperf also emits warning when a benchmark doesn't look reliable. For example, if the user ran various workload while the benchmark was running. pyperf also supports storing results in a JSON file which stores all values, but also metadata. I cannot force people to stop using timeit. But there are reason why pyperf is more reliable than timeit. Benchmarking is hard. See pyperf documentation giving hints how to get reproducible benchmark results: https://pyperf.readthedocs.io/en/latest/run_benchmark.html#how-to-get-reproducible-benchmark-results Read also this important article ;-) "Biased Benchmarks (honesty is hard)" http://matthewrocklin.com/blog/work/2017/03/09/biased-benchmarks -- ___ Python tracker <https://bugs.python.org/issue45261> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com