[issue44751] crypt.h should be in _cryptmodule.c, not in public header

2021-09-29 Thread STINNER Victor


STINNER Victor  added the comment:


New changeset 80285ecc8deaa2b0e7351bf4be863d1a0ad3c188 by Miss Islington (bot) 
in branch '3.10':
closes bpo-44751: Move crypt.h include from public header to _cryptmodule 
(GH-27394) (GH-28636)
https://github.com/python/cpython/commit/80285ecc8deaa2b0e7351bf4be863d1a0ad3c188


--
nosy: +vstinner

___
Python tracker 
<https://bugs.python.org/issue44751>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue43753] [C API] Add Py_Is(x, y) and Py_IsNone(x) functions

2021-09-29 Thread STINNER Victor


Change by STINNER Victor :


--
pull_requests: +27009
pull_request: https://github.com/python/cpython/pull/28641

___
Python tracker 
<https://bugs.python.org/issue43753>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue25653] ctypes+callbacks+fork+selinux = crash

2021-09-29 Thread STINNER Victor


STINNER Victor  added the comment:

Another recent crash involving libffi, closure, fork and SELinux: 
https://bugzilla.redhat.com/show_bug.cgi?id=1977410 This bug comes from libffi, 
not from Python (but it can be easily reproducing using ctypes which uses 
libffi).

--
nosy: +vstinner

___
Python tracker 
<https://bugs.python.org/issue25653>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45262] crash if asyncio is used before and after re-initialization if using python embedded in an application

2021-09-29 Thread STINNER Victor


STINNER Victor  added the comment:

> "Assertion failed: Py_IS_TYPE(rl, _Type), file 
> D:\a\1\s\Modules_asynciomodule.c, line 261"

The _asyncio extension uses static types and doesn't implement the multiphase 
initialization API. It doesn't work well with your "deinitialize/initialize the 
interpreter" use case.

The _asyncio extension should use heap types and the multiphase init API.

--
nosy: +vstinner

___
Python tracker 
<https://bugs.python.org/issue45262>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45325] Allow "p" in Py_BuildValue

2021-09-29 Thread STINNER Victor


STINNER Victor  added the comment:

> but there is a catch -- the arguments should be a C int. If it is not, you 
> can break a stack.

I never understood how "..." arguments work under the hood. What happens if you 
pass a double, it is stored as a double on the C stack, and then 
Py_BuildValue() will read junk data?

--

___
Python tracker 
<https://bugs.python.org/issue45325>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45325] Allow "p" in Py_BuildValue

2021-09-29 Thread STINNER Victor


STINNER Victor  added the comment:

I would be interested to see how the "p" format could be used in existing code. 
I found a few places would benefit of it. Either create a new draft PR, or put 
it in the same PR 28634.

Modules/_itertoolsmodule.c:

return Py_BuildValue("O(N)(OO)", Py_TYPE(lz), it, lz->saved, Py_True);

return Py_BuildValue("O(O)(OO)", Py_TYPE(lz), lz->it, lz->saved,
 lz->firstpass ? Py_True : Py_False);

Modules/_ssl.c:

ok = RAND_bytes((unsigned char*)PyBytes_AS_STRING(bytes), len);
if (ok == 0 || ok == 1)
return Py_BuildValue("NO", bytes, ok == 1 ? Py_True : Py_False);

return Py_BuildValue(
"{sksssisi"
"sO"
"}",
"id", cipher_id,
"name", cipher_name,
"protocol", cipher_protocol,
"description", buf,
"strength_bits", strength_bits,
"alg_bits", alg_bits
,"aead", aead ? Py_True : Py_False,
"symmetric", skcipher,
"digest", digest,
"kea", kx,
"auth", auth
   );


Modules/main.c:

runargs = PyTuple_Pack(2, module, set_argv0 ? Py_True : Py_False);

Objects/fileobject.c:

stream = _PyObject_CallMethodId(io, _open, "isisssO", fd, mode,
 buffering, encoding, errors,
 newline, closefd ? Py_True : Py_False);

--
nosy: +vstinner

___
Python tracker 
<https://bugs.python.org/issue45325>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue36207] robotsparser deny all with some rules

2021-09-29 Thread STINNER Victor


Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg402889

___
Python tracker 
<https://bugs.python.org/issue36207>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41710] acquire(timeout) of threading.Lock and threading.Condition is affected by jumps in system time: Python should use sem_clockwait(CLOCK_MONOTONIC)

2021-09-29 Thread STINNER Victor


Change by STINNER Victor :


--
keywords: +patch
pull_requests: +26999
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/28629

___
Python tracker 
<https://bugs.python.org/issue41710>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45322] [X86_64]test_concurrent_futures fail

2021-09-29 Thread STINNER Victor


STINNER Victor  added the comment:

> File 
> "/home/abuild/rpmbuild/BUILD/Python-3.7.9/Lib/multiprocessing/queues.py", 
> line 224 in _feed

Python 3.7 no longer accept bugfixes: I close the issue.

You can open a new issue if you can reproduce the issue on Python 3.9 or newer:
https://devguide.python.org/#status-of-python-branches

--
resolution:  -> wont fix
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue45322>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45319] Possible regression in __annotations__ descr for heap type subclasses

2021-09-29 Thread STINNER Victor


Change by STINNER Victor :


--
nosy: +petr.viktorin, vstinner

___
Python tracker 
<https://bugs.python.org/issue45319>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39026] Include/cpython/pystate.h contains non-relative of initconfig.h include causing macOS Framework include failure

2021-09-29 Thread STINNER Victor


STINNER Victor  added the comment:

I propose to only rename header files in Python 3.11.

For Python 3.9 and 3.10, I suggest to modify Include/cpython/pystate.h by 
replacing:
  #include "cpython/initconfig.h"
with:
  #include "initconfig.h"

Like the PR 20181.

--

___
Python tracker 
<https://bugs.python.org/issue39026>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39026] Include/cpython/pystate.h contains non-relative of initconfig.h include causing macOS Framework include failure

2021-09-29 Thread STINNER Victor


STINNER Victor  added the comment:

I'm not sure why Include/cpython/ is part of the compiler include directories. 
I am not sure why Include/ is *not* part of the compiler include directory.

Anyway, it's hard to control how people build Python and there are always legit 
use cases where you fall into such issue.

That's why I suggest to fix any risk of confusion by renaming Include/cpython/ 
header files to add a "cpython_" prefix. For example, rename 
Include/cpython/pystate.h to Include/cpython/cpython_pystate.h.

In the Include/internal/ directory, header files already have a prefix for the 
exact same reason. For example, there is a pycore_pystate.h file there.

--

___
Python tracker 
<https://bugs.python.org/issue39026>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45308] multiprocessing default fork child process will not free object, which is inherited from parent process

2021-09-29 Thread STINNER Victor


Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 
<https://bugs.python.org/issue45308>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39026] Include/cpython/pystate.h contains non-relative of initconfig.h include causing macOS Framework include failure

2021-09-29 Thread STINNER Victor


Change by STINNER Victor :


--
title: pystate.h contains non-relative of initconfig.h include causing macOS 
Framework include failure -> Include/cpython/pystate.h contains non-relative of 
initconfig.h include causing macOS Framework include failure

___
Python tracker 
<https://bugs.python.org/issue39026>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39026] pystate.h contains non-relative of initconfig.h include causing macOS Framework include failure

2021-09-29 Thread STINNER Victor


Change by STINNER Victor :


--
pull_requests: +26982
pull_request: https://github.com/python/cpython/pull/28612

___
Python tracker 
<https://bugs.python.org/issue39026>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39026] pystate.h contains non-relative of initconfig.h include causing macOS Framework include failure

2021-09-29 Thread STINNER Victor


STINNER Victor  added the comment:

I marked bpo-40642 as a duplicate of this issue.

--

___
Python tracker 
<https://bugs.python.org/issue39026>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40642] Cpython "pystate.h" subdirectory wrong

2021-09-29 Thread STINNER Victor


STINNER Victor  added the comment:

I mark this issue as a duplicate of bpo-39026.

--
resolution:  -> duplicate
stage: patch review -> resolved
status: open -> closed
superseder:  -> pystate.h contains non-relative of initconfig.h include causing 
macOS Framework include failure

___
Python tracker 
<https://bugs.python.org/issue40642>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40642] Cpython "pystate.h" subdirectory wrong

2021-09-29 Thread STINNER Victor


STINNER Victor  added the comment:

Having two header files with the same name in two directories which are part of 
the include directories is causing issues like this one :-(

For the internal C API, I solve the issue by adding "pycore_" prefix to 
filenames.

Maybe pystate.h should be renamed to cpython_pystate.h to avoid to fix the 
issue. It would include cpython_initconfig.h.

--
nosy: +vstinner

___
Python tracker 
<https://bugs.python.org/issue40642>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45201] API function PySignal_SetWakeupFd is not exported and unusable

2021-09-29 Thread STINNER Victor


STINNER Victor  added the comment:

See also bpo-45316: [C API] Functions not exported with PyAPI_FUNC().

--
nosy: +vstinner

___
Python tracker 
<https://bugs.python.org/issue45201>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45316] [C API] Functions not exported with PyAPI_FUNC()

2021-09-29 Thread STINNER Victor


New submission from STINNER Victor :

The Python C API contains multiple functions which are not exported with 
PyAPI_FUNC() and so are not usable.

Public functions:
---
void PyLineTable_InitAddressRange(const char *linetable, Py_ssize_t length, int 
firstlineno, PyCodeAddressRange *range);
int PyLineTable_NextAddressRange(PyCodeAddressRange *range);
int PyLineTable_PreviousAddressRange(PyCodeAddressRange *range);
---

=> Either make this functions private ("_Py" prefix) and move them to the 
internal C API, or add PyAPI_FUNC().


Private functions:
---
int _PyCode_InitAddressRange(PyCodeObject* co, PyCodeAddressRange *bounds);
int _PyCode_InitEndAddressRange(PyCodeObject* co, PyCodeAddressRange* bounds);

PyDictKeysObject *_PyDict_NewKeysForClass(void);
Py_ssize_t _PyDict_KeysSize(PyDictKeysObject *keys);
uint32_t _PyDictKeys_GetVersionForCurrentState(PyDictKeysObject *dictkeys);
Py_ssize_t _PyDictKeys_StringLookup(PyDictKeysObject* dictkeys, PyObject *key);

PyObject *_PyDict_Pop_KnownHash(PyObject *, PyObject *, Py_hash_t, PyObject *);
PyObject *_PyDict_FromKeys(PyObject *, PyObject *, PyObject *);
int _PyObjectDict_SetItem(PyTypeObject *tp, PyObject **dictptr, PyObject *name, 
PyObject *value);
PyObject *_PyDict_LoadGlobal(PyDictObject *, PyDictObject *, PyObject *);
Py_ssize_t _PyDict_GetItemHint(PyDictObject *, PyObject *, Py_ssize_t, PyObject 
**);

PyFrameObject* _PyFrame_New_NoTrack(struct _interpreter_frame *, int);
int PySignal_SetWakeupFd(int fd);
uint32_t _PyFunction_GetVersionForCurrentState(PyFunctionObject *func);
PyObject *_PyGen_yf(PyGenObject *);
PyObject *_PyCoro_GetAwaitableIter(PyObject *o);
PyObject *_PyAsyncGenValueWrapperNew(PyObject *);
void _PyArg_Fini(void);
int _Py_CheckPython3(void);
---

=> I suggest to move all these declarations to the internal C API.


Moreover, Include/moduleobject.h contains:
---
#ifdef Py_BUILD_CORE
extern int _PyModule_IsExtension(PyObject *obj);
#endif
---

IMO this declaration should be moved to the internal C API.


See also bpo-45201 about PySignal_SetWakeupFd().

--
components: C API
messages: 402827
nosy: vstinner
priority: normal
severity: normal
status: open
title: [C API] Functions not exported with PyAPI_FUNC()
versions: Python 3.11

___
Python tracker 
<https://bugs.python.org/issue45316>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue36819] Crash during encoding using UTF-16/32 and custom error handler

2021-09-29 Thread STINNER Victor


STINNER Victor  added the comment:

> We could just forbid error handlers returning position not in the range 
> (start , end]

Yeah, that sounds like a reasonable solution. I don't see the point of 
returning a position outside this range. What would be the use case?

For me, the only corner case is the "ignore" error handler which returns an 
empty string, but it returns a position in this range, no?

> it can break some code, so it is better to do this only in a new release.

Implementing custom error handlers is a rare use case, so it should only affect 
a minority of users. Moreover, IMO returning a position outside the valid range 
is a bug. It's common that security fixes change the behavior, like rejecting 
values which were previously acceptd, to prevent a Python crash.

--

___
Python tracker 
<https://bugs.python.org/issue36819>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45308] multiprocessing default fork child process will not free object, which is inherited from parent process

2021-09-29 Thread STINNER Victor


STINNER Victor  added the comment:

> I still have some questions about is

The Python bug tracker is not a forum for asking help how to use Python. Please 
stop reopening this issue.

--
resolution:  -> not a bug
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue45308>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue1170] shlex have problems with parsing unicode

2021-09-29 Thread STINNER Victor


STINNER Victor  added the comment:

> I would like to suggest making this change (or something similar) to the 
> official version of "shlex.py".  Would sending an email to 
> "python-id...@python.org" be a good place to make this proposal?

Yes, python-ideas is a good place to start discussion such idea. This issue is 
closed, if you discuss it here, you will get a limited audience.

--

___
Python tracker 
<https://bugs.python.org/issue1170>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42972] [C API] Heap types (PyType_FromSpec) must fully implement the GC protocol

2021-09-28 Thread STINNER Victor


Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg394832

___
Python tracker 
<https://bugs.python.org/issue42972>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45307] Removal of _PyImport_FindExtensionObject() in 3.10 limits custom extension module loaders

2021-09-28 Thread STINNER Victor


STINNER Victor  added the comment:

> Until 3.10 makes equivalent functionality available or another workaround is 
> supported,

I don't understand which long term solution do you propose.

> a properly designed public API is probably a better solution

Can you propose a public API?

--

___
Python tracker 
<https://bugs.python.org/issue45307>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45310] test_multiprocessing_forkserver: test_shared_memory_basics() failed with FileExistsError: [Errno 17] File exists: '/test01_tsmb'

2021-09-28 Thread STINNER Victor


New submission from STINNER Victor :

AMD64 Fedora Stable LTO + PGO 3.10 (build 357):
https://buildbot.python.org/all/#/builders/651/builds/357

First test_multiprocessing_forkserver failed, then test_multiprocessing_spawn, 
then test_multiprocessing_fork.

I confirm that these tests fail if the /test01_tsmb shared memory exists.

The test is supposed to unlink the shared memory once the test completes:

def test_shared_memory_basics(self):
sms = shared_memory.SharedMemory('test01_tsmb', create=True, size=512)
self.addCleanup(sms.unlink)

Maybe the whole process was killed while the test was running.

I removed it manually:

sudo ./python -c "import _posixshmem; 
_posixshmem.shm_unlink('/test01_tsmb')"

Logs:

Traceback (most recent call last):
  File 
"/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/resource_tracker.py",
 line 209, in main
cache[rtype].remove(name)
KeyError: '/psm_49a93592'
(...)
==
ERROR: test_shared_memory_basics 
(test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory)
--
Traceback (most recent call last):
  File 
"/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/test/_test_multiprocessing.py",
 line 3777, in test_shared_memory_basics
sms = shared_memory.SharedMemory('test01_tsmb', create=True, size=512)
  File 
"/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/shared_memory.py",
 line 103, in __init__
self._fd = _posixshmem.shm_open(
FileExistsError: [Errno 17] File exists: '/test01_tsmb'

--
components: Tests
messages: 402770
nosy: vstinner
priority: normal
severity: normal
status: open
title: test_multiprocessing_forkserver: test_shared_memory_basics() failed with 
FileExistsError: [Errno 17] File exists: '/test01_tsmb'
versions: Python 3.11

___
Python tracker 
<https://bugs.python.org/issue45310>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45308] multiprocessing default fork child process will not free object, which is inherited from parent process

2021-09-28 Thread STINNER Victor


STINNER Victor  added the comment:

That's the principle of fork. You should use a different spawn method if you 
don't want to inherit resources inherited from the parent process, like spawn 
or forkserver:
https://docs.python.org/dev/library/multiprocessing.html#contexts-and-start-methods

--
nosy: +vstinner
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue45308>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue1596321] KeyError at exit after 'import threading' in other thread

2021-09-27 Thread STINNER Victor


STINNER Victor  added the comment:

Better late than never. I only took 15 years to fix this old bug :-D

--
versions: +Python 3.10, Python 3.11, Python 3.9 -Python 2.7, Python 3.2

___
Python tracker 
<https://bugs.python.org/issue1596321>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue1596321] KeyError at exit after 'import threading' in other thread

2021-09-27 Thread STINNER Victor


STINNER Victor  added the comment:


New changeset 94d19f606fa18a1c4d2faca1caf2f470a8ce6d46 by Victor Stinner in 
branch '3.9':
bpo-1596321: Fix threading._shutdown() for the main thread (GH-28549) (GH-28589)
https://github.com/python/cpython/commit/94d19f606fa18a1c4d2faca1caf2f470a8ce6d46


--

___
Python tracker 
<https://bugs.python.org/issue1596321>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45301] pycore_condvar.h: remove Windows conditonal variable emulation, use Windows native conditional variable

2021-09-27 Thread STINNER Victor


STINNER Victor  added the comment:

Eryk:
> SleepConditionVariableSRW() can't be interrupted in PyCOND_WAIT() and 
> PyCOND_TIMEDWAIT().

Oh. A comment on StackOverlow says: 

"The WaitForXxx functions accept parameters of the generic HANDLE type, which 
represents a handle to a kernel object. Condition variables are user-mode 
objects, not kernel objects, so you cannot use them with these functions, since 
they work only with kernel objects."

https://stackoverflow.com/questions/37522108/can-you-really-wait-on-condition-variable-with-waitfor-objects

I only created this issue because of this comment in pycore_condvar.h:
---
/* non-emulated condition variables are provided for those that want
 * to target Windows Vista.  Modify this macro to enable them.
 */
#ifndef _PY_EMULATED_WIN_CV
#define _PY_EMULATED_WIN_CV 1  /* use emulated condition variables */
#endif
---

--
title: pycore_condvar.h: remove Windows conditonal variable emulation -> 
pycore_condvar.h: remove Windows conditonal variable emulation, use Windows 
native conditional variable

___
Python tracker 
<https://bugs.python.org/issue45301>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue1596321] KeyError at exit after 'import threading' in other thread

2021-09-27 Thread STINNER Victor


Change by STINNER Victor :


--
pull_requests: +26971
pull_request: https://github.com/python/cpython/pull/28589

___
Python tracker 
<https://bugs.python.org/issue1596321>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue1596321] KeyError at exit after 'import threading' in other thread

2021-09-27 Thread STINNER Victor


STINNER Victor  added the comment:


New changeset 95d31370829b7d729667588e0a9943217401ea5b by Victor Stinner in 
branch 'main':
bpo-1596321: Fix threading._shutdown() for the main thread (GH-28549)
https://github.com/python/cpython/commit/95d31370829b7d729667588e0a9943217401ea5b


--

___
Python tracker 
<https://bugs.python.org/issue1596321>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29971] Lock.acquire() not interruptible on Windows

2021-09-27 Thread STINNER Victor


STINNER Victor  added the comment:

time.sleep() is now implemented with a waitable timer and 
WaitForMultipleObjects() on Windows. It uses _PyOS_SigintEvent() so it can be 
interrupted with CTRL+C.

commit 58f8adfda3c2b42f654a55500e8e3a6433cb95f2
Author: Victor Stinner 
Date:   Wed Sep 22 16:09:30 2021 +0200

bpo-21302: time.sleep() uses waitable timer on Windows (GH-28483)

On Windows, time.sleep() now uses a waitable timer which has a
resolution of 100 ns (10^-7 sec). Previously, it had a solution of 1
ms (10^-3 sec).

* On Windows, time.sleep() now calls PyErr_CheckSignals() before
  resetting the SIGINT event.
* Add _PyTime_As100Nanoseconds() function.
* Complete and update time.sleep() documentation.

Co-authored-by: Livius 

--

___
Python tracker 
<https://bugs.python.org/issue29971>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45301] pycore_condvar.h: remove Windows conditonal variable emulation

2021-09-27 Thread STINNER Victor


STINNER Victor  added the comment:

See also bpo-29971 "Lock.acquire() not interruptible on Windows".

--

___
Python tracker 
<https://bugs.python.org/issue45301>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29971] Lock.acquire() not interruptible on Windows

2021-09-27 Thread STINNER Victor


STINNER Victor  added the comment:

See also bpo-45301 "pycore_condvar.h: remove Windows conditonal variable 
emulation".

--

___
Python tracker 
<https://bugs.python.org/issue29971>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45301] pycore_condvar.h: remove Windows conditonal variable emulation

2021-09-27 Thread STINNER Victor


New submission from STINNER Victor :

I recently worked on time.sleep() enhancement (bpo-21302) and threading 
bugfixes (bpo-45274, bpo-1596321). I saw one more time that Python emulates 
conditional variables to support Windows XP and older. But Python 3.11 requires 
Windows 8.1 or newer. IMO it's time to remove _PY_EMULATED_WIN_CV code path 
from pycore_condvar.h.

--
components: Library (Lib)
messages: 402720
nosy: paul.moore, steve.dower, tim.golden, vstinner, zach.ware
priority: normal
severity: normal
status: open
title: pycore_condvar.h: remove Windows conditonal variable emulation
versions: Python 3.11

___
Python tracker 
<https://bugs.python.org/issue45301>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29971] Lock.acquire() not interruptible on Windows

2021-09-27 Thread STINNER Victor


STINNER Victor  added the comment:

See also bpo-21822: some comments are about this issue. It's a duplicate of 
bpo-45274 "Race condition in Thread._wait_for_tstate_lock()".

--

___
Python tracker 
<https://bugs.python.org/issue29971>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45274] Race condition in Thread._wait_for_tstate_lock()

2021-09-27 Thread STINNER Victor


STINNER Victor  added the comment:

> This is the same as bpo-21822, so I suppose it should be closed as well with 
> a reference to this issue.

You're right. I marked bpo-21822 as a duplicate of this issue.

--

___
Python tracker 
<https://bugs.python.org/issue45274>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue21822] [Windows] KeyboardInterrupt during Thread.join hangs that Thread

2021-09-27 Thread STINNER Victor


STINNER Victor  added the comment:

I mark this issue as a duplicate of bpo-45274.

--

I fixed bpo-45274 with this change:

New changeset a22be4943c119fecf5433d999227ff78fc2e5741 by Victor Stinner in 
branch 'main':
bpo-45274: Fix Thread._wait_for_tstate_lock() race condition (GH-28532)
https://github.com/python/cpython/commit/a22be4943c119fecf5433d999227ff78fc2e5741

I tested join.py with the fix. It nows displays:
---
vstinner@DESKTOP-DK7VBIL C:\vstinner\python\main>python x.py
Running Debug|x64 interpreter...
started. Press Ctrl-C now
Ctrl-C [2.99]  done=True  alive=False
finish [2.99]  done=True  alive=False
Terminate batch job (Y/N)? n
---

The script no longer hangs.

--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> Race condition in Thread._wait_for_tstate_lock()

___
Python tracker 
<https://bugs.python.org/issue21822>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue21822] [Windows] KeyboardInterrupt during Thread.join hangs that Thread

2021-09-27 Thread STINNER Victor


STINNER Victor  added the comment:

> I am having the same blocked signal issue on Windows when using Thread.join. 
> This program does not print "interrupted" after pressing Ctrl+C:

This is a different issue: bpo-29971. Currently, threading.Lock.acquire() 
cannot be interrupted by CTRL+C.

--
nosy: +vstinner

___
Python tracker 
<https://bugs.python.org/issue21822>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29971] Lock.acquire() not interruptible on Windows

2021-09-27 Thread STINNER Victor


STINNER Victor  added the comment:

Copy of Antoine Pitrou's msg316024 (bpo-21822):

multiprocessing semaphores support Ctrl-C under Windows, so it should be doable 
for regular locks as well (notice the `sigint_event`):
https://github.com/python/cpython/blob/master/Modules/_multiprocessing/semaphore.c#L109-L146

--
nosy: +vstinner

___
Python tracker 
<https://bugs.python.org/issue29971>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45300] Building Python documentation with doctest logs a ResourceWarning in Doc/library/nntplib.rst

2021-09-27 Thread STINNER Victor


New submission from STINNER Victor :

Build Python documentatioin with:

make -C Doc/ PYTHON=../python venv
LANG= PYTHONTRACEMALLOC=10 make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W 
-j10" doctest 2>&1|tee log

See the logs:

/home/vstinner/python/main/Lib/socket.py:776: ResourceWarning: unclosed 

  self._sock = None
Object allocated at (most recent call last):
  File 
"/home/vstinner/python/main/Doc/venv/lib/python3.11/site-packages/sphinx/builders/__init__.py",
 lineno 361
self.write(docnames, list(updated_docnames), method)
  File 
"/home/vstinner/python/main/Doc/venv/lib/python3.11/site-packages/sphinx/ext/doctest.py",
 lineno 366
self.test_doc(docname, doctree)
  File 
"/home/vstinner/python/main/Doc/venv/lib/python3.11/site-packages/sphinx/ext/doctest.py",
 lineno 470
self.test_group(group)
  File 
"/home/vstinner/python/main/Doc/venv/lib/python3.11/site-packages/sphinx/ext/doctest.py",
 lineno 554
self.test_runner.run(test, out=self._warn_out, clear_globs=False)
  File "/home/vstinner/python/main/Lib/doctest.py", lineno 1495
return self.__run(test, compileflags, out)
  File "/home/vstinner/python/main/Lib/doctest.py", lineno 1348
exec(compile(example.source, filename, "single",
  File "", lineno 1
  File "/home/vstinner/python/main/Lib/nntplib.py", lineno 334
self.sock = self._create_socket(timeout)
  File "/home/vstinner/python/main/Lib/nntplib.py", lineno 399
return socket.create_connection((self.host, self.port), timeout)
  File "/home/vstinner/python/main/Lib/socket.py", lineno 828
sock = socket(af, socktype, proto)

--
assignee: docs@python
components: Documentation
messages: 402709
nosy: docs@python, vstinner
priority: normal
severity: normal
status: open
title: Building Python documentation with doctest logs a ResourceWarning in 
Doc/library/nntplib.rst
versions: Python 3.11

___
Python tracker 
<https://bugs.python.org/issue45300>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45274] Race condition in Thread._wait_for_tstate_lock()

2021-09-27 Thread STINNER Victor


Change by STINNER Victor :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.10, Python 3.9

___
Python tracker 
<https://bugs.python.org/issue45274>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45274] Race condition in Thread._wait_for_tstate_lock()

2021-09-27 Thread STINNER Victor


STINNER Victor  added the comment:


New changeset fae2694bea5e9e5a114af8cb40b60e7131a6340c by Miss Islington (bot) 
in branch '3.10':
bpo-45274: Fix Thread._wait_for_tstate_lock() race condition (GH-28532) 
(GH-28580)
https://github.com/python/cpython/commit/fae2694bea5e9e5a114af8cb40b60e7131a6340c


--

___
Python tracker 
<https://bugs.python.org/issue45274>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45274] Race condition in Thread._wait_for_tstate_lock()

2021-09-27 Thread STINNER Victor


STINNER Victor  added the comment:


New changeset a22be4943c119fecf5433d999227ff78fc2e5741 by Victor Stinner in 
branch 'main':
bpo-45274: Fix Thread._wait_for_tstate_lock() race condition (GH-28532)
https://github.com/python/cpython/commit/a22be4943c119fecf5433d999227ff78fc2e5741


--

___
Python tracker 
<https://bugs.python.org/issue45274>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45261] Unreliable (?) results from timeit (cache issue?)

2021-09-27 Thread STINNER Victor


Change by STINNER Victor :


--
resolution:  -> not a bug

___
Python tracker 
<https://bugs.python.org/issue45261>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue36717] Allow retrieval of return value from the target of a threading.Thread

2021-09-26 Thread STINNER Victor


STINNER Victor  added the comment:

Storing the result of Thread.run() can keep Python objects alive longer than 
expected and it may require a lock on it. I don't think that the additional 
complexity is worth it. I suggest to reorganize your code to pass the result 
differently to the caller thread. You can use a queue, a variable, whatever 
works.

--
nosy: +vstinner
resolution:  -> wont fix
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue36717>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue27978] concurrent.futures.threading: Add a timeout to Executor shutdown() method

2021-09-26 Thread STINNER Victor


Change by STINNER Victor :


--
title: [threading] Executor#shutdown with timeout -> 
concurrent.futures.threading: Add a timeout to Executor shutdown() method

___
Python tracker 
<https://bugs.python.org/issue27978>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue1736792] dict reentrant/threading request

2021-09-26 Thread STINNER Victor


STINNER Victor  added the comment:

> As I first mentioned in my post[1] on comp.lang.python, it seems possible to 
> modify a dict while a lookup is ongoing, without causing the lookup to 
> restart.

That's a bad practice. Python dict raises an exception in some cases: 
RuntimeError("dict mutated during update").

Detecting any change during a lookup would slow down, whereas dict performance 
is key in Python performance in general, since dict is used everywhere for 
Python namespaces.

I close the issue as "wont fix".

--
nosy: +vstinner
resolution:  -> wont fix
stage: test needed -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue1736792>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45292] Implement PEP 654: Exception Groups

2021-09-26 Thread STINNER Victor


Change by STINNER Victor :


--
title: Implement PEP 654 -> Implement PEP 654: Exception Groups

___
Python tracker 
<https://bugs.python.org/issue45292>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45020] Freeze all modules imported during startup.

2021-09-25 Thread STINNER Victor


STINNER Victor  added the comment:

Marc-Andre: I suppose that you're talking about LANDMARK in Modules/getpath.c 
and PC/getpathp.c.

--

___
Python tracker 
<https://bugs.python.org/issue45020>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux

2021-09-25 Thread STINNER Victor


STINNER Victor  added the comment:

> Do you have any information about when will be it released in 3.11?

Here is a schedule of Python 3.11 releases:
https://www.python.org/dev/peps/pep-0664/

In the meanwhile, you can develop a C extension to get the feature.

--

___
Python tracker 
<https://bugs.python.org/issue21302>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux

2021-09-25 Thread STINNER Victor


STINNER Victor  added the comment:

Thanks Livius for all these nice enhancements!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue21302>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux

2021-09-25 Thread STINNER Victor


STINNER Victor  added the comment:


New changeset 7834ff26cbcd4d8394d64d80d9f51a364d38b1c6 by Victor Stinner in 
branch 'main':
bpo-21302: Add nanosleep() implementation for time.sleep() in Unix (GH-28545)
https://github.com/python/cpython/commit/7834ff26cbcd4d8394d64d80d9f51a364d38b1c6


--

___
Python tracker 
<https://bugs.python.org/issue21302>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45274] Race condition in Thread._wait_for_tstate_lock()

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:

> I am not sure that it can be solved at Python level.

Right. In pure Python, we cannot write code which works in all cases. My PR 
28532 fix the most common case: application interrupted by a single CTRL+C.

> with suppress_interrupt(): (...)

It's important to be able to interrupt acquire() which can be called in 
blocking mode with no timeout: it's exactly what tox does, and users expect to 
be able to interrupt tox in this case.

> * Add a Lock method (or just a builtin function) which acquires and 
> immediately releases the lock, without possibility to interrupt.

The acquire()+release() sequence can be made atomic in C, but it doesn't solve 
the problem of _stop() which can be interrupted by a second exception.

This bug is likely as old as Python. I don't think that we should attempt to 
design a perfect solution. I only propose to make the race condition (way) less 
likely.

--

___
Python tracker 
<https://bugs.python.org/issue45274>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41299] Python3 threading.Event().wait time is twice as large as Python27

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:

I close the issue.

> bpo-41299: Fix EnterNonRecursiveMutex() (GH-28548)

I don't think that it's worth it to backport this change to 3.9 and 3.10. In 
practice, Py_FatalError() will never be called, and _PyTime_GetMonotonicClock() 
result must always be greater than 0.

But I wanted to remove it in the main branch to reduce the number of functions 
calling Py_FatalError().

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue41299>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41299] Python3 threading.Event().wait time is twice as large as Python27

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:


New changeset 6bc89116cb121c176453b9745e3daa2cc5df43ab by Victor Stinner in 
branch 'main':
bpo-41299: Mark private thread_nt.h functions as static (GH-28553)
https://github.com/python/cpython/commit/6bc89116cb121c176453b9745e3daa2cc5df43ab


--

___
Python tracker 
<https://bugs.python.org/issue41299>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing)

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:

Analysis use use_tracing usage in 3rd part code.

I see two main ways to add C API functions covering these use cases:

* Provide high-level functions like "call a trace function" (disable tracing, 
call trace function, reenable tracing, increment/decrement tstate->tracing)
* Provide low-level functions just to control use_tracing: make PyThreadState 
structure opaque, but stil make the assumption that it is possible to disable 
temporarily tracing and profiling (in practice, it's implemented as 
use_tracing=0).



(*) greenlet

greenlet disables temporarily tracing in g_calltrace(), and then restore it, to 
call a "tracing" function:
---
tstate->tracing++;
TSTATE_USE_TRACING(tstate) = 0;
retval = PyObject_CallFunction(tracefunc, "O(OO)", event, origin, target);
tstate->tracing--;
TSTATE_USE_TRACING(tstate) =
(tstate->tracing <= 0 &&
 ((tstate->c_tracefunc != NULL) || (tstate->c_profilefunc != NULL)));
---

It also saves and then restores use_tracing value:
---
ts__g_switchstack_use_tracing = tstate->cframe->use_tracing;
(...)
tstate->cframe->use_tracing = ts__g_switchstack_use_tracing;
---

=> it can use PyThreadState_IsTracing(), PyThreadState_DisableTracing() and 
PyThreadState_ResetTracing().

These functions don't handle "tstate->tracing++;" and "tstate->tracing--;" 
which is also used by greenlet.

greenlet also saves and restores tstate->cframe:
https://github.com/python-greenlet/greenlet/blob/master/src/greenlet/greenlet.c


(*) dipy

Code generated by Cython.


(*) smartcols

Code generated by Cython.


(*) yappi

yappi is Python profiler.

yappi sets use_tracing to 1 when it sets its profile function: 
"ts->c_profilefunc = _yapp_callback;".

It sets use_tracing to 0 when it clears the profile function: 
"ts->c_profilefunc = NULL;". That's wrong, it ignores the trace function.

PyEval_SetProfile() cannot be used because yappi works on a PyThreadState (ts).

Code: https://github.com/sumerc/yappi/blob/master/yappi/_yappi.c

It can use PyThreadState_DisableTracing() and PyThreadState_ResetTracing(). 
Maybe a PyThreadState_SetProfile(tstate, func) function would fit better 
yappi's use case.


(*) Cython

Cython defines 2 compatibility functions:

* __Pyx_IsTracing(tstate, check_tracing, check_funcs): it can check 
c_profilefunc and c_tracefunc
* __Pyx_SetTracing(tstate, enable)

Code: https://github.com/cython/cython/blob/0.29.x/Cython/Utility/Profile.c

The code is quite complicated. In short, it checks if tracing and/or profiling 
is enabled. If it's enabled, it disables temporarily tracing (use_tracing=0) 
while calling trace and profile functions.

=> it requires PyThreadState_IsTracing(), PyThreadState_DisableTracing() and 
PyThreadState_ResetTracing().

--

___
Python tracker 
<https://bugs.python.org/issue43760>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41299] Python3 threading.Event().wait time is twice as large as Python27

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:


New changeset d639e3170506a1a80c4b2f1b3a6250bb95ff190a by Victor Stinner in 
branch 'main':
bpo-41299: Fix EnterNonRecursiveMutex() (GH-28548)
https://github.com/python/cpython/commit/d639e3170506a1a80c4b2f1b3a6250bb95ff190a


--

___
Python tracker 
<https://bugs.python.org/issue41299>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41299] Python3 threading.Event().wait time is twice as large as Python27

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:


New changeset f35ddf24227e834c9b6b39ad23a0ec382b4de48b by Victor Stinner in 
branch 'main':
bpo-41299: QueryPerformanceFrequency() cannot fail (GH-28552)
https://github.com/python/cpython/commit/f35ddf24227e834c9b6b39ad23a0ec382b4de48b


--

___
Python tracker 
<https://bugs.python.org/issue41299>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41299] Python3 threading.Event().wait time is twice as large as Python27

2021-09-24 Thread STINNER Victor


Change by STINNER Victor :


--
pull_requests: +26936
pull_request: https://github.com/python/cpython/pull/28553

___
Python tracker 
<https://bugs.python.org/issue41299>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41299] Python3 threading.Event().wait time is twice as large as Python27

2021-09-24 Thread STINNER Victor


Change by STINNER Victor :


--
pull_requests: +26935
pull_request: https://github.com/python/cpython/pull/28552

___
Python tracker 
<https://bugs.python.org/issue41299>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40699] Memory leak in threading library with Python 3.6-3.8

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:

I fail to reproduce the leak using attached script.

I close the issue.

I get constant memory usage on Linux with the main branch of Python (future 
3.11):

(...)
VmRSS: 11344 kB
VmRSS: 11344 kB
VmRSS: 11344 kB
VmRSS: 11344 kB
VmRSS: 11344 kB
VmRSS: 11344 kB
VmRSS: 11344 kB
VmRSS: 11344 kB
VmRSS: 11344 kB
VmRSS: 11344 kB
(...)

--
nosy: +vstinner
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed
Added file: https://bugs.python.org/file50304/script.py

___
Python tracker 
<https://bugs.python.org/issue40699>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40657] Resource leaks with threading.Thread

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:

You must join thread using thread.join().

If I add .join() calls, threading.active_count() stays around 2, it doesn't 
grow.

It's not a bug in Python.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue40657>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue43050] threading timer memory leak

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:

I cannot reproduce the issue. IMO it has been fixed.

Moreover, you must join timers using timer.join(): a timer remains a thread.

Code:
---
import os
import threading
os.system(f"grep ^VmRSS /proc/{os.getpid()}/status")
# warmup
for n in range(10):
timer = threading.Timer(5, None)
timer.start()
timer.cancel()
timer.join()
os.system(f"grep ^VmRSS /proc/{os.getpid()}/status")
for n in range(1000):
timer = threading.Timer(5, None)
timer.start()
timer.cancel()
timer.join()
os.system(f"grep ^VmRSS /proc/{os.getpid()}/status")
---

Output on Linux with the main branch of Python (3.11):
---
VmRSS: 10924 kB
VmRSS: 11104 kB
VmRSS: 11104 kB
---

--
nosy: +vstinner
resolution: duplicate -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue43050>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue1596321] KeyError at exit after 'import threading' in other thread

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:

I proposed PR 28549 to fix this very old threading issue.

A C extension can spawn threads without using the threading module and then 
then run Python code which imports the threading module. In this case, 
threading._main_thread is the thread which imported first the threading module. 
My PR changes so threading._shutdown() simply ignores _main_thread when it 
happens.

--

___
Python tracker 
<https://bugs.python.org/issue1596321>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue1596321] KeyError at exit after 'import threading' in other thread

2021-09-24 Thread STINNER Victor


Change by STINNER Victor :


--
keywords: +patch
nosy: +vstinner
nosy_count: 14.0 -> 15.0
pull_requests: +26932
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/28549

___
Python tracker 
<https://bugs.python.org/issue1596321>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41299] Python3 threading.Event().wait time is twice as large as Python27

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:

Calling Py_FatalError() must be avoided by any mean. I spent significant time 
to remove Py_FatalError() at Python startup in my PEP 587 implementation: 
https://www.python.org/dev/peps/pep-0587/

I also removed Py_FatalError() calls to report errors to the caller: 
https://bugs.python.org/issue38631

Calling Py_FatalError() is really bad when Python is embedded in an 
application. It kills the whole process, the caller has no way to catch the 
error.

--

___
Python tracker 
<https://bugs.python.org/issue41299>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41299] Python3 threading.Event().wait time is twice as large as Python27

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:

I created PR #28548 to remove the Py_FatalError() code path.

--

___
Python tracker 
<https://bugs.python.org/issue41299>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41299] Python3 threading.Event().wait time is twice as large as Python27

2021-09-24 Thread STINNER Victor


Change by STINNER Victor :


--
pull_requests: +26931
pull_request: https://github.com/python/cpython/pull/28548

___
Python tracker 
<https://bugs.python.org/issue41299>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42969] pthread_exit & PyThread_exit_thread from PyEval_RestoreThread etc. are harmful

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:

Jeremy Maitin-Shepard: "In general, I view hanging threads as the least bad 
thing to do when faced with either acquiring the GIL or not returning at all.  
There is a lot of existing usage of Python that currently poses a risk of 
random crashes and memory corruption while Python is exiting, and I would like 
to fix that."

Showing warnings by default or not was discussed many times in Python. It was 
decided to *hide* DeprecationWarning by default. The PEP 565 is a minor 
trade-off to show them in the __main__ module.

For me, more generally, Python default behavior is designed for *users* who 
don't want to be annoyed by warnings or anything which would make sense for 
*developers*. That's why I designed a new "Python Development Mode" (-X dev):
https://docs.python.org/dev/library/devmode.html

Maybe in development mode, the behavior could be changed to call abort(). But 
honestly, I'm not really excited by that. I'm not embedding Python in a C++ 
application. I'm almot only use Python directly: the Unix command "python3". 
For this use case, IMO it's fine to call pthread_exit() by default.

--

___
Python tracker 
<https://bugs.python.org/issue42969>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42969] pthread_exit & PyThread_exit_thread from PyEval_RestoreThread etc. are harmful

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:

Another example where a developer asks to call abort() to notice bugs, whereas 
Python previously silently ignored it: bpo-36829. Calling abort() is a legit 
use case, but not really the best default behavior. Again, the problem was 
solved by letting developers setting their own callback: sys.unraisablehook.

If I understood correctly, pytest doesn't override it but "took" into the 
default implementation: it chains its own code with the default implementation. 
It's possible because there is a way to "get" the current hook: just read 
sys.unraisablehook ;-)

Another argument in favor of also adding Py_GetThreadExitCallback() ;-)

--

___
Python tracker 
<https://bugs.python.org/issue42969>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42969] pthread_exit & PyThread_exit_thread from PyEval_RestoreThread etc. are harmful

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:

Gregory P. Smith: Python has many API using callbacks:

PEP 445 added PyMem_SetAllocator() to set memory allocator. Adding 
PyMem_GetAllocator() also made possible to chain allocators and to "hook" into 
an existing allocator to execute code before and after it's called (the PEP 
contains an example).

I'm not sure if it's important or useless to chain callbacks with 
Py_SetThreadExitCallback(). I suggest to always override the previously set 
callback.

It would matter if library A sets a callback to emit log and library B sets a 
callback to hang threads. It maybe be nice to first emit a log and then hang 
the thread. But then the order in which callbacks are set starts to matter a 
lot :-)

I'm fine with adding Py_GetThreadExitCallback() if you consider that it matters.


> If someone passes nullptr does that undo it (please no!).

I don't think that we should bother with adding a special case. I prefer to 
consider developers as adults and let them make their own mistakes if they 
consider that they understand the code well enough ;-)

_PyEval_SetTrace() allows to remove the current trace function. It's a legit 
use case.

If library C is annoyed by library A and library B installed annoying 
callbacks, IMO it's also ok to let it "remove" the previously set callback, no?

IMO Py_SetThreadExitCallback(NULL) should simply set the callback to NULL, so 
restore the default behavior: call pthread_exit().

--

___
Python tracker 
<https://bugs.python.org/issue42969>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42969] pthread_exit & PyThread_exit_thread from PyEval_RestoreThread etc. are harmful

2021-09-24 Thread STINNER Victor


STINNER Victor  added the comment:

> In this case that unexpected thread simply disappearing can lead to a 
> deadlock in our process.

This problem also remains me the very complex case of bpo-6721: "Locks in the 
standard library should be sanitized on fork". The issue title looks simple, 
but 12 years after the issue was created, it's still open.

This issue is being solved by adding atfork callbacks to modules which must do 
something at fork in the child process (sometimes also in the parent process).

I added threading.Lock._at_fork_reinit() private method to simplify the 
implementation of these callbacks.

Such problem has no silver bullet solution, so it's better to let developers 
design their own solution with their specific requirements.

--

___
Python tracker 
<https://bugs.python.org/issue42969>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux

2021-09-24 Thread STINNER Victor


Change by STINNER Victor :


--
pull_requests: +26929
pull_request: https://github.com/python/cpython/pull/28545

___
Python tracker 
<https://bugs.python.org/issue21302>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing)

2021-09-24 Thread STINNER Victor


Change by STINNER Victor :


--
pull_requests: +26926
pull_request: https://github.com/python/cpython/pull/28542

___
Python tracker 
<https://bugs.python.org/issue43760>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45274] Race condition in Thread._wait_for_tstate_lock()

2021-09-23 Thread STINNER Victor


Change by STINNER Victor :


--
nosy: +pablogsal
stage: patch review -> 

___
Python tracker 
<https://bugs.python.org/issue45274>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45274] Race condition in Thread._wait_for_tstate_lock()

2021-09-23 Thread STINNER Victor


Change by STINNER Victor :


--
pull_requests: +26920
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/28532

___
Python tracker 
<https://bugs.python.org/issue45274>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45274] Race condition in Thread._wait_for_tstate_lock()

2021-09-23 Thread STINNER Victor


STINNER Victor  added the comment:

For curious people, see also bpo-44422 "Fix threading.enumerate() reentrant 
call": another example of race condition recently fixed in the threading 
module. But it's unrelated to this bug ;-)

--

___
Python tracker 
<https://bugs.python.org/issue45274>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45274] Race condition in Thread._wait_for_tstate_lock()

2021-09-23 Thread STINNER Victor


Change by STINNER Victor :


Added file: https://bugs.python.org/file50300/threading_bug.py

___
Python tracker 
<https://bugs.python.org/issue45274>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45274] Race condition in Thread._wait_for_tstate_lock()

2021-09-23 Thread STINNER Victor

New submission from STINNER Victor :

Bernát Gábor found an interesting bug on Windows. Sometimes, when a process is 
interrupted on Windows with CTRL+C, its parent process hangs in thread.join():
https://twitter.com/gjbernat/status/1440949682759426050

Reproducer:

* Install https://github.com/gaborbernat/tox/tree/2159
* Make an empty folder and put the above tox.ini in it
* Invoke python -m tox and once that command is sleeping press CTRL+C (the app 
should lock there indefinitely).

tox.ini:
---

[testenv]
skip_install = true
commands = python -c 'import os; print(f"start {os.getpid()}"); import time; 
time.sleep(100); print("end");'
---

Source: https://gist.github.com/gaborbernat/f1e1aff0f2ee514b50a92a4d019d4d13

I tried to attach the Python process in Python: there is a single thread, the 
main thread which is blocked in thread.join(). You can also see it in the 
faulthandler traceback.

I did a long analysis of the _tstate_lock and I checked that thread really 
completed. Raw debug traces:

== thread 6200 exit ==
 
thread_run[pid=3984, thread_id=6200]: clear
PyThreadState_Clear[pid=3984, thread_id=6200]: on_delete()
release_sentinel[pid=3984, thread_id=6200]: enter
release_sentinel[pid=3984, thread_id=6200]: release(obj=01C1122669C0, 
lock=01C110BBDA00)
release_sentinel[pid=3984, thread_id=6200]: exit
PyThreadState_Clear[pid=3984, thread_id=6200]: on_delete()--
 
== main thread is calling join() but gets a KeyboardInterrupt ==
 
[pid=3984, thread_id=8000] Lock.acquire() -> ACQUIRED
Current thread 0x1f40 (most recent call first):
  File "C:\vstinner\python\3.10\lib\threading.py", line 1118 in 
_wait_for_tstate_lock
  File "C:\vstinner\python\3.10\lib\threading.py", line 1092 in join
  File "C:\vstinner\env\lib\site-packages\tox\session\cmthread_run[pid=3984, 
thread_id=6200]: exit
d\run\common.py", line 203 in execute
  File "C:\vstinner\env\lib\site-packages\tox\session\cmd\run\sequential.py", 
line 20 in run_sequential
  File "C:\vstinner\env\lib\site-packages\tox\session\cmd\legacy.py", line 104 
in legacy
  File "C:\vstinner\env\lib\site-packages\tox\run.py", line 49 in main
  File "C:\vstinner\env\lib\site-packages\tox\run.py", line 23 in run
  File "C:\vstinner\env\lib\site-packages\tox\__main__.py", line 4 in 
  File "C:\vstinner\python\3.10\lib\runpy.py", line 86 in _run_code
  File "C:\vstinner\python\3.10\lib\runpy.py", line 196 in _run_module_as_main
_wait_for_tstate_lock[pid=3984, current thread_id=8000, self thread_id=6200]: 
EXC: KeyboardInterrupt(); acquired? None
 
== main thread calls repr(thread) ==
 
ROOT: [3984] KeyboardInterrupt - teardown started
_wait_for_tstate_lock[pid=3984, current thread_id=8000, self thread_id=6200]: 
acquire(block=False, timeout=-1): lock obj= 0x1c1122669c0
  File "C:\vstinner\python\3.10\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
  File "C:\vstinner\python\3.10\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
  File "C:\vstinner\env\lib\site-packages\tox\__main__.py", line 4, in 
run()
  File "C:\vstinner\env\lib\site-packages\tox\run.py", line 23, in run
result = main(sys.argv[1:] if args is None else args)
  File "C:\vstinner\env\lib\site-packages\tox\run.py", line 49, in main
result = handler(state)
  File "C:\vstinner\env\lib\site-packages\tox\session\cmd\legacy.py", line 104, 
in legacy
return run_sequential(state)
  File "C:\vstinner\env\lib\site-packages\tox\session\cmd\run\sequential.py", 
line 20, in run_sequential
return execute(state, max_workers=1, has_spinner=False, live=True)
  File "C:\vstinner\env\lib\site-packages\tox\session\cmd\run\common.py", line 
217, in execute
print(f'join {thread}')
  File "C:\vstinner\python\3.10\lib\threading.py", line 901, in __repr__
self.is_alive() # easy way to get ._is_stopped set when appropriate
  File "C:\vstinner\python\3.10\lib\threading.py", line 1181, in is_alive
self._wait_for_tstate_lock(False)
  File "C:\vstinner\python\3.10\lib\threading.py", line 1113, in 
_wait_for_tstate_lock
traceback.print_stack()
_wait_for_tstate_lock[pid=3984, current thread_id=8000, self thread_id=6200]: 
failed to acquire 0x1c1122669c0
_wait_for_tstate_lock[pid=3984, current thread_id=8000, self thread_id=6200]: 
exit 0x1c1122669c0
join 
 
== main thread calls thread.join()... which hangs ==
 
_wait_for_tstate_lock[pid=3984, current thread_id=8000, self thread_id=6200]: 
acquire(block=True, timeout=-1): lock obj= 0x1c1122669c0
  File "C:\vstinner\python\3.10\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
  File "C:\vstinner\python\3.10\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
  F

[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing)

2021-09-23 Thread STINNER Victor


STINNER Victor  added the comment:


New changeset f4ccb79d52ee726d58bbb038ea98b4deec52001d by Victor Stinner in 
branch 'main':
bpo-43760: Document PyThreadState.use_tracing removal (GH-28527)
https://github.com/python/cpython/commit/f4ccb79d52ee726d58bbb038ea98b4deec52001d


--

___
Python tracker 
<https://bugs.python.org/issue43760>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42969] pthread_exit & PyThread_exit_thread from PyEval_RestoreThread etc. are harmful

2021-09-23 Thread STINNER Victor


STINNER Victor  added the comment:

I'm not comfortable with PR 28525 which always hang threads which attempt to 
acquire the GIL after Python exited.

I would prefer to keep the current behavior by default, but give the ability to 
applications embedding Python to decide what to do.

With my Py_SetThreadExitCallback(func) idea, PyThread_exit_thread() would call 
func() and then pthread_exit(0). Applications can hang threads, log a message, 
call abort(), or whatever else.

I'm not comfortable with writing a portable function to "hang a thread". For 
example, I don't understand why PR 28525 processes Windows messages on hang 
threads.

Well, it's a complex problem :-(

--

___
Python tracker 
<https://bugs.python.org/issue42969>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41137] pdb uses the locale encoding for .pdbrc

2021-09-23 Thread STINNER Victor


STINNER Victor  added the comment:


New changeset ec7ffa4b5b262369f726a54e145e9c03eaeb4c1a by Victor Stinner in 
branch 'main':
bpo-41137: Reorganize What's New in Python 3.11 (GH-28518)
https://github.com/python/cpython/commit/ec7ffa4b5b262369f726a54e145e9c03eaeb4c1a


--

___
Python tracker 
<https://bugs.python.org/issue41137>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing)

2021-09-23 Thread STINNER Victor


STINNER Victor  added the comment:

I created PR 28527 to document PyThreadState.use_tracing removal and explain 
how to port existing code to Python 3.10.

--

___
Python tracker 
<https://bugs.python.org/issue43760>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing)

2021-09-23 Thread STINNER Victor


Change by STINNER Victor :


--
pull_requests: +26918
pull_request: https://github.com/python/cpython/pull/28527

___
Python tracker 
<https://bugs.python.org/issue43760>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45256] Remove the usage of the C stack in Python to Python calls

2021-09-22 Thread STINNER Victor


Change by STINNER Victor :


--
title: Remove the usage of the cstack in Python to Python calls -> Remove the 
usage of the C stack in Python to Python calls

___
Python tracker 
<https://bugs.python.org/issue45256>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45261] Unreliable (?) results from timeit (cache issue?)

2021-09-22 Thread STINNER Victor


STINNER Victor  added the comment:

I suggest to close this issue. timeit works as expected. It has limitations, 
but that's ok. Previous attempts to enhance timeit were rejected.

--

___
Python tracker 
<https://bugs.python.org/issue45261>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45261] Unreliable (?) results from timeit (cache issue?)

2021-09-22 Thread STINNER Victor


STINNER Victor  added the comment:

> but you give no reason for why you should not use the minimum.

See https://pyperf.readthedocs.io/en/latest/analyze.html#minimum-vs-average

I'm not really interested to convince you. Use the minimum if you believe that 
it better fits your needs.

But pyperf will stick to the mean to get more reproducible benchmark results ;-)

There are many believes and assumptions made by people running benchmarks. As I 
wrote, do your own experiments ;-) Enjoy multimodal distributions ;-)

--

___
Python tracker 
<https://bugs.python.org/issue45261>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux

2021-09-22 Thread STINNER Victor


STINNER Victor  added the comment:

See also bpo-19007: "precise time.time() under Windows 8: use 
GetSystemTimePreciseAsFileTime".

--

___
Python tracker 
<https://bugs.python.org/issue21302>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux

2021-09-22 Thread STINNER Victor


STINNER Victor  added the comment:

Livius: do you care about using nanosleep(), or can I close the issue?

--

___
Python tracker 
<https://bugs.python.org/issue21302>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux

2021-09-22 Thread STINNER Victor


STINNER Victor  added the comment:

> On Windows with a Python 3.11 debug build, I get:
> Mean +- std dev: 21.9 ms +- 7.8 ms (228 values)

I wrote an optimization to cache the Windows timer handle between time.sleep() 
calls (don't close it). I don't think that it's needed because they shortest 
sleep is about 15.6 ms. CreateWaitableTimerW() is likely way more fast than 
15.6 ms. So this optimization is basically useless.

--

___
Python tracker 
<https://bugs.python.org/issue21302>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux

2021-09-22 Thread STINNER Victor


STINNER Victor  added the comment:

bench.py: measure the shortest possible sleep. Use time.sleep(1e-10): 0.1 
nanosecond. It should be rounded to the resolution of the used sleep function, 
like 1 ns on Linux.

On Linux with Fedora 34 Python 3.10 executable, I get:

Mean +- std dev: 60.5 us +- 12.9 us (80783 values)

On Windows with a Python 3.11 debug build, I get:

Mean +- std dev: 21.9 ms +- 7.8 ms (228 values)

Sadly, it seems like on Windows 10, one of the following function still uses 
the infamous 15.6 ms resolution:

* CreateWaitableTimerW()
* SetWaitableTimer()
* WaitForMultipleObjects()

--
Added file: https://bugs.python.org/file50294/bench.py

___
Python tracker 
<https://bugs.python.org/issue21302>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux

2021-09-22 Thread STINNER Victor


STINNER Victor  added the comment:

Livius: your first PR modified Sleep() in Modules/_tkinter.c to use 
nanosleep(). I don't see the point since this function has a solution of 1 ms 
(10^-3). Using select() on Unix is enough: resolution of 1 us (10^-6).

--

___
Python tracker 
<https://bugs.python.org/issue21302>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue21302] time.sleep (floatsleep()) should use clock_nanosleep() on Linux

2021-09-22 Thread STINNER Victor


STINNER Victor  added the comment:


New changeset 58f8adfda3c2b42f654a55500e8e3a6433cb95f2 by Victor Stinner in 
branch 'main':
bpo-21302: time.sleep() uses waitable timer on Windows (GH-28483)
https://github.com/python/cpython/commit/58f8adfda3c2b42f654a55500e8e3a6433cb95f2


--

___
Python tracker 
<https://bugs.python.org/issue21302>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45256] Remove the usage of the cstack in Python to Python calls

2021-09-22 Thread STINNER Victor


Change by STINNER Victor :


--
nosy: +vstinner

___
Python tracker 
<https://bugs.python.org/issue45256>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28307] Accelerate 'string' % (value, ...) by using formatted string literals

2021-09-22 Thread STINNER Victor


STINNER Victor  added the comment:

commit a0bd9e9c11f5f52c7ddd19144c8230da016b53c6
Author: Serhiy Storchaka 
Date:   Sat May 8 22:33:10 2021 +0300

bpo-28307: Convert simple C-style formatting with literal format into 
f-string. (GH-5012)

C-style formatting with literal format containing only format codes
%s, %r and %a (with optional width, precision and alignment)
will be converted to an equivalent f-string expression.

It can speed up formatting more than 2 times by eliminating
runtime parsing of the format string and creating temporary tuple.

commit 8b010673185d36d13e69e5bf7d902a0b3fa63051
Author: Serhiy Storchaka 
Date:   Sun May 23 19:06:48 2021 +0300

bpo-28307: Tests and fixes for optimization of C-style formatting (GH-26318)

Fix errors:
* "%10.s" should be equal to "%10.0s", not "%10s".
* Tuples with starred expressions caused a SyntaxError.

--
nosy: +vstinner

___
Python tracker 
<https://bugs.python.org/issue28307>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45261] Unreliable (?) results from timeit (cache issue?)

2021-09-22 Thread STINNER Victor


STINNER Victor  added the comment:

> But timing results are not like that, the measurement errors are 
one-sided, not two: (..)

I suggest you to run tons of benchmarks and look at the distribution. The 
reality is more complex than what you may think.


> measurement = true value + random error

In my experience, there is no single "true value", they are multiple values. 
Concrete example where the Python randomized hash function gives different 
value each time you spawn a new Python process:
https://vstinner.github.io/journey-to-stable-benchmark-average.html

Each process has its own "true value", but pyperf spawns 20 Python processes :-)

There are multiple sources of randomness, not only the Python randomized hash 
function. On Linux, the process address space is randomized by ASLR. I may give 
different timing at each run.

Code placement, exact memory address, etc. Many things enter into the game when 
you look into functions which take less than 100 ns.

Here is report is about a value lower than a single nanosecond: 
"0.08ns/element".

--

I wrote articles about benchmarking:
https://vstinner.readthedocs.io/benchmark.html#my-articles

I gave a talk about it:

* 
https://raw.githubusercontent.com/vstinner/talks/main/2017-FOSDEM-Brussels/howto_run_stable_benchmarks.pdf
* https://archive.fosdem.org/2017/schedule/event/python_stable_benchmark/

Again, good luck with benchmarking, it's a hard problem ;-)

--

Once you will consider that you know everything about benchmarking, you should 
read the following paper and cry:
https://arxiv.org/abs/1602.00602

See also my analysis of PyPy performance:
https://vstinner.readthedocs.io/pypy_warmups.html

--

___
Python tracker 
<https://bugs.python.org/issue45261>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41137] pdb uses the locale encoding for .pdbrc

2021-09-22 Thread STINNER Victor


Change by STINNER Victor :


--
nosy: +vstinner
nosy_count: 4.0 -> 5.0
pull_requests: +26909
pull_request: https://github.com/python/cpython/pull/28518

___
Python tracker 
<https://bugs.python.org/issue41137>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45261] Unreliable (?) results from timeit (cache issue?)

2021-09-22 Thread STINNER Victor


STINNER Victor  added the comment:

PyPy emits a warning when the timeit module is used, suggesting to use pyperf.

timeit uses the minimum, whereas pyperf uses the average (arithmetic mean).

timeit uses a single process, pyperf spawns 21 processes: 1 just for the loop 
calibration, 20 to compute values.

timeit computes 5 values, pyperf computes 60 values.

timeit uses all computed values, pyperf ignores the first value considered as a 
"warmup value" (the number of warmup values can be configured).

timeit doesn't compute the standard deviation, pyperf does. The standard 
deviation gives an idea if the benchmark looks reliable or not. IMO results 
without standard deviation should not be trusted.

pyperf also emits warning when a benchmark doesn't look reliable. For example, 
if the user ran various workload while the benchmark was running.

pyperf also supports storing results in a JSON file which stores all values, 
but also metadata.

I cannot force people to stop using timeit. But there are reason why pyperf is 
more reliable than timeit.

Benchmarking is hard. See pyperf documentation giving hints how to get 
reproducible benchmark results:
https://pyperf.readthedocs.io/en/latest/run_benchmark.html#how-to-get-reproducible-benchmark-results

Read also this important article ;-)
"Biased Benchmarks (honesty is hard)"
http://matthewrocklin.com/blog/work/2017/03/09/biased-benchmarks

--

___
Python tracker 
<https://bugs.python.org/issue45261>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



<    11   12   13   14   15   16   17   18   19   20   >