[issue38428] Can't gracefully ctrl+C multiprocessing pool on Python3 & Windows

[Eryk Sun]

Eryk Sun  added the comment:

In bpo-29971 it was suggested to split the PyThread lock API in Windows into an 
"interruptible lock" API that's based on emulated condition variables and a 
"fast lock" API that's based on native condition variables and SRW locks. Maybe 
the only API change that's needed in that regard is to add 
PyThread_allocate_lock_ex(int interruptible), and otherwise dispatch on the 
lock type that's set in the PNRMUTEX.

--
versions: +Python 3.10 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38428] Can't gracefully ctrl+C multiprocessing pool on Python3 & Windows

[Eryk Sun]

Eryk Sun  added the comment:

> Python 2.7 had totally different code, IIRC.

In 2.7 the wait() method of threading._Condition implements the timeout with a 
loop the calls time.sleep() with an increasing delay from 500 microseconds up 
to 50 ms, and no more than the remaining time.

> But the good news is that we should be able to use code similar to 
> what's in Modules/timemodule.c#L1878 (the pysleep() function) to also 
> wait on the event we use to emulate signals on Windows. 

If Python continues to use emulated condition variables, then the kernel wait 
in _PyCOND_WAIT_MS can easily be adapted to include the SIGINT event. However, 
isn't it planned at some point to remove the emulated condition-variable 
implementation in favor of native condition variables? If so and PyCOND_WAIT 
and PyCOND_TIMEDWAIT end up calling SleepConditionVariableSRW, there's no 
obvious way to interrupt this to support Ctrl+C and Ctrl+Break (given a custom 
SIGBREAK handler) in the main thread. 

In particular, in Windows 8+, SleepConditionVariableSRW begins by spinning in 
user mode for a fixed number of cycles. Then it calls NtWaitForAlertByThreadId 
to block until alerted by NtAlertThreadByThreadId, which gets called internally 
by Wake[All]ConditionVariable. It wouldn't be kosher to call 
NtAlertThreadByThreadId directly in order to alert the main thread because it's 
an undocumented system call, plus it's poking into the internal implementation 
details of SleepConditionVariableSRW and WakeConditionVariable, which could 
leave the variable in a bad state.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38428] Can't gracefully ctrl+C multiprocessing pool on Python3 & Windows

[Steve Dower]

Steve Dower  added the comment:

I think you're right about that change touching the right code, but I'm pretty 
sure it wasn't working before either. (Python 2.7 had totally different code, 
IIRC.)

But the good news is that we should be able to use code similar to what's in 
Modules/timemodule.c#L1878 (the pysleep() function) to also wait on the event 
we use to emulate signals on Windows. But perhaps there's a reason Antoine 
(nosied) didn't do that originally?

--
nosy: +davin, pitrou
stage:  -> test needed
type:  -> behavior
versions: +Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38428] Can't gracefully ctrl+C multiprocessing pool on Python3 & Windows

[Rebecca Fair]

Rebecca Fair  added the comment:

Sorry, I didn't realise it wouldn't link to a commit automatically, the commit 
is here: 
https://github.com/python/cpython/commit/7c3e5773954009d65520eb063621cf7724da88e3

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38428] Can't gracefully ctrl+C multiprocessing pool on Python3 & Windows

[Rebecca Fair]

New submission from Rebecca Fair :

I want to be able to Ctrl+C to exit a multiprocessing Pool.map gracefully, and 
have made a solution based on this Stack Overflow answer: 
https://stackoverflow.com/questions/11312525/catch-ctrlc-sigint-and-exit-multiprocesses-gracefully-in-python

However, this solution works on Linux, and on Windows 10 with Python 2.7, but 
not on Windows 10 with Python 3.7. The Ctrl+C is just ignored and I have to 
kill the processes manually. I've attached the minimum code required to 
reproduce the problem.

Is this a bug, expected, or is there a workaround? I believe it might be caused 
by the behaviour of threading.Condition.wait() being changed in commit 7c3e577 
but I don't know enough about signalling to say any more than that

--
components: Windows
files: bug.py
messages: 354344
nosy: paul.moore, rebeccafair, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Can't gracefully ctrl+C multiprocessing pool on Python3 & Windows
versions: Python 3.7
Added file: https://bugs.python.org/file48653/bug.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com