[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[Trey Hunner]

Trey Hunner added the comment:

This is a problem I experience occasionally while teaching and while developing 
teaching curriculum.

I tend to close problem windows quickly enough to avoid a computer crash in 
front of a live audience, but it's still an annoyance to get the REPL state 
back to the way I had it before killing the process.

This mostly happens when I'm teaching iterators or itertools, but occasionally 
I hit a "Ctrl-C free zone" in other ways.  I believe this memory-filling 
snippet wouldn't respond to Ctrl-C either: x=[0]*2**30

I hadn't thought to report a bug on this because my use seemed niche.  I mostly 
get this error while demonstrating concepts via weird/incorrect code at the 
REPL.

--
nosy: +trey

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[STINNER Victor]

STINNER Victor added the comment:

You should try https://github.com/python/performance to get reliable
benchmark results ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[Nick Coghlan]

Nick Coghlan added the comment:

George's initial patch that naively checks for signals on every iteration could 
be used to get an upper bound on the likely benchmark impact.

I do think this is a case where we'll want a dedicated microbenchmark to 
complement the macrobenchmark suite, though - Raymond's right that these 
functions are frequently used to optimise sections of code that have already 
been identified as performance bottlenecks in a particular application, so we 
need to be really careful with changes that might make them slower (even if the 
current macrobenchmarks say things are still broadly OK).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[Terry J. Reedy]

Terry J. Reedy added the comment:

The request is to check 'occasionally'.  To me this means perhaps once a 
second, which is to say, ^c should generally interrupt within a second, and on 
average with half a second.  The following takes just under a second: "sum(i 
for i in range(1000))" whereas "for i in range(1000): pass" takes a 
fourth of that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[STINNER Victor]

STINNER Victor added the comment:

Can someone work on a patch? Then we can benchmark it to take a
decision ;-) Maybe we might expose signalmodule.c internals to have a
cheaper check?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[Nick Coghlan]

Nick Coghlan added the comment:

As far as the "What's the benefit to users?" question goes, I think the main 
intended beneficiaries would be children and other folks playing at the command 
prompt and trying out different things.

The "no segfaults from normal Python code" rule aims to make that kind of 
exploration a significantly more positive experience than it is in a language 
like C - you're far more likely to get a traceback than you are to have the 
interpreter fall over completely. Tracebacks can be intimidating to new users, 
but they still give them new information to work with.

Infinite loops at the Python level are similarly about as friendly to ad hoc 
exploration as we can possibly make them: Ctrl-C will break you out of them 
with a traceback.

Implementation level infinite (or near-infinite, or 
finite-but-eating-all-of-RAM) loops by contrast are much closer to their 
traditional C level counterparts: your only way out is via destructive 
termination of the entire process.

So that's why I think this is an idea worth exploring further, even though it 
may still turn out to be impractical for code readability or runtime speed 
reasons.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[Nick Coghlan]

Nick Coghlan added the comment:

While I agree with Raymond regarding the performance implications if this isn't 
handled carefully, I think we're also getting to a point where better 
accounting for signal handling latency in inner loops is something that could 
be considered for 3.7 - the benchmarking infrastructure being built out to 
measure performance optimisations would also allow overhead tuning of a 
"batched iteration" idiom where signals were checked for either every N 
thousand iterations, periodically based on time, or some combination of the two.

Benchmarking to measure the speed impact is going to be essential, though - 
this is a case where changing the behaviour is clearly possible, so the key 
questions are whether or not the resulting runtime overhead can be made low 
enough to be deemed acceptable, and whether or not it can be done in a way that 
doesn't make the affected parts of the code base effectively unreadable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[Raymond Hettinger]

Raymond Hettinger added the comment:

I think this needs more discussion on python-dev before going down this path.

In reality, we have many places that have "long running" C code when fed 
extreme arguments.  In practice, we almost never have a problem with these 
except for cute toy bug reports.   To "fix" this, we would need to alter all 
possible long running data consumers or alter all possible long running data 
producers.  This kind of change is hard to test, gums up the code, and hinders 
future maintainability for near zero benefit to ordinary users.

min(range(1000))# There a lots of places like this

The proposed patch is indelicate about the addition of signal checking.  It 
check on every single iteration right in the middle of most highly optimizied, 
tightest, most speed critical loops in Python, making every use pay a cost for 
something almost no one will ever benefit from.

--
priority: normal -> low
versions: +Python 3.7 -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[George Slavin]

George Slavin added the comment:

I've attached the test for this patch (I couldn't figure out how to upload two 
files with one comment).

--
Added file: http://bugs.python.org/file44747/test_sig_int_builtins.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[George Slavin]

George Slavin added the comment:

I have a patch that checks for KeyboardInterrupt during builtin operations.  
This allows sum, max, min, list, dict, set, and tuple calls to be interrupted 
when they are working on infinite iterators.

I've attached the patch, and a test I wrote to show that you can ctrl-c out of 
all the above calls.

This is my first attempt at a patch, so I would appreciate any feedback on what 
I need to fix to allow it to be submitted :)

--
keywords: +patch
nosy: +gslavin
Added file: http://bugs.python.org/file44746/KeyboardInterrupt.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[wim glenn]

Changes by wim glenn :


--
nosy: +wim.glenn

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[Terry J. Reedy]

Terry J. Reedy added the comment:

I verified that ^Break(pause) works.  It even causes ^C to be printed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[STINNER Victor]

STINNER Victor added the comment:

> Great idea.  In Windows, closing the window with [x] will kill the process, 
> at the cost of loosing its contents.

Note: after a few years, I heard that Windows supports something like SIGKILL: 
CTRL+Pause kills the current process ;-) You loose the process, but you don't 
have to close the terminal, confirm and reopen a new terminal, go back to your 
working directly, etc.

Note 2: Even more off-topic, type .~ in an SSH session to kill it, again 
it avoids to reopen a terminal window ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[Terry J. Reedy]

Terry J. Reedy added the comment:

Great idea.  In Windows, closing the window with [x] will kill the process, at 
the cost of loosing its contents.  In IDLE's Shell, Restart Shell will do the 
same without killing IDLE, but it is easy to not know of or forget that option 
in a moment of panic.

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[Nick Coghlan]

Nick Coghlan added the comment:

At least list() and potentially other container constructors are also affected.

While it's mentioned in the thread, I'll explicitly note here that the problem 
is specifically with iterators implemented in C, like itertools.count().

Iterators implemented in Python already evaluate Python code on each iteration, 
which means Ctrl-C gets detected and converted to KeyboardInterrupt.

--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[Raymond Hettinger]

Raymond Hettinger added the comment:

FWIW, this doesn't seem to be solving any real life problems, only contrived 
examples.  If any changes are made, they should be very restricted in scope, 
taking care to not wreck the performance of tools designed for speed and not to 
add useless crud to otherwise clean code.

I think thought needs to be given to the question of whether there is a general 
purpose means of interrupting C code.  The real world problems I see occurring 
aren't trivial examples like sum(itertools.count()) -- what people routinely 
bump into is IDLE's use of Tkinter becoming unresponsive, numpy being asked to 
apply a high algorithmic complexity transformation to too large of matrix, 
mistakes during development with ctypes, etc.  In other words, we should 
collect a list of real problems that cause a spinning wheel of death, rather 
than cases where the user specifically asked for an infinite summation.

If there are patches aimed at the toy examples, I think the emphasis should be 
on the consumer side (min, max, sum, list, sorted, etc) rather than on the 
producer side (i.e. itertools).  One reason to focus on the consumer side is 
that we control more of those (the world is full of third-party C extension 
producers).  Another reason is the lazy-evaluation style of functional 
programming puts the termination responsibility with consumers to provide the 
most flexibility in grouping the tools (see 
http://worrydream.com/refs/Hughes-WhyFunctionalProgrammingMatters.pdf ).  
Lastly, the focus on the consumer is suggested by itertools programming 
patterns (i.e. chains of nested iterators driven by a single consumer of 
mapping/zipping together infinite iterators with a single finite iterator such 
as map(somefunc, count(), repeat(somearg), repeat(otherarg))).  In those cases, 
focusing on the many producers would result in many repeated checks per 
iterator; whereas, focusing on the consumer woul
 d give only one check per iteration.

In thinking about the real world issues I've seen, ISTM we should worry instead 
about unbounded consumption of memory resources without demanding that a user 
realize what is going on and making them jump in with a Cntl-C before something 
bad happens.  If you run list(range(1000)), memory will fill, then 
thrash your SSD with swaps, and make your CPU melt.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
Removed message: http://bugs.python.org/msg260255

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[Steven D'Aprano]

New submission from Steven D'Aprano:

There are a few operations such as summing or unpacking infinite iterators 
where the interpreter can become unresponsive and ignore Ctrl-C 
KeyboardInterrupt. Guido suggests that such places should occasionally check 
for signals:

https://mail.python.org/pipermail/python-ideas/2016-February/038426.html

--
components: Interpreter Core
messages: 260189
nosy: steven.daprano
priority: normal
severity: normal
status: open
title: Occasionally check for Ctrl-C in long-running operations like sum
type: behavior
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com