Re: [Python-Dev] PEP-419: Protecting cleanup statements from interruptions
Hi Antoine, On Mon, Apr 9, 2012 at 12:06 AM, Antoine Pitrou wrote: > > Hello Paul, > > Thanks for the PEP and the description of the various issues. > >> An example implementation of a SIGINT handler that interrupts safely >> might look like:: >> >> import inspect, sys, functools >> >> def sigint_handler(sig, frame): >> if inspect.getcleanupframe(frame) is None: >> raise KeyboardInterrupt() >> sys.setcleanuphook(functools.partial(sigint_handler, 0)) > > It is not clear whether you are proposing this for the default signal > handler, or only as an example that third-party libraries or frameworks > could implement. > Only as an example. The reason is in "Modifying KeyboardInterrupt" section under "Unresolved Issues". So it might be changed if there is demand. -- Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is why we shouldn't call it a "monotonic clock" (was: PEP 418 is too divisive and confusing and should be postponed)
On Sun, Apr 8, 2012 at 5:00 PM, Victor Stinner wrote: >> IOW "What's good enough for sleep() is good enough for >> user-implemented timeouts and scheduling." as a way to reach at least >> one decision for a platform with agreed-upon cross-platform >> characteristics that are useful. > > sleep() is implemented in the kernel. The kernel is notified when a > clock is set, and so can choose how to handle time adjustement. Most > "sleeping" functions use the system clock but don't care of clock > adjustement. We're going around in circles. I'm not asking what sleep does, I want on principle a timer that does the same thing as sleep(), regardless of how sleep() works. So if on some OS sleep() uses the same algorithm as CLOCK_MONOTONIC_RAW, I want my timer to use that too. But if on some other OS sleep() uses CLOCK_MONOTONIC, I want my timer there to use that. And if on some OS sleep() is buggy and uses the time-of-day clock, well, I wouldn't mind if my timer used the same thing. >> I personally have a need for one potentially different clock -- to >> measure short intervals for benchmarks and profiling. This might be >> called time.performancetimer()? > > I deferred this topic because it is unclear to me if such timer has to > count elapsed time during a sleep or not. For example, time.clock() > does on UNIX, whereas it doesn't on Windows. I will declare that that was a mistake in clock(), but one that's too late to fix, because fixing it would break too many programs (those on *nix that use it to measure CPU time, and those on Windows that use it to measure real time). >You may need two clocks > for this: > * time.perf_counter(): high-resolution timer for benchmarking, count > time elasped during a sleep > * time.process_time(): High-resolution (?) per-process timer from the > CPU. (other possible names: time.process_cpu_time() or > time.cpu_time()) TBH I don't need another timer that measures CPU time (not even on Windows). In a sense, measuring CPU time is a relic from the age of mainframes and timesharing, where CPU time was the most precious resource (and in some cases the unit in which other resources were expressed for accounting purposes). In modern days, it's much more likely that the time you're measuring is somehow related to how long a use has to wait for some result (e.g. web response times) and here "wait time" is just as real as CPU time. > On Windows, GetProcessTimes() has not a "high-resolution": it has a > accuracy of 1 ms in the best case. QueryPerformanceCounter() counts > time elapsed during a sleep, I don't know for GetProcessTimes. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is why we shouldn't call it a "monotonic clock" (was: PEP 418 is too divisive and confusing and should be postponed)
On 09Apr2012 02:00, Victor Stinner wrote: | > I personally have a need for one potentially different clock -- to | > measure short intervals for benchmarks and profiling. This might be | > called time.performancetimer()? | | I deferred this topic because it is unclear to me if such timer has to | count elapsed time during a sleep or not. For example, time.clock() | does on UNIX, whereas it doesn't on Windows. You may need two clocks | for this: | * time.perf_counter(): high-resolution timer for benchmarking, count | time elasped during a sleep For POSIX, sounds like CLOCK_MONOTONIC_RAW to me. | * time.process_time(): High-resolution (?) per-process timer from the | CPU. (other possible names: time.process_cpu_time() or | time.cpu_time()) POSIX offers CLOCK_PROCESS_CPUTIME_ID and CLOCK_THREAD_CPUTIME_ID that seem to suit this need, depending on your threading situation (and what you're measuring). | On Windows, GetProcessTimes() has not a "high-resolution": it has a | accuracy of 1 ms in the best case. This page: http://msdn.microsoft.com/en-us/library/windows/desktop/ms683223%28v=vs.85%29.aspx says "100-nanosecond time units". Am I going to the wrong place to learn about these functions? -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ I distrust a research person who is always obviously busy on a task. - Robert Frosch, VP, GM Research ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] an alternative to embedding policy in PEP 418
On Fri, Apr 6, 2012 at 23:26, Ethan Furman wrote: > Huh? Your point is that all APIs are less than ideal because you have to > read the docs to know for certain how they work? No. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is why we shouldn't call it a "monotonic clock" (was: PEP 418 is too divisive and confusing and should be postponed)
> IOW "What's good enough for sleep() is good enough for > user-implemented timeouts and scheduling." as a way to reach at least > one decision for a platform with agreed-upon cross-platform > characteristics that are useful. sleep() is implemented in the kernel. The kernel is notified when a clock is set, and so can choose how to handle time adjustement. Most "sleeping" functions use the system clock but don't care of clock adjustement. > I personally have a need for one potentially different clock -- to > measure short intervals for benchmarks and profiling. This might be > called time.performancetimer()? I deferred this topic because it is unclear to me if such timer has to count elapsed time during a sleep or not. For example, time.clock() does on UNIX, whereas it doesn't on Windows. You may need two clocks for this: * time.perf_counter(): high-resolution timer for benchmarking, count time elasped during a sleep * time.process_time(): High-resolution (?) per-process timer from the CPU. (other possible names: time.process_cpu_time() or time.cpu_time()) On Windows, GetProcessTimes() has not a "high-resolution": it has a accuracy of 1 ms in the best case. QueryPerformanceCounter() counts time elapsed during a sleep, I don't know for GetProcessTimes. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP-419: Protecting cleanup statements from interruptions
2012/4/8 Paul Colomiets : > Function 'sys.setcleanuphook' > - > > A new function for the ``sys`` module is proposed. This function sets > a callback which is executed every time ``f_in_cleanup`` becomes > false. Callbacks get a frame object as their sole argument, so that > they can figure out where they are called from. Calling a function every time you leave a finally block? Isn't that a bit expensive? -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP-419: Protecting cleanup statements from interruptions
Hello Paul, Thanks for the PEP and the description of the various issues. > An example implementation of a SIGINT handler that interrupts safely > might look like:: > > import inspect, sys, functools > > def sigint_handler(sig, frame): > if inspect.getcleanupframe(frame) is None: > raise KeyboardInterrupt() > sys.setcleanuphook(functools.partial(sigint_handler, 0)) It is not clear whether you are proposing this for the default signal handler, or only as an example that third-party libraries or frameworks could implement. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP-419: Protecting cleanup statements from interruptions
Hi,
I present my first PEP.
http://www.python.org/dev/peps/pep-0419/
Added text to the end of email for easier reference. Comments are welcome.
--
Paul
PEP: 419
Title: Protecting cleanup statements from interruptions
Version: $Revision$
Last-Modified: $Date$
Author: Paul Colomiets
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 06-Apr-2012
Python-Version: 3.3
Abstract
This PEP proposes a way to protect Python code from being interrupted
inside a finally clause or during context manager cleanup.
Rationale
=
Python has two nice ways to do cleanup. One is a ``finally``
statement and the other is a context manager (usually called using a
``with`` statement). However, neither is protected from interruption
by ``KeyboardInterrupt`` or ``GeneratorExit`` caused by
``generator.throw()``. For example::
lock.acquire()
try:
print('starting')
do_something()
finally:
print('finished')
lock.release()
If ``KeyboardInterrupt`` occurs just after the second ``print()``
call, the lock will not be released. Similarly, the following code
using the ``with`` statement is affected::
from threading import Lock
class MyLock:
def __init__(self):
self._lock_impl = Lock()
def __enter__(self):
self._lock_impl.acquire()
print("LOCKED")
def __exit__(self):
print("UNLOCKING")
self._lock_impl.release()
lock = MyLock()
with lock:
do_something
If ``KeyboardInterrupt`` occurs near any of the ``print()`` calls, the
lock will never be released.
Coroutine Use Case
--
A similar case occurs with coroutines. Usually coroutine libraries
want to interrupt the coroutine with a timeout. The
``generator.throw()`` method works for this use case, but there is no
way of knowing if the coroutine is currently suspended from inside a
``finally`` clause.
An example that uses yield-based coroutines follows. The code looks
similar using any of the popular coroutine libraries Monocle [1]_,
Bluelet [2]_, or Twisted [3]_. ::
def run_locked():
yield connection.sendall('LOCK')
try:
yield do_something()
yield do_something_else()
finally:
yield connection.sendall('UNLOCK')
with timeout(5):
yield run_locked()
In the example above, ``yield something`` means to pause executing the
current coroutine and to execute coroutine ``something`` until it
finishes execution. Therefore the coroutine library itself needs to
maintain a stack of generators. The ``connection.sendall()`` call waits
until the socket is writable and does a similar thing to what
``socket.sendall()`` does.
The ``with`` statement ensures that all code is executed within 5
seconds timeout. It does so by registering a callback in the main
loop, which calls ``generator.throw()`` on the top-most frame in the
coroutine stack when a timeout happens.
The ``greenlets`` extension works in a similar way, except that it
doesn't need ``yield`` to enter a new stack frame. Otherwise
considerations are similar.
Specification
=
Frame Flag 'f_in_cleanup'
-
A new flag on the frame object is proposed. It is set to ``True`` if
this frame is currently executing a ``finally`` clause. Internally,
the flag must be implemented as a counter of nested finally statements
currently being executed.
The internal counter also needs to be incremented during execution of
the ``SETUP_WITH`` and ``WITH_CLEANUP`` bytecodes, and decremented
when execution for these bytecodes is finished. This allows to also
protect ``__enter__()`` and ``__exit__()`` methods.
Function 'sys.setcleanuphook'
-
A new function for the ``sys`` module is proposed. This function sets
a callback which is executed every time ``f_in_cleanup`` becomes
false. Callbacks get a frame object as their sole argument, so that
they can figure out where they are called from.
The setting is thread local and must be stored in the
``PyThreadState`` structure.
Inspect Module Enhancements
---
Two new functions are proposed for the ``inspect`` module:
``isframeincleanup()`` and ``getcleanupframe()``.
``isframeincleanup()``, given a frame or generator object as its sole
argument, returns the value of the ``f_in_cleanup`` attribute of a
frame itself or of the ``gi_frame`` attribute of a generator.
``getcleanupframe()``, given a frame object as its sole argument,
returns the innermost frame which has a true value of
``f_in_cleanup``, or ``None`` if no frames in the stack have a nonzero
value for that attribute. It starts to inspect from the specified
frame and walks to outer frames using ``f_back`` pointers, just like
``getouterframes()`` does.
Example
===
An example implementation of a SIGINT handler that interrupts safely
might look like::
import inspect,
Re: [Python-Dev] this is why we shouldn't call it a "monotonic clock" (was: PEP 418 is too divisive and confusing and should be postponed)
On Sun, 8 Apr 2012 07:29:30 -0700 Guido van Rossum wrote: > > What to name it can't be decided this way, although I might put > forward time.sleeptimer(). interval_timer() ? I would suggest timer() simply, but it's too close to time(). > I personally have a need for one potentially different clock -- to > measure short intervals for benchmarks and profiling. This might be > called time.performancetimer()? It's called perf_counter() in the PEP: http://www.python.org/dev/peps/pep-0418/#deferred-api-time-perf-counter Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is why we shouldn't call it a "monotonic clock" (was: PEP 418 is too divisive and confusing and should be postponed)
On Sun, Apr 8, 2012 at 3:42 AM, Antoine Pitrou wrote: > >> | I made the same suggestion earlier but I don't know that anyone did >> | anything with it. :-( It would be nice to know what clock sleep() uses >> | on each of the major platforms. >> >> I saw it but didn't know what I could do with it, or even if it can be >> found out in any very general sense. >> >> Looking at nanosleep(2) on a recent Linux system says: > > time.sleep() uses select(), not nanosleep(). > select() is not specified to use a particular clock. However, since it > takes a timeout rather than a deadline, it would be reasonable for it > to use a non-adjustable clock :-) > http://pubs.opengroup.org/onlinepubs/9699919799/functions/select.html Still, my hope was to cut short a bunch of discussion by declaring that on every platform, one of the timers available should match the one used by sleep(), select() and the like -- assuming they all use the same timer underneath in a typical OS, even though (due to standardization at different times by different standards bodies) they aren't all specified the same. IOW "What's good enough for sleep() is good enough for user-implemented timeouts and scheduling." as a way to reach at least one decision for a platform with agreed-upon cross-platform characteristics that are useful. What to name it can't be decided this way, although I might put forward time.sleeptimer(). I personally have a need for one potentially different clock -- to measure short intervals for benchmarks and profiling. This might be called time.performancetimer()? -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is why we shouldn't call it a "monotonic clock" (was: PEP 418 is too divisive and confusing and should be postponed)
> | I made the same suggestion earlier but I don't know that anyone did > | anything with it. :-( It would be nice to know what clock sleep() uses > | on each of the major platforms. > > I saw it but didn't know what I could do with it, or even if it can be > found out in any very general sense. > > Looking at nanosleep(2) on a recent Linux system says: time.sleep() uses select(), not nanosleep(). select() is not specified to use a particular clock. However, since it takes a timeout rather than a deadline, it would be reasonable for it to use a non-adjustable clock :-) http://pubs.opengroup.org/onlinepubs/9699919799/functions/select.html Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
