Re: [Python-Dev] Problem with signals in a single threaded application
Greg Ewing schrieb:
Correct me if I'm wrong, but what I got from the OP
was that the current method does
Ok, I'm correcting you: This is not what the current
method does:
if (is_tripped) {
for each signal {
if the signal has occurred, call its handler
}
is_tripped = 0;
}
Instead, the current method does this:
if (!is_tripped)
return;
for each signal {
if the signal has occurred, call its handler
}
is_tripped = 0
and the problem is that any setting of is_tripped that
occurs in the midst of calling the handlers gets
wiped out at the end.
Changing this to
while (is_tripped) {
for each signal {
if the signal has occurred, call its handler
}
is_tripped = 0;
}
My proposal (wrap the for loop with another is_tripped
looo) would literally give
if (!is_tripped)
return;
while(is_tripped) {
for each signal {
if the signal has occurred, call its handler
}
}
is_tripped = 0
Of course, that doesn't make much sense as it gives
an infinite loop. You need to clear is_tripped somewhere,
and I thought clearing it before the for loop would
work
if (!is_tripped)
return;
while(is_tripped) {
is_tripped = 0;
for each signal {
if the signal has occurred, call its handler
}
}
If you really care, you can make that a while instead
of an if so that you don't have to wait until the next
CheckSignals. But if the signal had arrived a few
microseconds later you'd have to do that anyway, so
I don't see it as a big deal.
Sure. However, it's not that the signal would occur
randomly a few microseconds later, but instead
occurs *in* the signal handler.
I think Python currently has some guarantee that
a signal handler will be called quickly: either the
signal gets tripped in a blocking system call, which
ought to abort the system call, or the interpreter
is executing byte codes, when it will check for
signals every nth instruction.
A signal can get delayed significantly if the
interpreter makes a blocking system call before
handling the signal. I think this can happen
even today (if there is a blocking call between
the signal and the nth instruction, the signal
may still get delayed). However, in this specific
case, I think the chance that that the signal
gets delayed is high, and the case can be easily
implemented to avoid that risk.
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
Greg Ewing schrieb: Please try to come up with a patch (e.g. by putting a while(is_tripped) loop around the for loop). That isn't going to fix it. Why not? What's needed is to somehow atomically test and clear is_tripped at the beginning. How would that help? The case in question was a signal raised inside a signal handler. With my proposed solution, that would be processed in the next while loop; if it is cleared at the beginning, the call will wait for the next CheckSignals invocation. Also, why does it need to clear is_tripped atomically? If it is only cleared if it is set, I see no need to make the test and the clearing atomic. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
I apologise for going off-topic, but this is an explanation of why I said that signal handling is not reliable. The only relevance to Python is that Python should avoid relying on signals if possible, and try to be a little defensive if not. Signals will USUALLY do what is expected, but not always :-( Anything further by Email, please. Greg Ewing [EMAIL PROTECTED] wrote: This one looks like an oversight in Python code, and so is a bug, but it is important to note that signals do NOT work reliably under any Unix or Microsoft system. That's a rather pessimistic way of putting it. In my experience, signals in Unix mostly do what they're meant to do quite reliably -- it's just a matter of understanding what they're meant to do. Yes, it is pessimistic, but I am afraid that my experience is that it is so :-( That doesn't deny your point that they MOSTLY do 'work', but car drivers MOSTLY don't need to wear seat belts, either. I am talking about high-RAS objectives, and ones where very rare failure modes can become common (e.g. HPC and other specialist uses). More commonly, there are plain bugs in the implementations which are sanctioned by the standards (Linux is relatively disdainful of such legalistic games). Because they say that everything is undefined behaviour, many vendors' support mechanisms will refuse to accept bug reports unless you push like hell. And, as some are DIABOLICALLY difficult to explain, let alone demonstrate, they can remain lurking for years or decades. There may be bugs in certain systems that cause signals to get lost under obscure circumstances, but that's no reason for Python to make the situation worse by introducing bugs of its own. 100% agreed. Two related signals received between two 'checkpoints' (i.e. when the signal is tested and cleared). You may only get one of them, and 'related' does not mean 'the same'. I wasn't aware that this could happen between different signals. If it can, there must be some rationale as to why the second signal is considered redundant. Otherwise there's a bug in either the design or the implementation. Nope. There is often a clash between POSIX and the hardware, or a cause where a 'superior' signal overrides an 'inferior' one. I have seen SIGKILL flush some other signals, for example. And, on some systems, SIGFPE may be divided into the basic hardware exceptions. If you catch SIGFPE as such, all of those may be cleared. I don't think that many (any?) current systems do that. And it is actually specified to occur for the SISSTOP, SIGTSTP, SIGTTIN, SIGTTOU, SIGCONT group. A second signal received while the first is being 'handled' by the operating system or language run-time system. That one sounds odd to me. I would expect a signal received during the execution of a handler to be flagged and cause the handler to be called again after it returns. But then I'm used to the BSD signal model, which is relatively sane. It's nothing to do with the BSD model, which may be saner but still isn't 100% reliable, but occurs at a lower layer. At the VERY lowest level, when a genuine hardware event causes an interrupt, the FLIH (first-level interrupt handler) runs in God mode (EVERYTHING disabled) until it classifies what is going on. This is a ubiquitous misdesign of modern hardware, but that is off-topic. Hardware 'signals' from other CPUs/devices may well get lost if they occur in that window. And there are other, but less extreme, causes at higher levels in the operating system. Unix and Microsoft do NOT have a reliable signal delivery model, where the sender of a signal checks if the recipient has got it and retries if not. Some operating systems do - but I don't think that BSD does. A signal sent while the operating system is doing certain things to the application (including, sometimes, when it is swapped out or deep in I/O.) That sounds like an outright bug. I can't think of any earthly reason why the handler shouldn't be called eventually, if it remains installed and the process lives long enough. See above. It gets lost at a low level. That is why you can cause serious time drift on an IBM PC (most modern ones) by hammering the video card or generating streams of floating-point fixups. Most people don't notice, because xntp or equivalent fixes it up. And there are worse problems. I could start on cross-CPU TLB and ECC handling on large shared memory systems. I managed to get an Origin in a state where it wouldn't even power down from the power-off button, and I had to flip breakers, due to THAT one! I have reason to believe that all largish SMP systems have similar problems. Again, it is possible to design an operating system to avoid those issues, but we are talking about mainstream ones, and they don't. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.:
Re: [Python-Dev] Problem with signals in a single threaded application
Martin v. Löwis wrote:
Greg Ewing schrieb:
Please
try to come up with a patch (e.g. by putting a while(is_tripped) loop
around the for loop).
That isn't going to fix it.
Why not?
Correct me if I'm wrong, but what I got from the OP
was that the current method does
if (is_tripped) {
for each signal {
if the signal has occurred, call its handler
}
is_tripped = 0;
}
and the problem is that any setting of is_tripped that
occurs in the midst of calling the handlers gets
wiped out at the end.
Changing this to
while (is_tripped) {
for each signal {
if the signal has occurred, call its handler
}
is_tripped = 0;
}
doesn't solve that, because is_tripped still gets
set to 0 before it's tested again.
Also, why does it need to clear is_tripped atomically?
Thinking about it more, probably it doesn't. What's
important is to clear it *before* testing whether any
handlers need to be called, i.e.
if (is_tripped) {
is_tripped = 0;
for each signal {
if the signal has occurred, call its handler
}
}
If you really care, you can make that a while instead
of an if so that you don't have to wait until the next
CheckSignals. But if the signal had arrived a few
microseconds later you'd have to do that anyway, so
I don't see it as a big deal.
--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
On 1/27/07, Greg Ewing [EMAIL PROTECTED] wrote:
Why not?
Correct me if I'm wrong, but what I got from the OP
was that the current method does
if (is_tripped) {
for each signal {
if the signal has occurred, call its handler
}
is_tripped = 0;
}
and the problem is that any setting of is_tripped that
occurs in the midst of calling the handlers gets
wiped out at the end.
Changing this to
while (is_tripped) {
for each signal {
if the signal has occurred, call its handler
}
is_tripped = 0;
}
doesn't solve that, because is_tripped still gets
set to 0 before it's tested again.
Agreed.
Thinking about it more, probably it doesn't. What's
important is to clear it *before* testing whether any
handlers need to be called, i.e.
if (is_tripped) {
is_tripped = 0;
for each signal {
if the signal has occurred, call its handler
}
}
That's exactly what my patch does as you can see here:
http://www.python.org/sf/1643738
Regards,
-- Ulisses
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
Martin v. Löwis wrote: Please try to come up with a patch (e.g. by putting a while(is_tripped) loop around the for loop). That isn't going to fix it. What's needed is to somehow atomically test and clear is_tripped at the beginning. You can put a while loop around it as well if you want, but it's not strictly necessary. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
Nick Maclaren wrote: This one looks like an oversight in Python code, and so is a bug, but it is important to note that signals do NOT work reliably under any Unix or Microsoft system. That's a rather pessimistic way of putting it. In my experience, signals in Unix mostly do what they're meant to do quite reliably -- it's just a matter of understanding what they're meant to do. There may be bugs in certain systems that cause signals to get lost under obscure circumstances, but that's no reason for Python to make the situation worse by introducing bugs of its own. Two related signals received between two 'checkpoints' (i.e. when the signal is tested and cleared). You may only get one of them, and 'related' does not mean 'the same'. In the case where they're the same, I wouldn't say that the second signal has been lost. Rather, it's simply redundant -- a call to the handler is already pending, and will happen eventually. (If you're expecting one call per signal, then you're using the wrong mental model for signals.) I wasn't aware that this could happen between different signals. If it can, there must be some rationale as to why the second signal is considered redundant. Otherwise there's a bug in either the design or the implementation. A second signal received while the first is being 'handled' by the operating system or language run-time system. That one sounds odd to me. I would expect a signal received during the execution of a handler to be flagged and cause the handler to be called again after it returns. But then I'm used to the BSD signal model, which is relatively sane. A signal sent while the operating system is doing certain things to the application (including, sometimes, when it is swapped out or deep in I/O.) That sounds like an outright bug. I can't think of any earthly reason why the handler shouldn't be called eventually, if it remains installed and the process lives long enough. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
On 1/24/07, Martin v. Löwis [EMAIL PROTECTED] wrote: Gustavo Carneiro schrieb: What about http://www.python.org/sf/1564547 ? It tries to solve a different problem, but I think it also fixes this one; at least as much as possible with the braindead unix signal programming interface... I'm sceptical. It is way too much code for me to review, so I'm unable to comment whether it fixes the problem under discussion. Due to a bug, my patch didn't fix this bug. But I fixed the patch and it now fixes both my problem and Ulisses'. I feel that this problem should find a much simpler solution. The problem is that if you apply Ulisses' patch then my patch, Ulisses' changes will simply disappear because my patch handles signals and a much safer way, completely bypassing the 'add/make pending calls' system, since this system is patently *not* async safe, no matter how much you tweak it. Yes, I know my patch is not very small (though not that big either), but the signal module was in dire need of refactoring. And you can observe how much simpler the signal module becomes after that patch. Regards. -- Gustavo J. A. M. Carneiro The universe is always one step beyond logic. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
On Tue, Jan 23, 2007, Ulisses Furquim wrote: I've read some threads about signals in the archives and I was under the impression signals should work reliably on single-threaded applications. Am I right? I've thought about a way to fix this, but I don't know what is the current plan for signals support in python, so what can be done? This one looks like an oversight in Python code, and so is a bug, but it is important to note that signals do NOT work reliably under any Unix or Microsoft system. Inter alia, all of the following are likely to lead to lost signals: Two related signals received between two 'checkpoints' (i.e. when the signal is tested and cleared). You may only get one of them, and 'related' does not mean 'the same'. A second signal received while the first is being 'handled' by the operating system or language run-time system. A signal sent while the operating system is doing certain things to the application (including, sometimes, when it is swapped out or deep in I/O.) And there is more, some of which can cause program misbehaviour or crashes. You are also right that threading makes the situation a lot worse. Obviously, Unix and Microsoft systems depend on signals, so you can't simply regard them as hopelessly broken, but you can't assume that they are RELIABLE. All code should be designed to cope with the case of signals getting lost, if at all possible. Defending yourself against the other failures is an almost hopeless task, but luckily they are extremely rare except on specialist systems. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
On 1/24/07, Martin v. Löwis [EMAIL PROTECTED] wrote: I agree it's a bug, and I agree with your proposed analysis. Please try to come up with a patch (e.g. by putting a while(is_tripped) loop around the for loop). Also make sure you include test case. Ok. I was discussing this problem with a colleague of mine and he had a nice idea on how to fix this. We couldn't think of anything wrong with the solution, so I'm testing it right now and gonna write a test case afterwards. Thanks for contributing, No problem. :-) Regards, -- Ulisses ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
On 1/24/07, Nick Maclaren [EMAIL PROTECTED] wrote: Obviously, Unix and Microsoft systems depend on signals, so you can't simply regard them as hopelessly broken, but you can't assume that they are RELIABLE. All code should be designed to cope with the case of signals getting lost, if at all possible. Defending yourself against the other failures is an almost hopeless task, but luckily they are extremely rare except on specialist systems. I couldn't agree more. I might have misused the word reliably. At first glance I thought the signal was being lost but then I've discovered what was really happening. Regards, -- Ulisses ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
On 1/24/07, Martin v. Löwis [EMAIL PROTECTED] wrote: Ulisses Furquim schrieb: I've read some threads about signals in the archives and I was under the impression signals should work reliably on single-threaded applications. Am I right? I've thought about a way to fix this, but I don't know what is the current plan for signals support in python, so what can be done? I agree it's a bug, and I agree with your proposed analysis. Please try to come up with a patch (e.g. by putting a while(is_tripped) loop around the for loop). Also make sure you include test case. What about http://www.python.org/sf/1564547 ? It tries to solve a different problem, but I think it also fixes this one; at least as much as possible with the braindead unix signal programming interface... -- Gustavo J. A. M. Carneiro The universe is always one step beyond logic. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
Gustavo Carneiro schrieb: What about http://www.python.org/sf/1564547 ? It tries to solve a different problem, but I think it also fixes this one; at least as much as possible with the braindead unix signal programming interface... I'm sceptical. It is way too much code for me to review, so I'm unable to comment whether it fixes the problem under discussion. I feel that this problem should find a much simpler solution. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
On 1/24/07, Ulisses Furquim [EMAIL PROTECTED] wrote: On 1/24/07, Martin v. Löwis [EMAIL PROTECTED] wrote: I agree it's a bug, and I agree with your proposed analysis. Please try to come up with a patch (e.g. by putting a while(is_tripped) loop around the for loop). Also make sure you include test case. Ok. I was discussing this problem with a colleague of mine and he had a nice idea on how to fix this. We couldn't think of anything wrong with the solution, so I'm testing it right now and gonna write a test case afterwards. Here is the link for the bug with both the test program and first attempt to solve it. http://www.python.org/sf/1643738 Regards, -- Ulisses ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Problem with signals in a single threaded application
Hi,
I'm aware of the problems with signals in a multithreaded application,
but I was using signals in a single-threaded application and noticed
something that seemed wrong. Some signals were apparently being lost,
but when another signal came in the python handler for that lost
signal was being called.
The problem seems to be inside the signal module. The global variable
is_tripped is incremented every time a signal arrives. Then, inside
PyErr_CheckSignals() (the pending call that calls all python handlers
for signals that arrived) we can return immediately if is_tripped is
zero. If is_tripped is different than zero, we loop through all
signals calling the registered python handlers and after that we zero
is_tripped. This seems to be ok, but what happens if a signal arrives
after we've returned from its handler (or even after we've checked if
that signal arrived) and before we zero is_tripped? I guess we can
have a situation where is_tripped is zero but some Handlers[i].tripped
are not. In fact, I've inserted some debugging output and could see
that this actually happens and then I've written the following test
program to reproduce the problem.
#!/usr/bin/env python2.5
import sys
import os
import time
import signal
def alarm_handler(*args):
sys.stderr.write('alarmmm!\n')
def sigio_handler(*args):
sys.stderr.write('Entering SIGIO handler\n')
os.kill(os.getpid(), signal.SIGALRM)
sys.stderr.write('Leaving SIGIO handler\n')
signal.signal(signal.SIGIO, sigio_handler)
signal.signal(signal.SIGALRM, alarm_handler)
os.kill(os.getpid(), signal.SIGIO)
ini = time.time()
while True :
if time.time() - ini 3.0:
sys.stderr.write('Loop!\n')
ini = time.time()
When we run this program, the handler for the SIGALRM isn't called
after we return from the SIGIO handler. We return to our main loop
and print 'Loop!' every 3 seconds aprox. and the SIGALRM handler is
called only when another signal arrives (like when we hit Ctrl-C).
I've read some threads about signals in the archives and I was under
the impression signals should work reliably on single-threaded
applications. Am I right? I've thought about a way to fix this, but I
don't know what is the current plan for signals support in python, so
what can be done?
Best regards,
-- Ulisses
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
On Tue, Jan 23, 2007, Ulisses Furquim wrote: I've read some threads about signals in the archives and I was under the impression signals should work reliably on single-threaded applications. Am I right? I've thought about a way to fix this, but I don't know what is the current plan for signals support in python, so what can be done? For starters, please post a bug report to SF so that this issue doesn't get lost. -- Aahz ([EMAIL PROTECTED]) * http://www.pythoncraft.com/ Help a hearing-impaired person: http://rule6.info/hearing.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
Ulisses Furquim schrieb: I've read some threads about signals in the archives and I was under the impression signals should work reliably on single-threaded applications. Am I right? I've thought about a way to fix this, but I don't know what is the current plan for signals support in python, so what can be done? I agree it's a bug, and I agree with your proposed analysis. Please try to come up with a patch (e.g. by putting a while(is_tripped) loop around the for loop). Also make sure you include test case. Thanks for contributing, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
