[HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)
On Sun, Jun 23, 2013 at 01:55:19PM +0900, MauMau wrote: From: Robert Haas robertmh...@gmail.com On Fri, Jun 21, 2013 at 10:02 PM, MauMau maumau...@gmail.com wrote: I'm comfortable with 5 seconds. We are talking about the interval between sending SIGQUIT to the children and then sending SIGKILL to them. In most situations, the backends should terminate immediately. However, as I said a few months ago, ereport() call in quickdie() can deadlock indefinitely. This is a PostgreSQL's bug. So let's fix that bug. Then we don't need this. [quoted part filtered to undo caps lock] There are two ways to fix that bug. You are suggesting 1 of the following two methods, aren't you? I suspect Robert meant to prevent the deadlock by limiting quickdie() to calling only async-signal-safe functions. Implementing that looked grotty when we last discussed it, though, and it would not help against a library or trusted PL function suspending SIGQUIT. 1. (Robert-san's idea) Upon receipt of immediate shutdown request, postmaster sends SIGKILL to its children. 2. (Tom-san's idea) Upon receipt of immediate shutdown request, postmaster first sends SIGQUIT to its children, wait a while for them to terminate, then sends SIGKILL to them. At first I proposed 1. Then Tom san suggested 2 so that the server is as friendly to the clients as now by notifying them of the server shutdown. I was convinced by that idea and chose 2. Basically, I think both ideas are right. They can solve the original problem. Agreed. However, re-considering the meaning of immediate shutdown, I feel 1 is a bit better, because trying to do something in quickdie() or somewhere does not match the idea of immediate. We may not have to be friendly to Perhaps so, but more important than the definition of the word immediate is what an immediate shutdown has long meant in PostgreSQL. All this time it has made some effort to send a message to the client. Note also that the patch under discussion affects both immediate shutdowns and the automatic reset in response to a backend crash. I think the latter is the more-important use case, and the message is nice to have there. the clients at the immediate shutdown. The code gets much simpler. In addition, it may be better that we similarly send SIGKILL in backend crash (FatalError) case, eliminate the use of SIGQUIT and remove quickdie() and other SIGQUIT handlers. My take is that the client message has some value, and we shouldn't just discard it to simplify the code slightly. Finishing the shutdown quickly has value, of course. The relative importance of those things should guide the choice of a timeout under method #2, but I don't see a rigorous way to draw the line. I feel five seconds is, if anything, too high. What about using deadlock_timeout? It constitutes a rough barometer on the system's tolerance for failure detection latency, and we already overload it by having it guide log_lock_waits. The default of 1s makes sense to me for this purpose, too. We can always add a separate immediate_shutdown_timeout if there's demand, but I don't expect such demand. (If we did add such a GUC, setting it to zero could be the way to request method 1. If some folks strongly desire method 1, that's probably the way to offer it.) Thanks, nm -- Noah Misch EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)
On 2013-06-21 23:19:27 -0400, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Jun 21, 2013 at 5:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: The traditional theory has been that that would be less robust, not more so. Child backends are (mostly) able to carry out queries whether or not the postmaster is around. I think that's the Tom Lane theory. The Robert Haas theory is that if the postmaster has died, there's no reason to suppose that it hasn't corrupted shared memory on the way down, or that the system isn't otherwise heavily fuxxored in some way. Eh? The postmaster does its level best never to touch shared memory (after initialization anyway). I am not sure that will never happen - but I think the chain of argument misses the main point. Normally we rely on the postmaster to kill off all other backends if a backend PANICs or segfaults for all the known reasons. As soon as there's no postmaster anymore we loose that capability. And *that* is scary imo. Especially as I would say the chance of getting PANICs or segfaults increases if there's no postmaster anymore since we might reach code branches we otherwise won't. True, you can't make new connections, but how does killing the existing children make that better? It allows you to start a new postmaster in a timely fashion, instead of waiting for an idle connection that may not ever terminate without operator intervention. And it's no always easy to figure out which cluster those backends belong to if there are multiple postgres instances running as the same user. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)
On 2013-06-21 16:56:57 -0400, Alvaro Herrera wrote: What we could do to improve the robustness a bit - at least on linux - is to prctl(PR_SET_PDEATHSIG, SIGKILL) which would cause children to be killed if the postmaster goes away... This is an interesting idea (even if it has no relationship to the patch at hand). Well, we could just set the deathsig to SIGKILL and exit normally - which would be a twoliner or so. Admittedly the cross-platform issue makes this approach not so great... Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)
On Fri, Jun 21, 2013 at 11:19 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think that's the Tom Lane theory. The Robert Haas theory is that if the postmaster has died, there's no reason to suppose that it hasn't corrupted shared memory on the way down, or that the system isn't otherwise heavily fuxxored in some way. Eh? The postmaster does its level best never to touch shared memory (after initialization anyway). And yet it certainly does - see pmsignal.c, for example. Besides which, as Andres points out, if the postmaster is dead, there is zip for a guarantee that some OTHER backend hasn't panicked. I think it's just ridiculous to suppose that the system can run in any sort of reasonable way without the postmaster. The whole reason why we work so hard to make sure that the postmaster doesn't die in the first place is because we need it to clean up when things go horribly wrong. If that cleanup function is important, then we need a living postmaster at all times. If it's not important, then our extreme paranoia about what operations the postmaster is permitted to engage in is overblown. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)
On 2013-06-20 22:36:45 -0400, Alvaro Herrera wrote: Noah Misch escribió: On Thu, Jun 20, 2013 at 12:33:25PM -0400, Alvaro Herrera wrote: MauMau escribi?: Here, reliable means that the database server is certainly shut down when pg_ctl returns, not telling a lie that I shut down the server processes for you, so you do not have to be worried that some postgres process might still remain and write to disk. I suppose reliable shutdown is crucial especially in HA cluster. If pg_ctl stop -mi gets stuck forever when there is an unkillable process (in what situations does this happen? OS bug, or NFS hard mount?), I think the DBA has to notice this situation from the unfinished pg_ctl, investigate the cause, and take corrective action. So you're suggesting that keeping postmaster up is a useful sign that the shutdown is not going well? I'm not really sure about this. What do others think? It would be valuable for pg_ctl -w -m immediate stop to have the property that an subsequent start attempt will not fail due to the presence of some backend still attached to shared memory. (Maybe that's true anyway or can be achieved a better way; I have not investigated.) Well, the only case where a process that's been SIGKILLed does not go away, as far as I know, is when it is in some uninterruptible sleep due to in-kernel operations that get stuck. Personally I have never seen this happen in any other case than some network filesystem getting disconnected, or a disk that doesn't respond. And whenever the filesystem starts to respond again, the process gets out of its sleep only to die due to the signal. Those are the situation in which it takes a really long time, yes. But there can be timing issues involved. Consider a backend that's currently stuck in a write() because it hit the dirtying limit. Say you have a postgres cluster that's currently slowing down to a crawl because it's overloaded and hitting the dirty limit. Somebody very well might just want to restart it with -m immediate. In that case a delay of a second or two till enough dirty memory has been written that write() can continue is enough for the postmaster to start up again and try to attach to shared memory where it will find the shared memory to be still in use. I don't really see any argument for *not* waiting. Sure it might take a bit longer till it's shut down, but if it didn't wait that will cause problems down the road. If we leave postmaster running after SIGKILLing its children, the only thing we can do is have it continue to SIGKILL processes continuously every few seconds until they die (or just sit around doing nothing until they all die). I don't think this will have a different effect than postmaster going away trusting the first SIGKILL to do its job eventually. I think it should just wait till all its child processes are dead. No need to repeat sending the signals - as you say, that won't help. What we could do to improve the robustness a bit - at least on linux - is to prctl(PR_SET_PDEATHSIG, SIGKILL) which would cause children to be killed if the postmaster goes away... Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)
Andres Freund escribió: On 2013-06-20 22:36:45 -0400, Alvaro Herrera wrote: If we leave postmaster running after SIGKILLing its children, the only thing we can do is have it continue to SIGKILL processes continuously every few seconds until they die (or just sit around doing nothing until they all die). I don't think this will have a different effect than postmaster going away trusting the first SIGKILL to do its job eventually. I think it should just wait till all its child processes are dead. No need to repeat sending the signals - as you say, that won't help. OK, I can buy that. So postmaster stays around waiting in ServerLoop until all children are gone; and if any persists for whatever reason, well, tough. What we could do to improve the robustness a bit - at least on linux - is to prctl(PR_SET_PDEATHSIG, SIGKILL) which would cause children to be killed if the postmaster goes away... This is an interesting idea (even if it has no relationship to the patch at hand). -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)
Alvaro Herrera alvhe...@2ndquadrant.com writes: Andres Freund escribió: What we could do to improve the robustness a bit - at least on linux - is to prctl(PR_SET_PDEATHSIG, SIGKILL) which would cause children to be killed if the postmaster goes away... This is an interesting idea (even if it has no relationship to the patch at hand). The traditional theory has been that that would be less robust, not more so. Child backends are (mostly) able to carry out queries whether or not the postmaster is around. True, you can't make new connections, but how does killing the existing children make that better? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)
On Fri, Jun 21, 2013 at 5:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: Andres Freund escribió: What we could do to improve the robustness a bit - at least on linux - is to prctl(PR_SET_PDEATHSIG, SIGKILL) which would cause children to be killed if the postmaster goes away... This is an interesting idea (even if it has no relationship to the patch at hand). The traditional theory has been that that would be less robust, not more so. Child backends are (mostly) able to carry out queries whether or not the postmaster is around. I think that's the Tom Lane theory. The Robert Haas theory is that if the postmaster has died, there's no reason to suppose that it hasn't corrupted shared memory on the way down, or that the system isn't otherwise heavily fuxxored in some way. True, you can't make new connections, but how does killing the existing children make that better? It allows you to start a new postmaster in a timely fashion, instead of waiting for an idle connection that may not ever terminate without operator intervention. Even if it were possible to start a new postmaster that attached to the existing shared memory segment and began spawning new children, I think I'd be heavily in favor of killing the old ones off first and doing a full system reset just for safety. But it isn't, so what you get is a crippled system that never recovers without operator intervention. And note that I'm not talking about pg_ctl restart; that fails because the children have the shmem segment still attached and the postmaster, which is the only thing listed in the PID file, is already dead. I'm talking about killall -9 postgres, at least. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)
Robert Haas robertmh...@gmail.com writes: On Fri, Jun 21, 2013 at 5:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: The traditional theory has been that that would be less robust, not more so. Child backends are (mostly) able to carry out queries whether or not the postmaster is around. I think that's the Tom Lane theory. The Robert Haas theory is that if the postmaster has died, there's no reason to suppose that it hasn't corrupted shared memory on the way down, or that the system isn't otherwise heavily fuxxored in some way. Eh? The postmaster does its level best never to touch shared memory (after initialization anyway). True, you can't make new connections, but how does killing the existing children make that better? It allows you to start a new postmaster in a timely fashion, instead of waiting for an idle connection that may not ever terminate without operator intervention. There may be something in that argument, but I find the other one completely bogus. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)
On Thu, Jun 20, 2013 at 12:33:25PM -0400, Alvaro Herrera wrote: MauMau escribi?: Here, reliable means that the database server is certainly shut down when pg_ctl returns, not telling a lie that I shut down the server processes for you, so you do not have to be worried that some postgres process might still remain and write to disk. I suppose reliable shutdown is crucial especially in HA cluster. If pg_ctl stop -mi gets stuck forever when there is an unkillable process (in what situations does this happen? OS bug, or NFS hard mount?), I think the DBA has to notice this situation from the unfinished pg_ctl, investigate the cause, and take corrective action. So you're suggesting that keeping postmaster up is a useful sign that the shutdown is not going well? I'm not really sure about this. What do others think? It would be valuable for pg_ctl -w -m immediate stop to have the property that an subsequent start attempt will not fail due to the presence of some backend still attached to shared memory. (Maybe that's true anyway or can be achieved a better way; I have not investigated.) -- Noah Misch EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)
Noah Misch escribió: On Thu, Jun 20, 2013 at 12:33:25PM -0400, Alvaro Herrera wrote: MauMau escribi?: Here, reliable means that the database server is certainly shut down when pg_ctl returns, not telling a lie that I shut down the server processes for you, so you do not have to be worried that some postgres process might still remain and write to disk. I suppose reliable shutdown is crucial especially in HA cluster. If pg_ctl stop -mi gets stuck forever when there is an unkillable process (in what situations does this happen? OS bug, or NFS hard mount?), I think the DBA has to notice this situation from the unfinished pg_ctl, investigate the cause, and take corrective action. So you're suggesting that keeping postmaster up is a useful sign that the shutdown is not going well? I'm not really sure about this. What do others think? It would be valuable for pg_ctl -w -m immediate stop to have the property that an subsequent start attempt will not fail due to the presence of some backend still attached to shared memory. (Maybe that's true anyway or can be achieved a better way; I have not investigated.) Well, the only case where a process that's been SIGKILLed does not go away, as far as I know, is when it is in some uninterruptible sleep due to in-kernel operations that get stuck. Personally I have never seen this happen in any other case than some network filesystem getting disconnected, or a disk that doesn't respond. And whenever the filesystem starts to respond again, the process gets out of its sleep only to die due to the signal. So a subsequent start attempt will either find that the filesystem is not responding, in which case it'll probably fail to work properly anyway (presumably the filesystem corresponds to part of the data directory), or that it has revived in which case the old backends have already gone away. If we leave postmaster running after SIGKILLing its children, the only thing we can do is have it continue to SIGKILL processes continuously every few seconds until they die (or just sit around doing nothing until they all die). I don't think this will have a different effect than postmaster going away trusting the first SIGKILL to do its job eventually. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
