Hi,
On 2025-01-13 12:20:39 +0000, Bertrand Drouvot wrote:
> > We have
> > multiple copies of code to go to FatalError, that doesn't seem great.
>
> + * FIXME: This should probably not have a copy fo the code to
> + * transition into FatalError state.
> + */
> + ereport(LOG,
> + (errmsg("WAL was shut down unexpectedly")));
>
> Indeed, might be worth extracting this into a helper function or such.
That is quite a rats nest of issues :(. There are quite a few oddities around
the related code. The attached series contains a few commits trying to unify
the paths transitioning to FatalError = true into the new HandleFatalError().
The current code has the weird behaviour of going through PM_WAIT_BACKENDS. I
left it like that for now. In fact more paths do so now, because we did so for
PM_WAIT_XLOG_SHUTDOWN but *didn't* do so for PM_WAIT_XLOG_ARCHIVAL, which
seems rather weird.
I suspect it'd be better to switch directly to PM_DEAD_END and have
HandleFatalError always do ConfigurePostmasterWaitSet(false), but that's left
for another time.
After the earlier commit that just moves the code I turned the existing if ()
into a switch so that whoever adds new states is told to look at that code by
the compiler.
Thoughts?
> > Another thing I started worrying about is that the commit added
> > PM_WAIT_CHECKPOINTER *after* PM_WAIT_DEAD_END. However, we have a few places
> > where we directly jump to PM_WAIT_DEAD_END in fatal situations:
Attached patch is implemented that way.
Greetings,
Andres Freund
>From 833ee00ea2a1341b2e20e88c96d8824c09189df2 Mon Sep 17 00:00:00 2001
From: Andres Freund <and...@anarazel.de>
Date: Fri, 10 Jan 2025 11:11:40 -0500
Subject: [PATCH v4 1/7] checkpointer: Request checkpoint via latch instead of
signal
The main reason for this is that a future commit would like to use SIGINT for
another purpose. But it's also a tad nicer and tad more efficient to use
SetLatch(), as it avoids a signal when checkpointer already is busy.
Reviewed-by: Bertrand Drouvot <bertranddrouvot...@gmail.com>
Reviewed-by: Nazir Bilal Yavuz <byavu...@gmail.com>
Discussion: https://postgr.es/m/kgng5nrvnlv335evmsuvpnh354rw7qyazl73kdysev2cr2v5zu@m3cfzxicm5kp
---
src/backend/postmaster/checkpointer.c | 60 +++++++++------------------
1 file changed, 19 insertions(+), 41 deletions(-)
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 9bfd0fd665c..dd2c8376c6e 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -159,9 +159,6 @@ static bool ImmediateCheckpointRequested(void);
static bool CompactCheckpointerRequestQueue(void);
static void UpdateSharedMemoryConfig(void);
-/* Signal handlers */
-static void ReqCheckpointHandler(SIGNAL_ARGS);
-
/*
* Main entry point for checkpointer process
@@ -191,7 +188,7 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
* tell us it's okay to shut down (via SIGUSR2).
*/
pqsignal(SIGHUP, SignalHandlerForConfigReload);
- pqsignal(SIGINT, ReqCheckpointHandler); /* request checkpoint */
+ pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
@@ -860,23 +857,6 @@ IsCheckpointOnSchedule(double progress)
}
-/* --------------------------------
- * signal handler routines
- * --------------------------------
- */
-
-/* SIGINT: set flag to run a normal checkpoint right away */
-static void
-ReqCheckpointHandler(SIGNAL_ARGS)
-{
- /*
- * The signaling process should have set ckpt_flags nonzero, so all we
- * need do is ensure that our main loop gets kicked out of any wait.
- */
- SetLatch(MyLatch);
-}
-
-
/* --------------------------------
* communication with backends
* --------------------------------
@@ -990,38 +970,36 @@ RequestCheckpoint(int flags)
SpinLockRelease(&CheckpointerShmem->ckpt_lck);
/*
- * Send signal to request checkpoint. It's possible that the checkpointer
- * hasn't started yet, or is in process of restarting, so we will retry a
- * few times if needed. (Actually, more than a few times, since on slow
- * or overloaded buildfarm machines, it's been observed that the
- * checkpointer can take several seconds to start.) However, if not told
- * to wait for the checkpoint to occur, we consider failure to send the
- * signal to be nonfatal and merely LOG it. The checkpointer should see
- * the request when it does start, with or without getting a signal.
+ * Set checkpointer's latch to request checkpoint. It's possible that the
+ * checkpointer hasn't started yet, so we will retry a few times if
+ * needed. (Actually, more than a few times, since on slow or overloaded
+ * buildfarm machines, it's been observed that the checkpointer can take
+ * several seconds to start.) However, if not told to wait for the
+ * checkpoint to occur, we consider failure to set the latch to be
+ * nonfatal and merely LOG it. The checkpointer should see the request
+ * when it does start, with or without the SetLatch().
*/
#define MAX_SIGNAL_TRIES 600 /* max wait 60.0 sec */
for (ntries = 0;; ntries++)
{
- if (CheckpointerShmem->checkpointer_pid == 0)
+ volatile PROC_HDR *procglobal = ProcGlobal;
+ ProcNumber checkpointerProc = procglobal->checkpointerProc;
+
+ if (checkpointerProc == INVALID_PROC_NUMBER)
{
if (ntries >= MAX_SIGNAL_TRIES || !(flags & CHECKPOINT_WAIT))
{
elog((flags & CHECKPOINT_WAIT) ? ERROR : LOG,
- "could not signal for checkpoint: checkpointer is not running");
- break;
- }
- }
- else if (kill(CheckpointerShmem->checkpointer_pid, SIGINT) != 0)
- {
- if (ntries >= MAX_SIGNAL_TRIES || !(flags & CHECKPOINT_WAIT))
- {
- elog((flags & CHECKPOINT_WAIT) ? ERROR : LOG,
- "could not signal for checkpoint: %m");
+ "could not notify checkpoint: checkpointer is not running");
break;
}
}
else
- break; /* signal sent successfully */
+ {
+ SetLatch(&GetPGProcByNumber(checkpointerProc)->procLatch);
+ /* notified successfully */
+ break;
+ }
CHECK_FOR_INTERRUPTS();
pg_usleep(100000L); /* wait 0.1 sec, then retry */
--
2.45.2.746.g06e570c0df.dirty
>From 67a3fbb10f46be4dfcaf5adeada4c8c30b5e01b0 Mon Sep 17 00:00:00 2001
From: Andres Freund <and...@anarazel.de>
Date: Mon, 13 Jan 2025 23:20:25 -0500
Subject: [PATCH v4 2/7] postmaster: Don't open-code TerminateChildren() in
HandleChildCrash()
After removing the duplication no user of sigquit_child() remains, therefore
remove it.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/postmaster/postmaster.c | 42 +++--------------------------
1 file changed, 4 insertions(+), 38 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 5f615d0f605..8153edc446c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -424,7 +424,6 @@ static int BackendStartup(ClientSocket *client_sock);
static void report_fork_failure_to_client(ClientSocket *client_sock, int errnum);
static CAC_state canAcceptConnections(BackendType backend_type);
static void signal_child(PMChild *pmchild, int signal);
-static void sigquit_child(PMChild *pmchild);
static bool SignalChildren(int signal, BackendTypeMask targetMask);
static void TerminateChildren(int signal);
static int CountChildren(BackendTypeMask targetMask);
@@ -2699,32 +2698,12 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
/*
* Signal all other child processes to exit. The crashed process has
* already been removed from ActiveChildList.
+ *
+ * We could exclude dead-end children here, but at least when sending
+ * SIGABRT it seems better to include them.
*/
if (take_action)
- {
- dlist_iter iter;
-
- dlist_foreach(iter, &ActiveChildList)
- {
- PMChild *bp = dlist_container(PMChild, elem, iter.cur);
-
- /* We do NOT restart the syslogger */
- if (bp == SysLoggerPMChild)
- continue;
-
- if (bp == StartupPMChild)
- StartupStatus = STARTUP_SIGNALED;
-
- /*
- * This backend is still alive. Unless we did so already, tell it
- * to commit hara-kiri.
- *
- * We could exclude dead-end children here, but at least when
- * sending SIGABRT it seems better to include them.
- */
- sigquit_child(bp);
- }
- }
+ TerminateChildren(send_abort_for_crash ? SIGABRT : SIGQUIT);
if (Shutdown != ImmediateShutdown)
FatalError = true;
@@ -3347,19 +3326,6 @@ signal_child(PMChild *pmchild, int signal)
#endif
}
-/*
- * Convenience function for killing a child process after a crash of some
- * other child process. We apply send_abort_for_crash to decide which signal
- * to send. Normally it's SIGQUIT -- and most other comments in this file are
- * written on the assumption that it is -- but developers might prefer to use
- * SIGABRT to collect per-child core dumps.
- */
-static void
-sigquit_child(PMChild *pmchild)
-{
- signal_child(pmchild, (send_abort_for_crash ? SIGABRT : SIGQUIT));
-}
-
/*
* Send a signal to the targeted children.
*/
--
2.45.2.746.g06e570c0df.dirty
>From c2d8fb9f2779ff828ee3dfb9050e5866aa4011bb Mon Sep 17 00:00:00 2001
From: Andres Freund <and...@anarazel.de>
Date: Mon, 13 Jan 2025 23:30:46 -0500
Subject: [PATCH v4 3/7] postmaster: Don't repeatedly transition to crashing
state
Previously HandleChildCrash() skipped logging and signalling child exits if
already in an immediate shutdown or FatalError, but still transitioned server
state in response to a crash. That's redundant.
To make it easier to combine different paths for entering FatalError state,
only do so once.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/postmaster/postmaster.c | 19 +++++++------------
1 file changed, 7 insertions(+), 12 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 8153edc446c..939b1b2ef82 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2676,8 +2676,6 @@ CleanupBackend(PMChild *bp,
static void
HandleChildCrash(int pid, int exitstatus, const char *procname)
{
- bool take_action;
-
/*
* We only log messages and send signals if this is the first process
* crash and we're not doing an immediate shutdown; otherwise, we're only
@@ -2685,15 +2683,13 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
* signaled children, nonzero exit status is to be expected, so don't
* clutter log.
*/
- take_action = !FatalError && Shutdown != ImmediateShutdown;
+ if (FatalError || Shutdown == ImmediateShutdown)
+ return;
- if (take_action)
- {
- LogChildExit(LOG, procname, pid, exitstatus);
- ereport(LOG,
- (errmsg("terminating any other active server processes")));
- SetQuitSignalReason(PMQUIT_FOR_CRASH);
- }
+ LogChildExit(LOG, procname, pid, exitstatus);
+ ereport(LOG,
+ (errmsg("terminating any other active server processes")));
+ SetQuitSignalReason(PMQUIT_FOR_CRASH);
/*
* Signal all other child processes to exit. The crashed process has
@@ -2702,8 +2698,7 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
* We could exclude dead-end children here, but at least when sending
* SIGABRT it seems better to include them.
*/
- if (take_action)
- TerminateChildren(send_abort_for_crash ? SIGABRT : SIGQUIT);
+ TerminateChildren(send_abort_for_crash ? SIGABRT : SIGQUIT);
if (Shutdown != ImmediateShutdown)
FatalError = true;
--
2.45.2.746.g06e570c0df.dirty
>From 34f4d2732672579e1cba046d9e3a7c344a133dcf Mon Sep 17 00:00:00 2001
From: Andres Freund <and...@anarazel.de>
Date: Tue, 14 Jan 2025 00:57:12 -0500
Subject: [PATCH v4 4/7] postmaster: Move code to switch into FatalError state
into function
There are two places switching to FatalError mode, behaving somewhat
differently. An upcoming commit will introduce a third. That doesn't seem seem
like a good idea.
This commit just moves the FatalError related code from HandleChildCrash()
into its own function, a subsequent commit will evolve the state machine
change to be suitable for other callers.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/postmaster/postmaster.c | 70 +++++++++++++++++++----------
1 file changed, 46 insertions(+), 24 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 939b1b2ef82..13d49eecd22 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2665,40 +2665,29 @@ CleanupBackend(PMChild *bp,
}
/*
- * HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, archiver, slot sync worker, or background worker.
- *
- * The objectives here are to clean up our local state about the child
- * process, and to signal all other remaining children to quickdie.
- *
- * The caller has already released its PMChild slot.
+ * Transition into FatalError state, in response to something bad having
+ * happened. Commonly the caller will have logged the reason for entering
+ * FatalError state.
*/
static void
-HandleChildCrash(int pid, int exitstatus, const char *procname)
+HandleFatalError(QuitSignalReason reason, bool consider_sigabrt)
{
- /*
- * We only log messages and send signals if this is the first process
- * crash and we're not doing an immediate shutdown; otherwise, we're only
- * here to update postmaster's idea of live processes. If we have already
- * signaled children, nonzero exit status is to be expected, so don't
- * clutter log.
- */
- if (FatalError || Shutdown == ImmediateShutdown)
- return;
+ int sigtosend;
+
+ SetQuitSignalReason(reason);
- LogChildExit(LOG, procname, pid, exitstatus);
- ereport(LOG,
- (errmsg("terminating any other active server processes")));
- SetQuitSignalReason(PMQUIT_FOR_CRASH);
+ if (consider_sigabrt && send_abort_for_crash)
+ sigtosend = SIGABRT;
+ else
+ sigtosend = SIGQUIT;
/*
- * Signal all other child processes to exit. The crashed process has
- * already been removed from ActiveChildList.
+ * Signal all other child processes to exit.
*
* We could exclude dead-end children here, but at least when sending
* SIGABRT it seems better to include them.
*/
- TerminateChildren(send_abort_for_crash ? SIGABRT : SIGQUIT);
+ TerminateChildren(sigtosend);
if (Shutdown != ImmediateShutdown)
FatalError = true;
@@ -2719,6 +2708,39 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
AbortStartTime = time(NULL);
}
+/*
+ * HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
+ * walwriter, autovacuum, archiver, slot sync worker, or background worker.
+ *
+ * The objectives here are to clean up our local state about the child
+ * process, and to signal all other remaining children to quickdie.
+ *
+ * The caller has already released its PMChild slot.
+ */
+static void
+HandleChildCrash(int pid, int exitstatus, const char *procname)
+{
+ /*
+ * We only log messages and send signals if this is the first process
+ * crash and we're not doing an immediate shutdown; otherwise, we're only
+ * here to update postmaster's idea of live processes. If we have already
+ * signaled children, nonzero exit status is to be expected, so don't
+ * clutter log.
+ */
+ if (FatalError || Shutdown == ImmediateShutdown)
+ return;
+
+ LogChildExit(LOG, procname, pid, exitstatus);
+ ereport(LOG,
+ (errmsg("terminating any other active server processes")));
+
+ /*
+ * Switch into error state. The crashed process has already been removed
+ * from ActiveChildList.
+ */
+ HandleFatalError(PMQUIT_FOR_CRASH, true);
+}
+
/*
* Log the death of a child process.
*/
--
2.45.2.746.g06e570c0df.dirty
>From 4df0e91aa56886db63e2a9552120b2c92def1982 Mon Sep 17 00:00:00 2001
From: Andres Freund <and...@anarazel.de>
Date: Tue, 14 Jan 2025 00:25:01 -0500
Subject: [PATCH v4 5/7] WIP: postmaster: Commonalize FatalError paths
This includes some behavioural changes:
- Previously PM_WAIT_XLOG_ARCHIVAL wasn't handled in HandleFatalError(), that
doesn't seem quite right.
- Failure to fork checkpointer now transitions through PM_WAIT_BACKENDS, like
child crashes. That's not necessarily great, but...
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/postmaster/postmaster.c | 60 +++++++++++++++++++++++------
1 file changed, 48 insertions(+), 12 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 13d49eecd22..6b3f047bd33 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2693,12 +2693,46 @@ HandleFatalError(QuitSignalReason reason, bool consider_sigabrt)
FatalError = true;
/* We now transit into a state of waiting for children to die */
- if (pmState == PM_RECOVERY ||
- pmState == PM_HOT_STANDBY ||
- pmState == PM_RUN ||
- pmState == PM_STOP_BACKENDS ||
- pmState == PM_WAIT_XLOG_SHUTDOWN)
- UpdatePMState(PM_WAIT_BACKENDS);
+ switch (pmState)
+ {
+ case PM_INIT:
+ /* shouldn't have any children */
+ Assert(false);
+ break;
+ case PM_STARTUP:
+ /* should have been handled in process_pm_child_exit */
+ Assert(false);
+ break;
+
+ /* wait for children to die */
+ case PM_RECOVERY:
+ case PM_HOT_STANDBY:
+ case PM_RUN:
+ case PM_STOP_BACKENDS:
+ UpdatePMState(PM_WAIT_BACKENDS);
+ break;
+
+ case PM_WAIT_BACKENDS:
+ /* there might be more backends to wait for */
+ break;
+
+ case PM_WAIT_XLOG_SHUTDOWN:
+ case PM_WAIT_XLOG_ARCHIVAL:
+ /*
+ * Note that we switch *back* to PM_WAIT_BACKENDS here. This way
+ * the PM_WAIT_BACKENDS && FatalError code in
+ * PostmasterStateMachine does not have to be duplicated.
+ *
+ * XXX: This seems rather ugly, but it's not obvious if the
+ * alternative is better.
+ */
+ UpdatePMState(PM_WAIT_BACKENDS);
+ break;
+
+ case PM_WAIT_DEAD_END:
+ case PM_NO_CHILDREN:
+ break;
+ }
/*
* .. and if this doesn't happen quickly enough, now the clock is ticking
@@ -2836,6 +2870,9 @@ PostmasterStateMachine(void)
* PM_WAIT_BACKENDS, but we signal the processes first, before waiting for
* them. Treating it as a distinct pmState allows us to share this code
* across multiple shutdown code paths.
+ *
+ * Note that HandleFatalError() switches to PM_WAIT_BACKENDS even if we
+ * were, before the fatal error, in a "more advanced" state.
*/
if (pmState == PM_STOP_BACKENDS || pmState == PM_WAIT_BACKENDS)
{
@@ -2967,13 +3004,12 @@ PostmasterStateMachine(void)
* We don't consult send_abort_for_crash here, as it's
* unlikely that dumping cores would illuminate the reason
* for checkpointer fork failure.
+ *
+ * XXX: Is it worth inventing a different PMQUIT value
+ * that signals that the cluster is in a bad state,
+ * without a process having crashed?
*/
- FatalError = true;
- UpdatePMState(PM_WAIT_DEAD_END);
- ConfigurePostmasterWaitSet(false);
-
- /* Kill the walsenders and archiver too */
- SignalChildren(SIGQUIT, btmask_all_except(B_LOGGER));
+ HandleFatalError(PMQUIT_FOR_CRASH, false);
}
}
}
--
2.45.2.746.g06e570c0df.dirty
>From a42558e2b42fd7c029477b53904a3aef3cc0b7d1 Mon Sep 17 00:00:00 2001
From: Andres Freund <and...@anarazel.de>
Date: Tue, 14 Jan 2025 00:31:03 -0500
Subject: [PATCH v4 6/7] postmaster: Adjust which processes we expect to have
exited
Comments and code stated that we expect checkpointer to have been signalled in
case of immediate shutdown / fatal errors, but didn't treat archiver and
walsenders the same. That doesn't seem right.
I had started digging through the history to see where this oddity was
introduced, but it's not the fault of a single commit.
Instead treat archiver, checkpointer, and walsenders the same.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/postmaster/postmaster.c | 28 ++++++++++++++++++----------
1 file changed, 18 insertions(+), 10 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 6b3f047bd33..de234e7994f 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2905,16 +2905,20 @@ PostmasterStateMachine(void)
/*
* If we are doing crash recovery or an immediate shutdown then we
- * expect the checkpointer to exit as well, otherwise not.
+ * expect archiver, checkpointer and walsender to exit as well,
+ * otherwise not.
*/
if (FatalError || Shutdown >= ImmediateShutdown)
- targetMask = btmask_add(targetMask, B_CHECKPOINTER);
+ targetMask = btmask_add(targetMask,
+ B_CHECKPOINTER,
+ B_ARCHIVER,
+ B_WAL_SENDER);
/*
- * Walsenders and archiver will continue running; they will be
- * terminated later after writing the checkpoint record. We also let
- * dead-end children to keep running for now. The syslogger process
- * exits last.
+ * Normally walsenders and archiver will continue running; they will
+ * be terminated later after writing the checkpoint record. We also
+ * let dead-end children to keep running for now. The syslogger
+ * process exits last.
*
* This assertion checks that we have covered all backend types,
* either by including them in targetMask, or by noting here that they
@@ -2925,13 +2929,17 @@ PostmasterStateMachine(void)
BackendTypeMask remainMask = BTYPE_MASK_NONE;
remainMask = btmask_add(remainMask,
- B_WAL_SENDER,
- B_ARCHIVER,
B_DEAD_END_BACKEND,
B_LOGGER);
- /* checkpointer may or may not be in targetMask already */
- remainMask = btmask_add(remainMask, B_CHECKPOINTER);
+ /*
+ * Archiver, checkpointer and walsender may or may not be in
+ * targetMask already.
+ */
+ remainMask = btmask_add(remainMask,
+ B_ARCHIVER,
+ B_CHECKPOINTER,
+ B_WAL_SENDER);
/* these are not real postmaster children */
remainMask = btmask_add(remainMask,
--
2.45.2.746.g06e570c0df.dirty
>From d95b0f1b0d52c3f1db191f3d79e62e3489bf73ee Mon Sep 17 00:00:00 2001
From: Andres Freund <and...@anarazel.de>
Date: Fri, 10 Jan 2025 11:45:13 -0500
Subject: [PATCH v4 7/7] Change shutdown sequence to terminate checkpointer
last
The main motivation for this change is to have a process that can serialize
stats after all other processes have terminated. Serializing stats already
happens in checkpointer, even though walsenders can be active longer.
The only reason the current state does not actively cause problems is that
walsender currently generate any stats. However, there is a patch to change
that.
Another need for this change originates in the AIO patchset, where IO
workers (which, in some edge cases, can emit stats of their own) need to run
while the shutdown checkpoint is being written.
This commit changes the shutdown sequence so checkpointer is signalled (via
SIGINT) to trigger writing the shutdown checkpoint without terminating
it. Once checkpointer wrote the checkpoint it will wait for a termination
signal (SIGUSR2, as before).
Postmaster now triggers the shutdown checkpoint via SIGINT, where we
previously did so by terminating checkpointer. Checkpointer now is terminated
after all children other than dead-end ones have been terminated, tracked
using the new PM_WAIT_CHECKPOINTER PMState.
Reviewed-by: Heikki Linnakangas <hlinn...@iki.fi>
Reviewed-by: Bertrand Drouvot <bertranddrouvot...@gmail.com>
Discussion: https://postgr.es/m/kgng5nrvnlv335evmsuvpnh354rw7qyazl73kdysev2cr2v5zu@m3cfzxicm5kp
---
src/include/storage/pmsignal.h | 3 +-
src/backend/postmaster/checkpointer.c | 125 +++++++++++----
src/backend/postmaster/postmaster.c | 143 +++++++++++++-----
.../utils/activity/wait_event_names.txt | 1 +
4 files changed, 200 insertions(+), 72 deletions(-)
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 3fbe5bf1136..d84a383047e 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -40,9 +40,10 @@ typedef enum
PMSIGNAL_BACKGROUND_WORKER_CHANGE, /* background worker state change */
PMSIGNAL_START_WALRECEIVER, /* start a walreceiver */
PMSIGNAL_ADVANCE_STATE_MACHINE, /* advance postmaster's state machine */
+ PMSIGNAL_XLOG_IS_SHUTDOWN, /* ShutdownXLOG() completed */
} PMSignalReason;
-#define NUM_PMSIGNALS (PMSIGNAL_ADVANCE_STATE_MACHINE+1)
+#define NUM_PMSIGNALS (PMSIGNAL_XLOG_IS_SHUTDOWN+1)
/*
* Reasons why the postmaster would send SIGQUIT to its children.
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index dd2c8376c6e..767bf9f5cf8 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -10,10 +10,13 @@
* fill WAL segments; the checkpointer itself doesn't watch for the
* condition.)
*
- * Normal termination is by SIGUSR2, which instructs the checkpointer to
- * execute a shutdown checkpoint and then exit(0). (All backends must be
- * stopped before SIGUSR2 is issued!) Emergency termination is by SIGQUIT;
- * like any backend, the checkpointer will simply abort and exit on SIGQUIT.
+ * The normal termination sequence is that checkpointer is instructed to
+ * execute the shutdown checkpoint by SIGINT. After that checkpointer waits
+ * to be terminated via SIGUSR2, which instructs the checkpointer to exit(0).
+ * All backends must be stopped before SIGINT or SIGUSR2 is issued!
+ *
+ * Emergency termination is by SIGQUIT; like any backend, the checkpointer
+ * will simply abort and exit on SIGQUIT.
*
* If the checkpointer exits unexpectedly, the postmaster treats that the same
* as a backend crash: shared memory may be corrupted, so remaining backends
@@ -51,6 +54,7 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
#include "storage/shmem.h"
@@ -141,6 +145,7 @@ double CheckPointCompletionTarget = 0.9;
* Private state
*/
static bool ckpt_active = false;
+static volatile sig_atomic_t ShutdownXLOGPending = false;
/* these values are valid when ckpt_active is true: */
static pg_time_t ckpt_start_time;
@@ -159,6 +164,9 @@ static bool ImmediateCheckpointRequested(void);
static bool CompactCheckpointerRequestQueue(void);
static void UpdateSharedMemoryConfig(void);
+/* Signal handlers */
+static void ReqShutdownXLOG(SIGNAL_ARGS);
+
/*
* Main entry point for checkpointer process
@@ -188,7 +196,7 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
* tell us it's okay to shut down (via SIGUSR2).
*/
pqsignal(SIGHUP, SignalHandlerForConfigReload);
- pqsignal(SIGINT, SIG_IGN);
+ pqsignal(SIGINT, ReqShutdownXLOG);
pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
@@ -211,8 +219,11 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
* process during a normal shutdown, and since checkpointer is shut down
* very late...
*
- * Walsenders are shut down after the checkpointer, but currently don't
- * report stats. If that changes, we need a more complicated solution.
+ * While e.g. walsenders are active after the shutdown checkpoint has been
+ * written (and thus could produce more stats), checkpointer stays around
+ * after the shutdown checkpoint has been written. postmaster will only
+ * signal checkpointer to exit after all processes that could emit stats
+ * have been shut down.
*/
before_shmem_exit(pgstat_before_server_shutdown, 0);
@@ -327,7 +338,7 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
ProcGlobal->checkpointerProc = MyProcNumber;
/*
- * Loop forever
+ * Loop until we've been asked to write shutdown checkpoint or terminate.
*/
for (;;)
{
@@ -346,7 +357,10 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
* Process any requests or signals received recently.
*/
AbsorbSyncRequests();
+
HandleCheckpointerInterrupts();
+ if (ShutdownXLOGPending || ShutdownRequestPending)
+ break;
/*
* Detect a pending checkpoint request by checking whether the flags
@@ -517,8 +531,13 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
ckpt_active = false;
- /* We may have received an interrupt during the checkpoint. */
+ /*
+ * We may have received an interrupt during the checkpoint and the
+ * latch might have been reset (e.g. in CheckpointWriteDelay).
+ */
HandleCheckpointerInterrupts();
+ if (ShutdownXLOGPending || ShutdownRequestPending)
+ break;
}
/* Check for archive_timeout and switch xlog files if necessary. */
@@ -557,6 +576,56 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
cur_timeout * 1000L /* convert to ms */ ,
WAIT_EVENT_CHECKPOINTER_MAIN);
}
+
+ /*
+ * From here on, elog(ERROR) should end with exit(1), not send control
+ * back to the sigsetjmp block above.
+ */
+ ExitOnAnyError = true;
+
+ if (ShutdownXLOGPending)
+ {
+ /*
+ * Close down the database.
+ *
+ * Since ShutdownXLOG() creates restartpoint or checkpoint, and
+ * updates the statistics, increment the checkpoint request and flush
+ * out pending statistic.
+ */
+ PendingCheckpointerStats.num_requested++;
+ ShutdownXLOG(0, 0);
+ pgstat_report_checkpointer();
+ pgstat_report_wal(true);
+
+ /*
+ * Tell postmaster that we're done.
+ */
+ SendPostmasterSignal(PMSIGNAL_XLOG_IS_SHUTDOWN);
+ }
+
+ /*
+ * Wait until we're asked to shut down. By separating the writing of the
+ * shutdown checkpoint from checkpointer exiting, checkpointer can perform
+ * some should-be-as-late-as-possible work like writing out stats.
+ */
+ for (;;)
+ {
+ /* Clear any already-pending wakeups */
+ ResetLatch(MyLatch);
+
+ HandleCheckpointerInterrupts();
+
+ if (ShutdownRequestPending)
+ break;
+
+ (void) WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_EXIT_ON_PM_DEATH,
+ 0,
+ WAIT_EVENT_CHECKPOINTER_SHUTDOWN);
+ }
+
+ /* Normal exit from the checkpointer is here */
+ proc_exit(0); /* done */
}
/*
@@ -586,29 +655,6 @@ HandleCheckpointerInterrupts(void)
*/
UpdateSharedMemoryConfig();
}
- if (ShutdownRequestPending)
- {
- /*
- * From here on, elog(ERROR) should end with exit(1), not send control
- * back to the sigsetjmp block above
- */
- ExitOnAnyError = true;
-
- /*
- * Close down the database.
- *
- * Since ShutdownXLOG() creates restartpoint or checkpoint, and
- * updates the statistics, increment the checkpoint request and flush
- * out pending statistic.
- */
- PendingCheckpointerStats.num_requested++;
- ShutdownXLOG(0, 0);
- pgstat_report_checkpointer();
- pgstat_report_wal(true);
-
- /* Normal exit from the checkpointer is here */
- proc_exit(0); /* done */
- }
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
@@ -729,6 +775,7 @@ CheckpointWriteDelay(int flags, double progress)
* in which case we just try to catch up as quickly as possible.
*/
if (!(flags & CHECKPOINT_IMMEDIATE) &&
+ !ShutdownXLOGPending &&
!ShutdownRequestPending &&
!ImmediateCheckpointRequested() &&
IsCheckpointOnSchedule(progress))
@@ -857,6 +904,20 @@ IsCheckpointOnSchedule(double progress)
}
+/* --------------------------------
+ * signal handler routines
+ * --------------------------------
+ */
+
+/* SIGINT: set flag to trigger writing of shutdown checkpoint */
+static void
+ReqShutdownXLOG(SIGNAL_ARGS)
+{
+ ShutdownXLOGPending = true;
+ SetLatch(MyLatch);
+}
+
+
/* --------------------------------
* communication with backends
* --------------------------------
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index de234e7994f..d38251dea8f 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -334,6 +334,7 @@ typedef enum
* ckpt */
PM_WAIT_XLOG_ARCHIVAL, /* waiting for archiver and walsenders to
* finish */
+ PM_WAIT_CHECKPOINTER, /* waiting for checkpointer to shut down */
PM_WAIT_DEAD_END, /* waiting for dead-end children to exit */
PM_NO_CHILDREN, /* all important children have exited */
} PMState;
@@ -2354,35 +2355,19 @@ process_pm_child_exit(void)
{
ReleasePostmasterChildSlot(CheckpointerPMChild);
CheckpointerPMChild = NULL;
- if (EXIT_STATUS_0(exitstatus) && pmState == PM_WAIT_XLOG_SHUTDOWN)
+ if (EXIT_STATUS_0(exitstatus) && pmState == PM_WAIT_CHECKPOINTER)
{
/*
* OK, we saw normal exit of the checkpointer after it's been
- * told to shut down. We expect that it wrote a shutdown
- * checkpoint. (If for some reason it didn't, recovery will
- * occur on next postmaster start.)
+ * told to shut down. We know checkpointer wrote a shutdown
+ * checkpoint, otherwise we'd still be in
+ * PM_WAIT_XLOG_SHUTDOWN state.
*
- * At this point we should have no normal backend children
- * left (else we'd not be in PM_WAIT_XLOG_SHUTDOWN state) but
- * we might have dead-end children to wait for.
- *
- * If we have an archiver subprocess, tell it to do a last
- * archive cycle and quit. Likewise, if we have walsender
- * processes, tell them to send any remaining WAL and quit.
- */
- Assert(Shutdown > NoShutdown);
-
- /* Waken archiver for the last time */
- if (PgArchPMChild != NULL)
- signal_child(PgArchPMChild, SIGUSR2);
-
- /*
- * Waken walsenders for the last time. No regular backends
- * should be around anymore.
+ * At this point only dead-end children should be left.
*/
- SignalChildren(SIGUSR2, btmask(B_WAL_SENDER));
-
- UpdatePMState(PM_WAIT_XLOG_ARCHIVAL);
+ UpdatePMState(PM_WAIT_DEAD_END);
+ ConfigurePostmasterWaitSet(false);
+ SignalChildren(SIGTERM, btmask_all_except(B_LOGGER));
}
else
{
@@ -2718,6 +2703,7 @@ HandleFatalError(QuitSignalReason reason, bool consider_sigabrt)
case PM_WAIT_XLOG_SHUTDOWN:
case PM_WAIT_XLOG_ARCHIVAL:
+ case PM_WAIT_CHECKPOINTER:
/*
* Note that we switch *back* to PM_WAIT_BACKENDS here. This way
* the PM_WAIT_BACKENDS && FatalError code in
@@ -2979,9 +2965,9 @@ PostmasterStateMachine(void)
SignalChildren(SIGQUIT, btmask(B_DEAD_END_BACKEND));
/*
- * We already SIGQUIT'd walsenders and the archiver, if any,
- * when we started immediate shutdown or entered FatalError
- * state.
+ * We already SIGQUIT'd archiver, checkpointer and walsenders,
+ * if any, when we started immediate shutdown or entered
+ * FatalError state.
*/
}
else
@@ -2995,10 +2981,10 @@ PostmasterStateMachine(void)
/* Start the checkpointer if not running */
if (CheckpointerPMChild == NULL)
CheckpointerPMChild = StartChildProcess(B_CHECKPOINTER);
- /* And tell it to shut down */
+ /* And tell it to write the shutdown checkpoint */
if (CheckpointerPMChild != NULL)
{
- signal_child(CheckpointerPMChild, SIGUSR2);
+ signal_child(CheckpointerPMChild, SIGINT);
UpdatePMState(PM_WAIT_XLOG_SHUTDOWN);
}
else
@@ -3023,22 +3009,39 @@ PostmasterStateMachine(void)
}
}
+ /*
+ * The state transition from PM_WAIT_XLOG_SHUTDOWN to
+ * PM_WAIT_XLOG_ARCHIVAL is in proccess_pm_pmsignal(), in response to
+ * PMSIGNAL_XLOG_IS_SHUTDOWN.
+ */
+
if (pmState == PM_WAIT_XLOG_ARCHIVAL)
{
/*
- * PM_WAIT_XLOG_ARCHIVAL state ends when there's no other children
- * than dead-end children left. There shouldn't be any regular
- * backends left by now anyway; what we're really waiting for is
- * walsenders and archiver.
+ * PM_WAIT_XLOG_ARCHIVAL state ends when there's no children other
+ * than checkpointer and dead-end children left. There shouldn't be
+ * any regular backends left by now anyway; what we're really waiting
+ * for is for walsenders and archiver to exit.
*/
- if (CountChildren(btmask_all_except(B_LOGGER, B_DEAD_END_BACKEND)) == 0)
+ if (CountChildren(btmask_all_except(B_CHECKPOINTER, B_LOGGER, B_DEAD_END_BACKEND)) == 0)
{
- UpdatePMState(PM_WAIT_DEAD_END);
- ConfigurePostmasterWaitSet(false);
- SignalChildren(SIGTERM, btmask_all_except(B_LOGGER));
+ UpdatePMState(PM_WAIT_CHECKPOINTER);
+
+ /*
+ * Now that everyone important is gone, tell checkpointer to shut
+ * down too. That allows checkpointer to perform some last bits of
+ * cleanup without other processes interfering.
+ */
+ if (CheckpointerPMChild != NULL)
+ signal_child(CheckpointerPMChild, SIGUSR2);
}
}
+ /*
+ * The state transition from PM_WAIT_CHECKPOINTER to PM_WAIT_DEAD_END is
+ * in proccess_pm_child_exit().
+ */
+
if (pmState == PM_WAIT_DEAD_END)
{
/*
@@ -3175,6 +3178,7 @@ pmstate_name(PMState state)
PM_TOSTR_CASE(PM_WAIT_XLOG_SHUTDOWN);
PM_TOSTR_CASE(PM_WAIT_XLOG_ARCHIVAL);
PM_TOSTR_CASE(PM_WAIT_DEAD_END);
+ PM_TOSTR_CASE(PM_WAIT_CHECKPOINTER);
PM_TOSTR_CASE(PM_NO_CHILDREN);
}
#undef PM_TOSTR_CASE
@@ -3592,6 +3596,8 @@ ExitPostmaster(int status)
static void
process_pm_pmsignal(void)
{
+ bool request_state_update = false;
+
pending_pm_pmsignal = false;
ereport(DEBUG2,
@@ -3703,9 +3709,67 @@ process_pm_pmsignal(void)
WalReceiverRequested = true;
}
+ if (CheckPostmasterSignal(PMSIGNAL_XLOG_IS_SHUTDOWN))
+ {
+ /* Checkpointer completed the shutdown checkpoint */
+ if (pmState == PM_WAIT_XLOG_SHUTDOWN)
+ {
+ /*
+ * If we have an archiver subprocess, tell it to do a last archive
+ * cycle and quit. Likewise, if we have walsender processes, tell them
+ * to send any remaining WAL and quit.
+ */
+ Assert(Shutdown > NoShutdown);
+
+ /* Waken archiver for the last time */
+ if (PgArchPMChild != NULL)
+ signal_child(PgArchPMChild, SIGUSR2);
+
+ /*
+ * Waken walsenders for the last time. No regular backends should be
+ * around anymore.
+ */
+ SignalChildren(SIGUSR2, btmask(B_WAL_SENDER));
+
+ UpdatePMState(PM_WAIT_XLOG_ARCHIVAL);
+ }
+ else if (!FatalError && Shutdown != ImmediateShutdown)
+ {
+ /*
+ * Checkpointer only ought to perform the shutdown checkpoint
+ * during shutdown. If somehow checkpointer did so in another
+ * situation, we have no choice but to crash-restart.
+ *
+ * It's possible however that we get PMSIGNAL_XLOG_IS_SHUTDOWN
+ * outside of PM_WAIT_XLOG_SHUTDOWN if an orderly shutdown was
+ * "interrupted" by a crash or an immediate shutdown.
+ */
+ ereport(LOG,
+ (errmsg("WAL was shut down unexpectedly")));
+
+ /*
+ * Doesn't seem likely to help to take send_abort_for_crash into
+ * account here.
+ */
+ HandleFatalError(PMQUIT_FOR_CRASH, false);
+ }
+
+ /*
+ * Need to run PostmasterStateMachine() to check if we already can go
+ * to the next state.
+ */
+ request_state_update = true;
+ }
+
/*
* Try to advance postmaster's state machine, if a child requests it.
- *
+ */
+ if (CheckPostmasterSignal(PMSIGNAL_ADVANCE_STATE_MACHINE))
+ {
+ request_state_update = true;
+ }
+
+ /*
* Be careful about the order of this action relative to this function's
* other actions. Generally, this should be after other actions, in case
* they have effects PostmasterStateMachine would need to know about.
@@ -3713,7 +3777,7 @@ process_pm_pmsignal(void)
* cannot have any (immediate) effect on the state machine, but does
* depend on what state we're in now.
*/
- if (CheckPostmasterSignal(PMSIGNAL_ADVANCE_STATE_MACHINE))
+ if (request_state_update)
{
PostmasterStateMachine();
}
@@ -4024,6 +4088,7 @@ bgworker_should_start_now(BgWorkerStartTime start_time)
switch (pmState)
{
case PM_NO_CHILDREN:
+ case PM_WAIT_CHECKPOINTER:
case PM_WAIT_DEAD_END:
case PM_WAIT_XLOG_ARCHIVAL:
case PM_WAIT_XLOG_SHUTDOWN:
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 0b53cba807d..e199f071628 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -56,6 +56,7 @@ AUTOVACUUM_MAIN "Waiting in main loop of autovacuum launcher process."
BGWRITER_HIBERNATE "Waiting in background writer process, hibernating."
BGWRITER_MAIN "Waiting in main loop of background writer process."
CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
+CHECKPOINTER_SHUTDOWN "Waiting for checkpointer process to be terminated."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
--
2.45.2.746.g06e570c0df.dirty
