Hi Alexander, On Thu, Jul 9, 2026 at 8:24 PM Xuneng Zhou <[email protected]> wrote: > > Hi! > > As Noah pointed out in [1], auxiliary processes are supported in the > facility, but proper cleanup is absent when they exit. I have attached > a patch trying to address that as suggested before getting sidetracked > for too long with the issue raised in [2].
I'd like to propose a series of patches to do some further clean-ups & enhancements. The first few are mostly mechanical, straightforward, and the last two are more subtle. I appreciate your inputs/thoughts on them. 1) Patch 1 to address the issue raised by Noah. 2) Patch 2 to fix two small docs mismatches. 3) Patch 3 to optimize the WaitLSNLock acquisition in deleteLSNWaiter. 4) Patch 4 to clarify lsn waiter cleanup after wakeup. 5) Patch 5 to address a rare issue of deregistration of waiters whose target lsn has reached but then flashed back. Currently, the waiters are removed from the heap by the wakers if their target lsn has reached. We offer no re-registration service in the reception desk for them assuming they won't need to extend their stay. This assumption holds if the lsn is monotonically increasing, which is true for most cases. However, there seems to be a catch for the wal write/flush lsn from the wal receiver side. In the cases of restarting wal receiver, the write/flush lsn could be reset to positions lagging behind(the attached restarting cases file summarized by Sol with inputs from me shows a matrix of this). One problematic interleaving, for reset cases 2–4 in the attached file, is the following: Let W be the old published write position, X the waiter’s target, S the new streaming segment start, and R the replay position, with: max(N, R) < X ≤ W [T0] A backend registers a standby_write waiter for target X and sleeps. [T1] The walreceiver publishes writtenUpto = W and calls WaitLSNWakeup(W). Because X ≤ W, the waker removes the waiter. from the heap, sets inHeap = false, and sets its latch. [T2] Before the waiter runs, streaming is restarted. RequestXLogStreaming() resets shared writtenUpto from W to S. The effective standby_write position is now: max(writtenUpto, replay) = max(S, R) = R. Therefore the effective position has regressed below X. [T3] The waiter consumes the old latch notification and rechecks its condition. It observes R < X. In the pre-fix code, it goes back to sleep without re-registering, even though it is no longer in the waiters heap. [T4] The walreceiver later receives and publishes WAL through X and calls WaitLSNWakeup() again. The waiter is absent from the heap, so it receives no notification and can remain asleep even though its target has now been reached. One possible fix for the issue that I have considered is to make write/flush lsn monotonic so it won't flash back once advanced. This seems to work well at first glance. However, it has two shortcomings: one is that it can blur the definition of write lsn and another is that it won't fix the cross-timeline lsn regression. Currently, writtenUpto is the current walreceiver stream’s write frontier. Giving that value monotonicity would turn it into the greatest numeric lsn ever written by any receiver incarnation. Cross-timeline lsn flashback is expected given the fork point is smaller than the pre-restart lsns. Extending it across timelines would incorrectly associate progress on an old timeline with the new timeline. The patch places the fix in the waiter side -- let the backend do a recheck and re-registration if the published lsn is regressed. Another aspect is the test, it's relatively simple to construct the failure case, but seems hard to do so deterministically. I spent a lot of time(maybe too much for an unconfirmed rare issue) trying to tame it in a simple way; it seems hard to orchestrate the startup process, wal receiver, waiting backend into above-like interleavings end-to-end without using complex synchronization like three injection points. The tricky part is to ensure max(N, R) < X ≤ W. We cannot stop the startup then the wal receiver to make sure that because we need the former to manipulate the later one. The current workaround is to add a synthetic helper for removing the heap node even if its target lsn has not actually reached, then trigger the real wake-up call by advancing the published lsn to check whether the waiter is notified and the wait completes. The rationale behind this is that the premature removal from the heap without re-registration is the underlying issue regardless of the specific lsn types and failure scenarios. That said, I am unsure whether this is a proper way or better alternatives exist. 6) Patch 6 to cover a missing wake-up point for primary-flush waiters [WIP] -- Regards, Xuneng Zhou HighGo Software Co., Ltd.
v1-0005-Re-register-LSN-waiters-after-stale-wakeups.patch
Description: Binary data
v1-0002-Fix-WAIT-FOR-LSN-documentation-examples.patch
Description: Binary data
v1-0001-Add-wait-for-lsn-process-exit-cleanup-callback.patch
Description: Binary data
v1-0004-Clarify-LSN-waiter-cleanup-after-wakeup.patch
Description: Binary data
v1-0003-Avoid-unnecessary-WaitLSNLock-acquisition-after-w.patch
Description: Binary data
Definitions:
- `W`: previous stream’s final written/flushed position.
- `S`: segment start of the new streaming request.
- `R`: replay position when the waiter rechecks.
- `N`: position published by the first post-reconnect WAL write, usually `S + received bytes`.
- Exposure `(L, W]`: targets `X` where `L < X ≤ W` can become orphaned pre-fix.
- An interval is empty when its lower bound is `≥ W`.
| # | Restart flavor | Exit flush | Reset block | `flushedUpto` transition | `writtenUpto` transition | Flush-mode exposure | Write-mode exposure |
|---|---|---:|---|---|---|---|---|
| 0 | First stream in new shared memory; full standby restart; first-ever stream after archive-only recovery | N/A | Runs as initialization | `0 → S` | `0 → S` | None | None |
| 1 | Walreceiver process death, next request on same TLI | Yes, pending WAL reaches `W` | Skipped: valid `receiveStart`, same `receivedTLI` | Stays `W`, then advances monotonically | Stays `W` until data arrives, then `W → N` unconditionally | None | `(max(N,R), W]`, but only after `N < W` is published |
| 2 | Walreceiver process death, next request on different TLI | Yes | Runs because TLI differs | `W → S` | `W → S` | `(max(S,R), W]` | `(max(S,R), W]` |
| 3 | In-process idle restart, same TLI after end-of-WAL | Yes | Runs because idle path cleared `receiveStart` | `W → S` | `W → S` | `(max(S,R), W]` | `(max(S,R), W]` |
| 4 | In-process idle restart with timeline switch | Yes | Runs because `receiveStart` is invalid and TLI differs | `W → S` | `W → S` | `(max(S,R), W]` | `(max(S,R), W]` |
| 5 | Hard walreceiver failure causing postmaster crash/full restart | Not guaranteed | Shared memory is discarded; next boot behaves like row 0 | Wiped | Wiped | None—waiters are killed | None—waiters are killed |
At every successful streaming start, the process-local state is separately initialized as:
```text
LogstreamResult.Write = LogstreamResult.Flush = replay position
```
Then the first received chunk changes local `Write` to its actual received endpoint.
Two important interpretations:
- In row 1, there is no write-mode regression until the first new chunk publishes `N` through `writtenUpto`. If `N ≥ W`, there is no exposure.
- In rows 2–4, the exact post-reset observable for both modes is `max(S,R)`, not necessarily `R`.
Once a pre-fix waiter rechecks inside an exposure interval and sleeps while `inHeap == false`, subsequent WAL or replay progress cannot wake it. Such progress only prevents the orphan if it reaches the target before that recheck.
Before going case by case, keep these three kinds of position separate:
```text
Shared write position: WalRcv->writtenUpto
Shared flush position: WalRcv->flushedUpto
Replay position: R
```
`WAIT FOR LSN` evaluates:
```text
standby_write current = max(writtenUpto, R)
standby_flush current = max(flushedUpto, R)
```
At every successful streaming start, the walreceiver also sets its private bookkeeping to replay:
```text
LogstreamResult.Write = R
LogstreamResult.Flush = R
```
Those local assignments do not directly update the shared fields.
The pre-fix race requires this setup:
1. A waiter wants target `X`.
2. An old wakeup at `W ≥ X` removes it from the heap and sets its latch.
3. Before the waiter rechecks, the effective position moves below `X`.
4. The waiter sees “not reached” and sleeps again, but it is no longer registered.
5. Future position wakeups cannot find it.
## Case 0: First stream in new shared memory
Examples:
- first streaming start after starting PostgreSQL;
- full standby restart;
- first-ever streaming attempt after archive-only recovery.
Initial shared state is:
```text
writtenUpto = 0
flushedUpto = 0
receiveStart = invalid
```
Recovery requests streaming from some position. `RequestXLogStreaming()` rounds that down to segment start `S`.
Because `receiveStart` is invalid, the initialization block runs:
```text
writtenUpto: 0 → S
flushedUpto: 0 → S
```
When streaming successfully starts:
```text
local Write = R
local Flush = R
```
The effective positions are therefore:
```text
write current = max(S, R)
flush current = max(S, R)
```
Why there is no exposure:
- The shared positions moved forward from zero, not backward from an old `W`.
- After a full server restart, old waiting backends do not survive.
- If recovery already reached `R` through archive replay, applying the replay floor means resetting the walreceiver fields cannot make the observable position lower than `R`.
So there is no stale old wakeup crossing this initialization.
## Case 1: Walreceiver process dies, same timeline
Examples:
- replication connection is lost;
- upstream restarts;
- `primary_conninfo` or slot configuration causes receiver replacement;
- walreceiver exits normally through its cleanup callback.
Before dying, `WalRcvDie()` flushes pending WAL:
```text
writtenUpto = W
flushedUpto = W
```
Importantly, it does not clear `receiveStart`.
When recovery requests streaming again on the same timeline:
```text
receiveStart is valid
receivedTLI == requested TLI
```
Therefore, the reset block is skipped.
Initially after restart:
```text
shared writtenUpto = W
shared flushedUpto = W
local Write = R
local Flush = R
```
At this point there is still no regression:
```text
write current = max(W, R)
flush current = max(W, R)
```
### The first new chunk
Streaming restarts from segment boundary `S`, so the upstream may resend WAL that the standby already possesses.
Suppose the first received chunk ends at `N`, where `N < W`.
`XLogWalRcvWrite()` performs an unconditional shared update:
```text
local Write: R → N
shared writtenUpto: W → N
```
The effective write position becomes:
```text
max(N, R)
```
Therefore, a previously popped write waiter can be exposed when:
```text
max(N, R) < X ≤ W
```
There is no write-mode exposure:
- before the first chunk publishes `N`;
- if `N ≥ W`;
- if `R ≥ W`.
### Why flush mode is safe in this case
Although local `Flush` was initialized to `R`, shared `flushedUpto` is updated with a monotonic guard:
```c
if (walrcv->flushedUpto < LogstreamResult.Flush)
walrcv->flushedUpto = LogstreamResult.Flush;
```
So a low local value cannot replace shared `W`:
```text
shared flushedUpto stays W
flush current = max(W, R)
```
Thus row 1 can regress only the write observable, not the flush observable.
## Case 2: Walreceiver process dies, new timeline
The old process again performs its normal exit flush:
```text
writtenUpto = W
flushedUpto = W
```
`receiveStart` survives the process death, but the new request has a different timeline:
```text
receivedTLI != requested TLI
```
That is enough to run the reset block:
```text
writtenUpto: W → S
flushedUpto: W → S
```
Unlike the ordinary flush update, these initialization assignments are not monotonic. They deliberately establish a new streaming starting point.
At successful stream start:
```text
local Write = R
local Flush = R
```
Both effective positions become:
```text
write current = max(S, R)
flush current = max(S, R)
```
Therefore both modes have the same exposure:
```text
max(S, R) < X ≤ W
```
If `max(S,R) ≥ W`, the interval is empty and there is no regression.
The timeline change also introduces the separate timeline-blindness issue: `WAIT FOR LSN` compares numeric LSNs only. That issue is distinct from the orphaned-waiter race.
## Case 3: In-process idle restart, same timeline
This is easy to miss because the walreceiver process does not die.
The upstream reaches end-of-WAL on the requested timeline. The existing walreceiver:
1. flushes received WAL through `W`;
2. closes its WAL file;
3. enters `WALRCV_WAITING`;
4. explicitly clears:
```text
receiveStart = invalid
receiveStartTLI = 0
```
Later, recovery asks the same process to restart streaming on the same timeline.
Even though the timeline did not change, `receiveStart` is invalid, so the reset block runs:
```text
writtenUpto: W → S
flushedUpto: W → S
```
The process-local `LogstreamResult` survived because this is the same OS process. However, when streaming successfully restarts, it is explicitly overwritten:
```text
local Write = R
local Flush = R
```
The effective positions are:
```text
write current = max(S, R)
flush current = max(S, R)
```
Both exposure intervals are:
```text
max(S, R) < X ≤ W
```
The important distinction from case 1 is:
```text
Case 1: process dies, receiveStart survives, reset skipped
Case 3: process survives, receiveStart is cleared, reset runs
```
That is why “same timeline means no reset” is incorrect.
## Case 4: In-process idle restart with timeline switch
This starts like case 3:
1. The walreceiver reaches the end of the old timeline.
2. It flushes through `W`.
3. It enters the waiting state.
4. It clears `receiveStart`.
Recovery then discovers or selects a new timeline and requests streaming from `S` on that timeline.
Now both reset conditions are true:
```text
receiveStart is invalid
receivedTLI differs from requested TLI
```
The result is still one reset:
```text
writtenUpto: W → S
flushedUpto: W → S
```
After local initialization to replay:
```text
write current = max(S, R)
flush current = max(S, R)
```
Both modes have exposure:
```text
max(S, R) < X ≤ W
```
From the waiter’s perspective, cases 2, 3, and 4 are equivalent: both shared positions can move from `W` to `S`.
The difference is what caused the reset:
```text
Case 2: new walreceiver process + timeline difference
Case 3: same process + receiveStart cleared
Case 4: same process + receiveStart cleared + timeline difference
```
## Case 5: Hard failure and full postmaster restart
Examples include a walreceiver crash that causes PostgreSQL’s postmaster to treat the server as crashed.
In this case, the normal `WalRcvDie()` cleanup might not run, so pending WAL is not guaranteed to be flushed.
However, PostgreSQL also terminates the other server processes. Consequently:
```text
old waiters are killed
shared memory is discarded
writtenUpto is discarded
flushedUpto is discarded
```
On the next server boot, walreceiver shared memory is initialized to zero and the next streaming request behaves like case 0.
There is no orphaned waiter because no backend survives across the shared-memory reset. This case is safe from the liveness bug only because the entire old process population disappears.
## Compact summary
```text
Case 0:
initialization, no backward transition
Case 1:
reset skipped
first repeated write may lower writtenUpto only
Cases 2–4:
reset runs
both writtenUpto and flushedUpto may move W → S
Case 5:
shared memory and waiters are both destroyed
```
And the precise exposure formulas are:
```text
Case 1 write: (max(N,R), W]
Case 1 flush: none
Cases 2–4 write: (max(S,R), W]
Cases 2–4 flush: (max(S,R), W]
```
