Hello, We've found a bug where a streaming standby gets stuck if its walreceiver restarts while the last record it has received ends 8, 16 or 24 bytes before the end of a WAL segment, so that the next record header reaches the segment boundary. This issue reproduces on current master and the code is the same on all supported branches.
To read that header, XLogPageRead() asks WaitForWALToBecomeAvailable() for WAL up to the end of the header, or the end of the page if the header crosses it. Here that is exactly the end of the segment. WaitForWALToBecomeAvailable() passes that end position (RecPtr) to RequestXLogStreaming() as the start point, and RequestXLogStreaming() rounds it down to a segment start. That's a no-op here, so the new walreceiver starts streaming at the next segment and never fetches the end of the current one: LOG: walreceiver requested start point 0/05000000 on timeline 1 is ahead of the upstream server's flush position 0/04FFFFF8, waiting LOG: started streaming WAL from primary at 0/05000000 on timeline 1 LOG: invalid record length at 0/04FFFFF8: expected at least 24, got 0 [... repeated forever] The attached patch starts streaming at the beginning of the segment holding RecPtr - 1, which is the segment of the page being read, as c06380e976 intended when it switched to RecPtr. Nothing changes unless RecPtr is exactly on a segment boundary. It adds a TAP test, t/061_walrcv_restart_segment_boundary.pl, which pads WAL so that the last record ends 8 bytes before a boundary, restarts the walreceiver by changing primary_conninfo, and checks where streaming restarts. It fails on master without the fix and passes with it; I also checked gaps of 16 and 24 bytes by changing the test. src/test/recovery passes with make and meson. Related threads: - Konstantin's applyFlushedUpto patch [1] lets recovery get past WAL that is flushed but unreadable, and Jeff asked there how a standby ends up with such WAL. This is one way. With that patch the test recovers after a re-read; with this one the skip doesn't happen. These patches are independent of each other. - Paul's fix for the end LSN of an xlog-switch record [2] is a primary-side way into the same state. - Since b614de4876 [3], the walreceiver waits instead of failing when its start point is at most one segment ahead of the upstream, so this case now gets stuck without an error. Before that, START_REPLICATION failed and was retried until the primary had written past the boundary, and then the same skip happened. I think it makes sense to backport the fix since it applies to REL_15_STABLE through REL_19_STABLE. REL_14_STABLE has the same code in xlog.c, and I can send a version for it. [1] https://postgr.es/m/[email protected] [2] https://postgr.es/m/calj2acvsquwdyovodf4qfwfr9ye4pwcqzjoinigockpazjm...@mail.gmail.com [3] https://postgr.es/m/ca+nrd2ctutkkx5wxvzengtyyzbao6zv8k+tri-r0fblfuoy...@mail.gmail.com Thanks, David Shorten
v1-0001-Fix-walreceiver-restart-skipping-the-end-of-a-WAL.patch
Description: Binary data
