Hi, On Thu, 3 Sept 2026 at 00:27, Ayush Tiwari <[email protected]> wrote: > > I had another pass at this and have attached v3, now split into two patches. > The removal and recycling cases have one important difference: only recycling > exposes a new WAL filename to the write path. > > 0001 is the straightforward bit. For segments that a checkpoint removes, it > does the unlinks first, fsyncs pg_wal once, and then cleans up the archive > status files. I left the timeline-switch path alone. This is not only for > wal_recycle=off; with recycling enabled, surplus segments still take the > removal path. > > I reran the removal benchmark with the default 16 MiB WAL segment size. The > numbers below are for 0001 alone; 0002 was not applied. This was on ext4 with > fsync enabled and wal_recycle=off, so every eligible segment went through the > removal path. Each number is the median of five alternating runs after three > warmups: > > files removed master 0001 > 1 22 ms 23 ms > 8 53 ms 32 ms > 32 135 ms 64 ms > 64 201 ms 97 ms > 128 394 ms 154 ms > > 0002 is the awkward bit, and I have left it marked WIP. Going back over v2, > I found a hole in the durable-segment high-water mark. A foreground process > can find the names in the pending recycle batch already occupied, install its > own segment at a higher number, and move the frontier past lower renames that > are not durable yet. > > The current patch plugs that hole with an active-batch marker. While a batch > is in flight, XLogFileInit() takes the file-and-directory fsync fallback even > if the frontier says the segment is covered. The new TAP test forces exactly > this interleaving; without the marker check, it times out waiting for the > fallback. > > That handles the case I found, but I am still on the fence about 0002. It > feels like a fair amount of plumbing between checkpoint cleanup and the WAL > write path just to save these fsyncs. Is there a cleaner invariant that I am > missing, or a less invasive way to make the batched renames safe?
Attached v4, rebased on master. I noticed I'd made directory-sync failures PANIC by default in v3. 0001 now keeps the old LOG-level behavior and leaves archive status alone if the batch sync fails. 0002 is just rebased, with its TAP test renumbered. I've kept it WIP and added a nocfbot prefix pending discussion. Regards, Ayush
nocfbot-v4-0002-WIP-Batch-fsyncs-when-recycling-WAL-segments.patch
Description: Binary data
v4-0001-Batch-fsyncs-when-removing-old-WAL-segments.patch
Description: Binary data
