Hi, On Thu, Oct 15, 2026 at 4:10 PM Michael Paquier <[email protected]> wrote: > > Thanks. Applied v10-0001 after tweaking a bit the docs, to close the > monitoring gap with the tracking of WAL buffer bits.
Thanks. Please find attached the remaining patches, renumbered to keep CI happy. No code changes, changed only the patch numbers, so kept them as v10. -- Bharath Rupireddy Amazon Web Services: https://aws.amazon.com
From d22b4cbe80f510d6ad8f3dd95666be5eef39f294 Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Sat, 10 Oct 2026 18:50:00 +0000 Subject: [PATCH v10 1/3] Use WALReadFromBuffers() for logical walsenders. Commit 91f2cae7a4 introduced WALReadFromBuffers() but used it only for physical walsenders. This commit lets logical walsenders also read WAL from the WAL buffers instead of always going to a file. This reduces read IO and syscalls, especially when the logical replication consumer keeps up with WAL generation. It helps even more with WAL direct IO enabled, where a file read is always a physical disk read without the OS page cache. Author: Bharath Rupireddy <[email protected]> Reviewed-by: Michael Paquier <[email protected]> Reviewed-by: Kirill Reshke <[email protected]> Discussion: https://postgr.es/m/CALj2ACVfF2Uj9NoFy-5m98HNtjHpuD17EDE9twVeJng-jTAe7A%40mail.gmail.com --- src/backend/replication/walsender.c | 34 +++++++++++++++++----- src/test/subscription/t/001_rep_changes.pl | 9 +++--- 2 files changed, 31 insertions(+), 12 deletions(-) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 10bc3e0cae4..2991dd6eb6f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1098,6 +1098,7 @@ logical_read_xlog_page(XLogReaderState *state, XLogRecPtr targetPagePtr, int req WALReadError errinfo; XLogSegNo segno; TimeLineID currTLI; + Size rbytes; /* * Make sure we have enough WAL available before retrieving the current @@ -1157,16 +1158,33 @@ logical_read_xlog_page(XLogReaderState *state, XLogRecPtr targetPagePtr, int req else count = flushptr - targetPagePtr; /* part of the page available */ - /* now actually read the data, we know it's there */ - if (!WALRead(state, - cur_page, - targetPagePtr, - count, - currTLI, /* Pass the current TLI because only + /* attempt to read WAL from WAL buffers first */ + rbytes = WALReadFromBuffers(cur_page, targetPagePtr, count, currTLI); + + /* now read the remaining WAL from WAL file */ + if (rbytes < count) + { + if (!WALRead(state, + cur_page + rbytes, + targetPagePtr + rbytes, + count - rbytes, + currTLI, /* Pass the current TLI because only * WalSndSegmentOpen controls whether new TLI * is needed. */ - &errinfo)) - WALReadRaiseError(&errinfo); + &errinfo)) + WALReadRaiseError(&errinfo); + } + else if (state->seg.ws_file >= 0 && + !XLByteInSeg(targetPagePtr, state->seg.ws_segno, + state->segcxt.ws_segsize)) + { + /* + * Close the segment when a read that comes entirely from the WAL + * buffers is not in the open segment, so that the next file read + * reopens the correct one. + */ + state->routine.segment_close(state); + } /* * After reading into the buffer, check that what we read was valid. We do diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 7d41715ed81..5eb74615bca 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -187,12 +187,13 @@ $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM tab_no_col"); is($result, qq(2), 'check replicated changes for table having no columns'); -# Wait for the logical WAL sender to update its IO statistics. This is -# done before the next restart, which would force a flush of its stats, and -# far enough from the reset done above to not impact the run time. +# Wait for the logical WAL sender to update its IO statistics (WAL reads from +# buffers and file). This is done before the next restart, which would force a +# flush of its stats, and far enough from the reset done above to not impact +# the run time. $node_publisher->poll_query_until( 'postgres', - qq[SELECT sum(reads) > 0 + qq[SELECT sum(reads) + sum(hits) > 0 FROM pg_catalog.pg_stat_io WHERE backend_type = 'walsender' AND object = 'wal'] -- 2.47.3
From 17f34ed4f364857efcb5b8506fc5291ba1fad387 Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Sat, 10 Oct 2026 18:53:49 +0000 Subject: [PATCH v10 2/3] Use WALReadFromBuffers() for local WAL reads. Commit 91f2cae7a4 introduced WALReadFromBuffers() and the previous commits used it for physical and logical walsenders. This commit lets the callers of read_local_xlog_page(), such as logical decoding SQL functions, 2PC, concurrent repack workers and pg_walinspect, also read WAL from the WAL buffers instead of always going to a file. This reduces read IO and syscalls, especially when these callers keep up with WAL generation. It helps even more with WAL direct IO enabled, where a file read is always a physical disk read without the OS page cache. The 2PC WAL read happens in the checkpointer, so pg_stat_io now reports hits on object wal for it as well. Author: Bharath Rupireddy <[email protected]> Reviewed-by: Michael Paquier <[email protected]> Reviewed-by: Kirill Reshke <[email protected]> Discussion: https://postgr.es/m/CALj2ACVfF2Uj9NoFy-5m98HNtjHpuD17EDE9twVeJng-jTAe7A%40mail.gmail.com --- src/backend/access/transam/xlogutils.c | 29 +++++++++++++++++++++++--- src/backend/utils/activity/pgstat_io.c | 5 +++-- 2 files changed, 29 insertions(+), 5 deletions(-) diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c index ff71e14015d..e59d59c03fb 100644 --- a/src/backend/access/transam/xlogutils.c +++ b/src/backend/access/transam/xlogutils.c @@ -903,6 +903,7 @@ read_local_xlog_page_guts(XLogReaderState *state, XLogRecPtr targetPagePtr, int count; WALReadError errinfo; TimeLineID currTLI; + Size rbytes; loc = targetPagePtr + reqLen; @@ -1034,9 +1035,31 @@ read_local_xlog_page_guts(XLogReaderState *state, XLogRecPtr targetPagePtr, count = read_upto - targetPagePtr; } - if (!WALRead(state, cur_page, targetPagePtr, count, tli, - &errinfo)) - WALReadRaiseError(&errinfo); + /* attempt to read WAL from WAL buffers first */ + rbytes = WALReadFromBuffers(cur_page, targetPagePtr, count, tli); + + /* now read the remaining WAL from WAL file */ + if (rbytes < count) + { + if (!WALRead(state, + cur_page + rbytes, + targetPagePtr + rbytes, + count - rbytes, + tli, + &errinfo)) + WALReadRaiseError(&errinfo); + } + else if (state->seg.ws_file >= 0 && + !XLByteInSeg(targetPagePtr, state->seg.ws_segno, + state->segcxt.ws_segsize)) + { + /* + * Close the segment when a read that comes entirely from the WAL + * buffers is not in the open segment, so that the next file read + * reopens the correct one. + */ + state->routine.segment_close(state); + } /* number of valid bytes in the buffer */ return count; diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 3c8fcb20aba..7b63d805999 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -516,8 +516,9 @@ pgstat_tracks_io_op(BackendType bktype, IOObject io_object, return false; if (bktype == B_CHECKPOINTER && - ((io_object != IOOBJECT_WAL && io_op == IOOP_READ) || - (io_op == IOOP_EVICT || io_op == IOOP_HIT))) + ((io_object != IOOBJECT_WAL && + (io_op == IOOP_READ || io_op == IOOP_HIT)) || + io_op == IOOP_EVICT)) return false; if ((bktype == B_BG_WRITER || bktype == B_CHECKPOINTER) && -- 2.47.3
From 2dbc19b2e4871e636106be30a02e1753d22a0c2a Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Sat, 10 Oct 2026 19:10:50 +0000 Subject: [PATCH v10 3/3] Use WALReadFromBuffers() for WAL summarizer. Commit 91f2cae7a4 introduced WALReadFromBuffers() and the previous commits used it for physical and logical walsenders and for the local WAL readers. This commit lets the WAL summarizer also read WAL from the WAL buffers instead of always going to a file. This reduces read IO and syscalls, especially when the summarizer keeps up with WAL generation. It helps even more with WAL direct IO enabled, where a file read is always a physical disk read without the OS page cache. Author: Bharath Rupireddy <[email protected]> Reviewed-by: Michael Paquier <[email protected]> Reviewed-by: Kirill Reshke <[email protected]> Discussion: https://postgr.es/m/CALj2ACVfF2Uj9NoFy-5m98HNtjHpuD17EDE9twVeJng-jTAe7A%40mail.gmail.com --- src/backend/postmaster/walsummarizer.c | 30 +++++++++++++++++++++++--- src/bin/pg_walsummary/t/002_blocks.pl | 5 +++-- 2 files changed, 30 insertions(+), 5 deletions(-) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 8333a4eaaa3..00a48f4c952 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -1659,6 +1659,7 @@ summarizer_read_local_xlog_page(XLogReaderState *state, int count; WALReadError errinfo; SummarizerReadLocalXLogPrivate *private_data; + Size rbytes; ProcessWalSummarizerInterrupts(); @@ -1750,9 +1751,32 @@ summarizer_read_local_xlog_page(XLogReaderState *state, } } - if (!WALRead(state, cur_page, targetPagePtr, count, - private_data->tli, &errinfo)) - WALReadRaiseError(&errinfo); + /* attempt to read WAL from WAL buffers first */ + rbytes = WALReadFromBuffers(cur_page, targetPagePtr, count, + private_data->tli); + + /* now read the remaining WAL from WAL file */ + if (rbytes < count) + { + if (!WALRead(state, + cur_page + rbytes, + targetPagePtr + rbytes, + count - rbytes, + private_data->tli, + &errinfo)) + WALReadRaiseError(&errinfo); + } + else if (state->seg.ws_file >= 0 && + !XLByteInSeg(targetPagePtr, state->seg.ws_segno, + state->segcxt.ws_segsize)) + { + /* + * Close the segment when a read that comes entirely from the WAL + * buffers is not in the open segment, so that the next file read + * reopens the correct one. + */ + state->routine.segment_close(state); + } /* Track that we read a page, for sleep time calculation. */ ++pages_read_since_last_sleep; diff --git a/src/bin/pg_walsummary/t/002_blocks.pl b/src/bin/pg_walsummary/t/002_blocks.pl index c5a9649d6ad..ca0bf0002fc 100644 --- a/src/bin/pg_walsummary/t/002_blocks.pl +++ b/src/bin/pg_walsummary/t/002_blocks.pl @@ -58,10 +58,11 @@ command_like( command_like([ 'pg_walsummary', '-q', @summaries ], qr/\A\z/, "-q suppresses all output"); -# The WAL summarizer should have generated some IO statistics. +# The WAL summarizer should have generated some IO statistics (WAL reads from +# buffers and file). $node1->poll_query_until( 'postgres', - q{SELECT sum(reads) > 0 FROM pg_stat_io + q{SELECT sum(hits) + sum(reads) > 0 FROM pg_stat_io WHERE backend_type = 'walsummarizer' AND object = 'wal'}) or die "Timed out while waiting for WAL summarizer to generate statistics for WAL reads"; -- 2.47.3
