On Tue, Mar 22, 2022 at 6:14 AM David Christensen
<[email protected]> wrote:
> Updated to include the V3 fixes as well as the unsigned int/enum fix.
Hi David,
I ran this though pg_indent and adjusted some remaining
non-project-style whitespace, and took it for a spin. Very minor
comments:
pg_waldump: error: could not parse valid relation from ""/ (expecting
"tablespace OID/database OID/relation filenode")
-> There was a stray "/" in that message, which I've removed in the attached.
pg_waldump: error: could not parse valid relation from "1664/0/1262"
(expecting "tablespace OID/database OID/relation filenode")
-> Why not? Shared relations like pg_database have invalid database
OID, so I think it should be legal to write --relation=1664/0/1262. I
took out that restriction.
+ if (sscanf(optarg, "%u",
&forknum) != 1 ||
+ forknum >= MAX_FORKNUM)
+ {
+ pg_log_error("could
not parse valid fork number (0..%d) \"%s\"",
+
MAX_FORKNUM - 1, optarg);
+ goto bad_argument;
+ }
I guess you did this because init fork references aren't really
expected in the WAL, but I think it's more consistent to allow up to
MAX_FORKNUM, not least because your documentation mentions 3 as a
valid value. So I adjust this to allow MAX_FORKNUM. Make sense?
Here are some more details I noticed, as a likely future user of this
very handy feature, which I haven't changed, because they seem more
debatable and you might disagree...
1. I think it'd be less surprising if the default value for --fork
wasn't 0... why not show all forks?
2. I think it'd be less surprising if --fork without --relation
either raised an error (like --block without --relation), or were
allowed, with the meaning "show me this fork of all relations".
3. It seems funny to have no short switch for --fork when everything
else has one... what about -F?
From 596181e9dfe2db9d5338b3ac3c899d230fe0fc78 Mon Sep 17 00:00:00 2001
From: David Christensen <[email protected]>
Date: Fri, 25 Feb 2022 12:52:56 -0600
Subject: [PATCH v5] Add additional filtering options to pg_waldump.
This feature allows you to only output records that are touch a specific
RelFileNode and optionally BlockNumber and ForkNum in this relation. We
also add the independent ability to filter Full Page Write records.
Author: David Christensen <[email protected]>
Reviewed-by: Peter Geoghegan <[email protected]>
Reviewed-by: Japin Li <[email protected]>
Reviewed-by: Bharath Rupireddy <[email protected]>
Reviewed-by: Cary Huang <[email protected]>
Reviewed-by: Thomas Munro <[email protected]>
Discussion: https://postgr.es/m/lzzgmgm6e5.fsf%40veeddrois.attlocal.net
---
doc/src/sgml/ref/pg_waldump.sgml | 48 +++++++++++
src/bin/pg_waldump/pg_waldump.c | 132 ++++++++++++++++++++++++++++++-
2 files changed, 179 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/pg_waldump.sgml b/doc/src/sgml/ref/pg_waldump.sgml
index 5735a161ce..f157175764 100644
--- a/doc/src/sgml/ref/pg_waldump.sgml
+++ b/doc/src/sgml/ref/pg_waldump.sgml
@@ -100,6 +100,44 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-k <replaceable>block</replaceable></option></term>
+ <term><option>--block=<replaceable>block</replaceable></option></term>
+ <listitem>
+ <para>
+ Display only records touching the given block. (Requires also
+ providing the relation via <option>--relation</option>.)
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--fork=<replaceable>fork</replaceable></option></term>
+ <listitem>
+ <para>
+ When using the <option>--relation</option> filter, output only records
+ from the given fork. The valid values here are <literal>0</literal>
+ for the main fork, <literal>1</literal> for the Free Space
+ Map, <literal>2</literal> for the Visibility Map,
+ and <literal>3</literal> for the Init fork. If unspecified, defaults
+ to the main fork.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-l <replaceable>tbl</replaceable>/<replaceable>db</replaceable>/<replaceable>rel</replaceable></option></term>
+ <term><option>--relation=<replaceable>tbl</replaceable>/<replaceable>db</replaceable>/<replaceable>rel</replaceable></option></term>
+ <listitem>
+ <para>
+ Display only records touching the given relation. The relation is
+ specified via tablespace OID, database OID, and relfilenode separated
+ by slashes, for instance, <literal>1234/12345/12345</literal>. This
+ is the same format used for relations in the WAL dump output.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-n <replaceable>limit</replaceable></option></term>
<term><option>--limit=<replaceable>limit</replaceable></option></term>
@@ -183,6 +221,16 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-w</option></term>
+ <term><option>--fullpage</option></term>
+ <listitem>
+ <para>
+ Filter records to only those which have full page writes.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-x <replaceable>xid</replaceable></option></term>
<term><option>--xid=<replaceable>xid</replaceable></option></term>
diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c
index fc081adfb8..eb0c9b2dcb 100644
--- a/src/bin/pg_waldump/pg_waldump.c
+++ b/src/bin/pg_waldump/pg_waldump.c
@@ -55,6 +55,12 @@ typedef struct XLogDumpConfig
bool filter_by_rmgr_enabled;
TransactionId filter_by_xid;
bool filter_by_xid_enabled;
+ RelFileNode filter_by_relation;
+ bool filter_by_relation_enabled;
+ BlockNumber filter_by_relation_block;
+ bool filter_by_relation_block_enabled;
+ ForkNumber filter_by_relation_forknum;
+ bool filter_by_fpw;
} XLogDumpConfig;
typedef struct Stats
@@ -391,6 +397,56 @@ WALDumpReadPage(XLogReaderState *state, XLogRecPtr targetPagePtr, int reqLen,
return count;
}
+/*
+ * Boolean to return whether the given WAL record matches a specific relation and optional block
+ */
+static bool
+XLogRecordMatchesRelationBlock(XLogReaderState *record, RelFileNode matchRnode, BlockNumber matchBlock, ForkNumber matchFork)
+{
+ int block_id;
+
+ for (block_id = 0; block_id <= XLogRecMaxBlockId(record); block_id++)
+ {
+ RelFileNode rnode;
+ ForkNumber forknum;
+ BlockNumber blk;
+
+ if (!XLogRecHasBlockRef(record, block_id))
+ continue;
+
+ XLogRecGetBlockTag(record, block_id, &rnode, &forknum, &blk);
+
+ if (forknum == matchFork &&
+ matchRnode.spcNode == rnode.spcNode &&
+ matchRnode.dbNode == rnode.dbNode &&
+ matchRnode.relNode == rnode.relNode &&
+ (matchBlock == InvalidBlockNumber || matchBlock == blk))
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Boolean to return whether the given WAL record contains a full page write
+ */
+static bool
+XLogRecordHasFPW(XLogReaderState *record)
+{
+ int block_id;
+
+ for (block_id = 0; block_id <= XLogRecMaxBlockId(record); block_id++)
+ {
+ if (!XLogRecHasBlockRef(record, block_id))
+ continue;
+
+ if (XLogRecHasBlockImage(record, block_id))
+ return true;
+ }
+
+ return false;
+}
+
/*
* Calculate the size of a record, split into !FPI and FPI parts.
*/
@@ -765,6 +821,10 @@ usage(void)
printf(_(" -b, --bkp-details output detailed information about backup blocks\n"));
printf(_(" -e, --end=RECPTR stop reading at WAL location RECPTR\n"));
printf(_(" -f, --follow keep retrying after reaching end of WAL\n"));
+ printf(_(" -k, --block=N with --relation, only show records matching this block\n"));
+ printf(_(" --fork=N with --relation, only show records matching this fork\n"
+ " (defaults to 0, the main fork)\n"));
+ printf(_(" -l, --relation=N/N/N only show records that touch a specific relation\n"));
printf(_(" -n, --limit=N number of records to display\n"));
printf(_(" -p, --path=PATH directory in which to find log segment files or a\n"
" directory with a ./pg_wal that contains such files\n"
@@ -777,6 +837,7 @@ usage(void)
" (default: 1 or the value used in STARTSEG)\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" -x, --xid=XID only show records with transaction ID XID\n"));
+ printf(_(" -w, --fullpage only show records with a full page write\n"));
printf(_(" -z, --stats[=record] show statistics instead of records\n"
" (optionally, show per-record statistics)\n"));
printf(_(" -?, --help show this help, then exit\n"));
@@ -800,12 +861,16 @@ main(int argc, char **argv)
static struct option long_options[] = {
{"bkp-details", no_argument, NULL, 'b'},
+ {"block", required_argument, NULL, 'k'},
{"end", required_argument, NULL, 'e'},
{"follow", no_argument, NULL, 'f'},
+ {"fork", required_argument, NULL, 1},
+ {"fullpage", no_argument, NULL, 'w'},
{"help", no_argument, NULL, '?'},
{"limit", required_argument, NULL, 'n'},
{"path", required_argument, NULL, 'p'},
{"quiet", no_argument, NULL, 'q'},
+ {"relation", required_argument, NULL, 'l'},
{"rmgr", required_argument, NULL, 'r'},
{"start", required_argument, NULL, 's'},
{"timeline", required_argument, NULL, 't'},
@@ -858,6 +923,10 @@ main(int argc, char **argv)
config.filter_by_rmgr_enabled = false;
config.filter_by_xid = InvalidTransactionId;
config.filter_by_xid_enabled = false;
+ config.filter_by_relation_enabled = false;
+ config.filter_by_relation_block_enabled = false;
+ config.filter_by_relation_forknum = MAIN_FORKNUM;
+ config.filter_by_fpw = false;
config.stats = false;
config.stats_per_record = false;
@@ -870,7 +939,7 @@ main(int argc, char **argv)
goto bad_argument;
}
- while ((option = getopt_long(argc, argv, "be:fn:p:qr:s:t:x:z",
+ while ((option = getopt_long(argc, argv, "be:fk:l:n:p:qr:s:t:wx:z",
long_options, &optindex)) != -1)
{
switch (option)
@@ -890,6 +959,44 @@ main(int argc, char **argv)
case 'f':
config.follow = true;
break;
+ case 1: /* fork number */
+ {
+ unsigned int forknum;
+
+ if (sscanf(optarg, "%u", &forknum) != 1 ||
+ forknum > MAX_FORKNUM)
+ {
+ pg_log_error("could not parse valid fork number (0..%d) \"%s\"",
+ MAX_FORKNUM, optarg);
+ goto bad_argument;
+ }
+ config.filter_by_relation_forknum = (ForkNumber) forknum;
+ }
+ break;
+ case 'k':
+ if (sscanf(optarg, "%u", &config.filter_by_relation_block) != 1 ||
+ !BlockNumberIsValid(config.filter_by_relation_block))
+ {
+ pg_log_error("could not parse valid block number \"%s\"", optarg);
+ goto bad_argument;
+ }
+ config.filter_by_relation_block_enabled = true;
+ break;
+ case 'l':
+ if (sscanf(optarg, "%u/%u/%u",
+ &config.filter_by_relation.spcNode,
+ &config.filter_by_relation.dbNode,
+ &config.filter_by_relation.relNode) != 3 ||
+ !OidIsValid(config.filter_by_relation.spcNode) ||
+ !OidIsValid(config.filter_by_relation.relNode))
+ {
+ pg_log_error("could not parse valid relation from \"%s\""
+ " (expecting \"tablespace OID/database OID/"
+ "relation filenode\")", optarg);
+ goto bad_argument;
+ }
+ config.filter_by_relation_enabled = true;
+ break;
case 'n':
if (sscanf(optarg, "%d", &config.stop_after_records) != 1)
{
@@ -947,6 +1054,9 @@ main(int argc, char **argv)
goto bad_argument;
}
break;
+ case 'w':
+ config.filter_by_fpw = true;
+ break;
case 'x':
if (sscanf(optarg, "%u", &config.filter_by_xid) != 1)
{
@@ -976,6 +1086,13 @@ main(int argc, char **argv)
}
}
+ if (config.filter_by_relation_block_enabled &&
+ !config.filter_by_relation_enabled)
+ {
+ pg_log_error("--block option requires --relation option to be specified");
+ goto bad_argument;
+ }
+
if ((optind + 2) < argc)
{
pg_log_error("too many command-line arguments (first is \"%s\")",
@@ -1148,6 +1265,19 @@ main(int argc, char **argv)
config.filter_by_xid != record->xl_xid)
continue;
+ /* check for extended filtering */
+ if (config.filter_by_relation_enabled &&
+ !XLogRecordMatchesRelationBlock(xlogreader_state,
+ config.filter_by_relation,
+ config.filter_by_relation_block_enabled ?
+ config.filter_by_relation_block :
+ InvalidBlockNumber,
+ config.filter_by_relation_forknum))
+ continue;
+
+ if (config.filter_by_fpw && !XLogRecordHasFPW(xlogreader_state))
+ continue;
+
/* perform any per-record work */
if (!config.quiet)
{
--
2.30.2
