Hi hackers,
Wait events are useful to know what backends are waiting for when there is/was
a performance issue: for this we can sample pg_stat_activity at regular
intervals
and record historical data. That’s how it is commonly used.
It could also be useful to observe the engine/backends behavior over time and
help answer questions like:
* Is the engine’s wait pattern the same over time?
* Is application’s "A" wait pattern the same over time?
* I observe a peak in wait event "W": is it because "W" is now waiting longer or
is it because it is hit more frequently?
* I observe a peak in some of them (say for example MultiXact%), is it due to a
workload change?
For the above use cases, we need a way to track the wait events that occur
between
samples: please find attached a proof of concept patch series doing so.
The patch series is made of:
0001 - It generates the WaitClassTable[], a lookup table that will be used by
the wait events statistics.
The array is indexed by classId (derived from the PG_WAIT_* constants), handles
gaps in the class ID numbering and provides metadata for wait events.
This new array is generated in generate-wait_event_types.pl, so that:
* it now needs lwlocklist.h and wait_classes.h as input parameters
* WAIT_EVENT_CLASS_MASK and WAIT_EVENT_ID_MASK have been moved away from
wait_event.c
In passing it adds several new macros that will be used by 0002.
0002 - It adds wait events statistics
It adds a new stat kind PGSTAT_KIND_WAIT_EVENT for the wait event statistics.
This new statistic kind is a fixed one because we know the maximum number of
wait
events. Indeed:
* it does not take into account custom wait events as extensions have all they
need
at their disposal to create custom stats on their own wait events should they
want to (limited by WAIT_EVENT_CUSTOM_HASH_MAX_SIZE though).
* it does not take into account LWLock > LWTRANCHE_FIRST_USER_DEFINED for the
same
reasons as above. That said, there is no maximum limitation in
LWLockNewTrancheId().
* we don't want to allocate memory in some places where the counters are
incremented (see 4feba03d8b9). We could still use the same implementation as
for
backend statistics (i.e, make use of flush_static_cb) if we really want/need to
switch to variable stats.
For the moment only the counters are added (an array of currently 285 counters),
we’ll study/discuss about adding the timings once the counters are fully done.
I think we’d have more discussion/debate around the timings (should we add them
by default, add a new GUC, enable them at compilation time?…), that’s why only
the counters are in this patch series.
I think it makes sense as the counters have merit on their own. We currently
have
273 wait events but the array is 285 long: the reason is that some wait events
classes have "holes".
A few questions:
* Do we need to serialize the stats based on their names (as for
PGSTAT_KIND_REPLSLOT)? This question is the same as "is the ordering preserved
if file stats format is not changed": I think the answer is yes (see
f98dbdeb51d)
, which means there is no need for to_serialized_name/from_serialized_name.
* What if a new wait event is added? We'd need to change the stats file format,
unless: the wait event stats kind becomes a variable one or we change a bit the
way fixed stats are written/read to/from the stat file (we could add a new
field
in the PgStat_KindInfo for example).
Note: for some backends the wait events stats are not flushed (walwriter for
example), so we need to find additional places to flush the wait events stats.
0003 - It adds the pg_stat_wait_event view
It also adds documentation and regression tests.
0004 - switching PGSTAT_KIND_WAIT_EVENT to variable sized
It might be better for PGSTAT_KIND_WAIT_EVENT to be a variable sized stats kind.
That way:
* It would be easier to add custom wait events if we want to
* It would be possible to add new wait events without having to change the stats
file format
So adding 0004 to see what it would look like to have a variable sized stats
kind
instead and decide how we want to move forward.
It uses the uint32 wait_event_info as the hash key while the hash key is defined
as uint64: that should not be an issue but this patch does explicit casting
though.
That said it might be better to use all the 64 bits (means not have the half
full
of zeroes) for the hash key (better hashing distribution?): we could imagine
using something like:
((uint64) wait_event_info) | (((uint64) wait_event_info) << 32)
for the hash key.
If we decide to go that way (means with variable sized kind) then a new patch
series will be provided and will not implement the fixed one but will start
directly with the variable one.
The more I think about it, the more I think we should go for the variable sized
proposal: that's more flexible.
Remarks:
* If we want to add some "ereport” in waitEventIncrementCounter() then we’d need
to take care of the race condition in ConditionVariableTimedSleep() that that
would
produce, see [1].
* Once we agree on fixed vs variable sized stats kind, I'll start measuring if
there is any performance regression and check if there is a need for
optimization
(partition the counters array?…).
* The pgstat_report_wait_end() is "inlined", but with the additional code added
here, compilers may ignore the inline keyword. Need to check if that's the case
and the impact (see above).
* After it goes in: add timings, add into per-backend stats too
[1]:
https://www.postgresql.org/message-id/aBhuTqNhMN3prcqe%40ip-10-97-1-34.eu-west-3.compute.internal
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
>From f9604a6a27d259552da92f973ca3a330beb3d6cd Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot...@gmail.com>
Date: Tue, 22 Apr 2025 07:39:16 +0000
Subject: [PATCH v1 1/4] Generate the WaitClassTable[]
The WaitClassTable[] array that will be used as a lookup table by the wait events
statistics (implemented in a following commit).
The array is indexed by classId (derived from the PG_WAIT_* constants), handles
gaps in the class ID numbering and provides metadata for wait events.
A new struct (WaitClassTableEntry) is used to store the wait events metadata.
This new array is generated in generate-wait_event_types.pl, so that:
- it now needs lwlocklist.h and wait_classes.h as input parameters
- WAIT_EVENT_CLASS_MASK and WAIT_EVENT_ID_MASK have been moved away from
wait_event.c
In passing it adds severals new macros that will be used by a following commit
implementing the wait events statistics.
This commit does not generate any new files but adds some code in
wait_event_types.h and pgstat_wait_event.c.
---
src/backend/Makefile | 2 +-
src/backend/utils/activity/Makefile | 4 +-
.../activity/generate-wait_event_types.pl | 244 +++++++++++++++++-
src/backend/utils/activity/wait_event.c | 3 -
src/include/utils/meson.build | 4 +-
5 files changed, 248 insertions(+), 9 deletions(-)
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 7344c8c7f5c..685d7a0a77e 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -114,7 +114,7 @@ parser/gram.h: parser/gram.y
storage/lmgr/lwlocknames.h: storage/lmgr/generate-lwlocknames.pl ../include/storage/lwlocklist.h utils/activity/wait_event_names.txt
$(MAKE) -C storage/lmgr lwlocknames.h
-utils/activity/wait_event_types.h: utils/activity/generate-wait_event_types.pl utils/activity/wait_event_names.txt
+utils/activity/wait_event_types.h: utils/activity/generate-wait_event_types.pl utils/activity/wait_event_names.txt ../include/storage/lwlocklist.h ../include/utils/wait_classes.h
$(MAKE) -C utils/activity wait_event_types.h pgstat_wait_event.c wait_event_funcs_data.c
# run this unconditionally to avoid needing to know its dependencies here:
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index 9c2443e1ecd..f9849bebc98 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -45,8 +45,8 @@ wait_event.o: pgstat_wait_event.c
pgstat_wait_event.c: wait_event_types.h
touch $@
-wait_event_types.h: $(top_srcdir)/src/backend/utils/activity/wait_event_names.txt generate-wait_event_types.pl
- $(PERL) $(srcdir)/generate-wait_event_types.pl --code $<
+wait_event_types.h: $(top_srcdir)/src/backend/utils/activity/wait_event_names.txt $(top_srcdir)/src/include/storage/lwlocklist.h $(top_srcdir)/src/include/utils/wait_classes.h generate-wait_event_types.pl
+ $(PERL) $(srcdir)/generate-wait_event_types.pl --code $(wordlist 1,3,$^)
clean:
rm -f wait_event_types.h pgstat_wait_event.c wait_event_funcs_data.c
diff --git a/src/backend/utils/activity/generate-wait_event_types.pl b/src/backend/utils/activity/generate-wait_event_types.pl
index 424ad9f115d..c18693aa68b 100644
--- a/src/backend/utils/activity/generate-wait_event_types.pl
+++ b/src/backend/utils/activity/generate-wait_event_types.pl
@@ -21,6 +21,12 @@ use Getopt::Long;
my $output_path = '.';
my $gen_docs = 0;
my $gen_code = 0;
+my $nb_waitclass_table_entries = 0;
+my $nb_wait_events_with_null = 0;
+my $nb_wait_events_per_class = 0;
+my %waitclass_values;
+my $wait_event_class_mask = 0xFF000000;
+my $wait_event_id_mask = 0x0000FFFF;
my $continue = "\n";
my %hashwe;
@@ -38,11 +44,50 @@ die "Not possible to specify --docs and --code simultaneously"
open my $wait_event_names, '<', $ARGV[0] or die;
+# When generating code, we need lwlocklist.h as the second argument
+my $lwlocklist_file = $ARGV[1] if $gen_code;
+
+# When generating code, we need wait_classes.h as the third argument
+my $wait_classes_file = $ARGV[2] if $gen_code;
+
my @abi_compatibility_lines;
my @lines;
my $abi_compatibility = 0;
my $section_name;
+# Function to parse wait_classes.h and extract wait class definitions
+sub parse_wait_classes_header
+{
+
+ open my $wait_classes_header, '<', $wait_classes_file
+ or die "Could not open $wait_classes_file: $!";
+
+ while (<$wait_classes_header>)
+ {
+ chomp;
+ if (/^\s*#define\s+(PG_WAIT_\w+)\s+(0x[0-9A-Fa-f]+)U?\s*$/)
+ {
+ my ($macro_name, $value) = ($1, $2);
+
+ $waitclass_values{$macro_name} = $value;
+ }
+ }
+
+ close $wait_classes_header;
+}
+
+# Function to get the macro from the wait class name
+sub waitclass_to_macro
+{
+
+ my $waitclass = shift;
+ my $last = $waitclass;
+ $last =~ s/^WaitEvent//;
+ my $lastuc = uc $last;
+
+ return "PG_WAIT_" . $lastuc;
+}
+
# Remove comments and empty lines and add waitclassname based on the section
while (<$wait_event_names>)
{
@@ -84,8 +129,39 @@ while (<$wait_event_names>)
# Sort the lines based on the second column.
# uc() is being used to force the comparison to be case-insensitive.
-my @lines_sorted =
- sort { uc((split(/\t/, $a))[1]) cmp uc((split(/\t/, $b))[1]) } @lines;
+
+my @lines_sorted;
+if ($gen_code)
+{
+ my @lwlock_lines;
+
+ # Separate LWLock lines from others
+ foreach my $line (@lines)
+ {
+ if ($line =~ /^WaitEventLWLock\t/)
+ {
+ push(@lwlock_lines, $line);
+ }
+ else
+ {
+ push(@lines_sorted, $line);
+ }
+ }
+
+ # Sort only non-LWLock lines
+ @lines_sorted =
+ sort { uc((split(/\t/, $a))[1]) cmp uc((split(/\t/, $b))[1]) }
+ @lines_sorted;
+
+ # Add LWLock lines back in their original order
+ push(@lines_sorted, @lwlock_lines);
+}
+else
+{
+ # For docs, use original alphabetical sorting for all
+ @lines_sorted =
+ sort { uc((split(/\t/, $a))[1]) cmp uc((split(/\t/, $b))[1]) } @lines;
+}
# If we are generating code, concat @lines_sorted and then
# @abi_compatibility_lines.
@@ -168,6 +244,10 @@ if ($gen_code)
printf $h $header_comment, 'wait_event_types.h';
printf $h "#ifndef WAIT_EVENT_TYPES_H\n";
printf $h "#define WAIT_EVENT_TYPES_H\n\n";
+ printf $h "#define WAIT_EVENT_CLASS_MASK 0x%08X\n",
+ $wait_event_class_mask;
+ printf $h "#define WAIT_EVENT_ID_MASK 0x%08X\n\n",
+ $wait_event_id_mask;
printf $h "#include \"utils/wait_classes.h\"\n\n";
printf $c $header_comment, 'pgstat_wait_event.c';
@@ -269,6 +349,166 @@ if ($gen_code)
}
}
+ printf $h "
+
+/* To represent wait_event_info as integers */
+typedef struct DecodedWaitInfo
+{
+ int classId;
+ int eventId;
+} DecodedWaitInfo;
+
+/* To extract classId and eventId as integers from wait_event_info */
+#define WAIT_EVENT_INFO_DECODE(d, i) \\
+ d.classId = ((i) & WAIT_EVENT_CLASS_MASK) / (WAIT_EVENT_CLASS_MASK & (-WAIT_EVENT_CLASS_MASK)), \\
+ d.eventId = (i) & WAIT_EVENT_ID_MASK
+
+/* To map wait event classes into the WaitClassTable */
+typedef struct
+{
+ const int classId;
+ const int numberOfEvents;
+ const int offSet;
+ const char *className;
+ const char *const *eventNames;
+} WaitClassTableEntry;
+
+extern WaitClassTableEntry WaitClassTable[];\n\n";
+
+ printf $c "
+/*
+ * Lookup table that is used by the wait events statistics.
+ * Indexed by classId (derived from the PG_WAIT_* constants), handle gaps
+ * in the class ID numbering and provide metadata for wait events.
+ */
+WaitClassTableEntry WaitClassTable[] = {\n";
+
+ parse_wait_classes_header();
+ my $next_index = 0;
+ my $class_divisor = $wait_event_class_mask & (-$wait_event_class_mask);
+
+ foreach my $waitclass (
+ sort {
+ my $macro_a = waitclass_to_macro($a);
+ my $macro_b = waitclass_to_macro($b);
+ hex($waitclass_values{$macro_a}) <=>
+ hex($waitclass_values{$macro_b})
+ } keys %hashwe)
+ {
+ my $event_names_array;
+ my $array_size;
+ my $last = $waitclass;
+ $last =~ s/^WaitEvent//;
+
+ $nb_waitclass_table_entries++;
+
+ # The LWLocks need to be handled differently than the other classes when
+ # building the WaitClassTable. We need to take care of the prefedined
+ # LWLocks as well as the additional ones.
+ if ($waitclass eq 'WaitEventLWLock')
+ {
+ # Parse lwlocklist.h to get LWLock definitions
+ open my $lwlocklist, '<', $lwlocklist_file
+ or die "Could not open $lwlocklist_file: $!";
+
+ my %predefined_lwlock_indices;
+ my $max_lwlock_index = -1;
+
+ while (<$lwlocklist>)
+ {
+ if (/^PG_LWLOCK\((\d+),\s+(\w+)\)$/)
+ {
+ my ($lockidx, $lockname) = ($1, $2);
+ $predefined_lwlock_indices{$lockname} = $lockidx;
+ $max_lwlock_index = $lockidx
+ if $lockidx > $max_lwlock_index;
+ }
+ }
+
+ close $lwlocklist;
+
+ # Iterates through wait_event_names.txt order
+ my @event_names_sparse;
+ my $next_additional_index = $max_lwlock_index + 1;
+
+ foreach my $wev (@{ $hashwe{$waitclass} })
+ {
+ my $lockname = $wev->[1];
+
+ if (exists $predefined_lwlock_indices{$lockname})
+ {
+ # This is a predefined one, place it at its specific index
+ my $index = $predefined_lwlock_indices{$lockname};
+ $event_names_sparse[$index] = "\"$lockname\"";
+ }
+ else
+ {
+ # This is an additional one, append it after predefined ones
+ $event_names_sparse[$next_additional_index] =
+ "\"$lockname\"";
+ $next_additional_index++;
+ }
+ }
+
+ # Fill gaps with NULL for missing predefined locks
+ for my $i (0 .. $max_lwlock_index)
+ {
+ $event_names_sparse[$i] = "NULL"
+ unless defined $event_names_sparse[$i];
+ }
+
+ # Build the array literal
+ $event_names_array = "(const char *const []){"
+ . join(", ", @event_names_sparse) . "}";
+ $array_size = scalar(@event_names_sparse);
+ }
+ else
+ {
+ # Construct a simple string array literal for this class
+ $event_names_array = "(const char *const []){";
+
+ # For each wait event in this class, add its name to the array
+ foreach my $wev (@{ $hashwe{$waitclass} })
+ {
+ $event_names_array .= "\"$wev->[1]\", ";
+ }
+
+ $event_names_array .= "}";
+ $array_size = scalar(@{ $hashwe{$waitclass} });
+ }
+
+ my $lastuc = uc $last;
+ my $pg_wait_class = "PG_WAIT_" . $lastuc;
+
+ my $index = hex($waitclass_values{$pg_wait_class}) / $class_divisor;
+
+ # Fill any holes with {0, 0, 0, NULL, NULL}
+ while ($next_index < $index)
+ {
+ printf $c "{0, 0, 0, NULL, NULL},\n";
+ $next_index++;
+ $nb_waitclass_table_entries++;
+ }
+
+ my $offset = $nb_wait_events_with_null;
+ $nb_wait_events_with_null += $array_size;
+
+ # Generate the entry
+ printf $c "{$pg_wait_class, $array_size, $offset, \"%s\", %s},\n",
+ $last, $event_names_array;
+
+ $next_index = $index + 1;
+ }
+
+ printf $c "};\n\n";
+
+ printf $h "#define NB_WAITCLASSTABLE_SIZE $nb_wait_events_with_null\n";
+ printf $h
+ "#define NB_WAITCLASSTABLE_ENTRIES $nb_waitclass_table_entries\n\n";
+ printf $h
+ "StaticAssertDecl(NB_WAITCLASSTABLE_SIZE > 0, \"Wait class table must have entries\");\n";
+ printf $h
+ "StaticAssertDecl(NB_WAITCLASSTABLE_ENTRIES > 0, \"Must have at least one wait class\");\n";
printf $h "#endif /* WAIT_EVENT_TYPES_H */\n";
close $h;
close $c;
diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c
index d9b8f34a355..eba7d338c1f 100644
--- a/src/backend/utils/activity/wait_event.c
+++ b/src/backend/utils/activity/wait_event.c
@@ -39,9 +39,6 @@ static const char *pgstat_get_wait_io(WaitEventIO w);
static uint32 local_my_wait_event_info;
uint32 *my_wait_event_info = &local_my_wait_event_info;
-#define WAIT_EVENT_CLASS_MASK 0xFF000000
-#define WAIT_EVENT_ID_MASK 0x0000FFFF
-
/*
* Hash tables for storing custom wait event ids and their names in
* shared memory.
diff --git a/src/include/utils/meson.build b/src/include/utils/meson.build
index 78c6b9b0a23..ff519242de7 100644
--- a/src/include/utils/meson.build
+++ b/src/include/utils/meson.build
@@ -2,7 +2,9 @@
wait_event_output = ['wait_event_types.h', 'pgstat_wait_event.c', 'wait_event_funcs_data.c']
wait_event_target = custom_target('wait_event_names',
- input: files('../../backend/utils/activity/wait_event_names.txt'),
+ input: files('../../backend/utils/activity/wait_event_names.txt',
+ '../../include/storage/lwlocklist.h',
+ '../../include/utils/wait_classes.h'),
output: wait_event_output,
command: [
perl, files('../../backend/utils/activity/generate-wait_event_types.pl'),
--
2.34.1
>From ae68cf0a0f256ee09e2e9f779d3e673dccef0486 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot...@gmail.com>
Date: Fri, 20 Jun 2025 11:24:32 +0000
Subject: [PATCH v1 2/4] Add wait events statistics
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Adding a new stat kind PGSTAT_KIND_WAIT_EVENT for the wait event statistics.
This new statistic kind is a fixed one because we know the maximum number of wait
events. Indeed:
* it does not take into account custom wait events as extensions have all they need
at their disposal to create custom stats on their own wait events should they
want to (limited by WAIT_EVENT_CUSTOM_HASH_MAX_SIZE though).
* it does not take into account LWLock > LWTRANCHE_FIRST_USER_DEFINED for the same
reasons as above. That said, there is no maximum limitation in LWLockNewTrancheId().
* we don't want to allocate memory in some places where the counters are
incremented (see 4feba03d8b9). We could still use the same implementation as for
backend statistics (i.e, make use of flush_static_cb) if we really want/need to
switch to variable stats.
Some notes about the current design/implementation done in this patch:
For the moment only the counters are added (an array of currently 285 counters),
, we’ll study/discuss about adding the timings once the counters are fully done.
I think we’d have more discussion/debate around the timings (should we add them
by default, add a new GUC, enable them at compilation time?…), that’s why only
the counters are in this patch.
I think it makes sense as the counters have merit on their own.
We currently have 273 wait events but the array is 285 long: the reason is that
some wait events classes have "holes".
For some backends type the wait events stats are not flushed (walwriter for
example), so we need to find additional places to flush the wait events stats.
A few questions:
* Do we need to serialize the stats based on their names (as for
PGSTAT_KIND_REPLSLOT)? This question is the same as "is the ordering preserved
if file stats format is not changed": I think the answer is yes (see f98dbdeb51d)
, which means there is no need for to_serialized_name/from_serialized_name.
* What if a new wait event is added? We'd need to change the stats file format,
unless: the wait event stats kind becomes a variable one or we change a bit the
way fixed stats are written/read to/from the stat file (we could add a new
field in the PgStat_KindInfo for example).
XXX: Bump stat file format
---
src/backend/utils/activity/Makefile | 1 +
src/backend/utils/activity/meson.build | 1 +
src/backend/utils/activity/pgstat.c | 18 ++
src/backend/utils/activity/pgstat_waitevent.c | 232 ++++++++++++++++++
src/backend/utils/activity/wait_event.c | 3 -
src/include/pgstat.h | 15 ++
src/include/utils/pgstat_internal.h | 20 ++
src/include/utils/pgstat_kind.h | 3 +-
src/include/utils/wait_event.h | 10 +
src/tools/pgindent/typedefs.list | 3 +
10 files changed, 302 insertions(+), 4 deletions(-)
create mode 100644 src/backend/utils/activity/pgstat_waitevent.c
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index f9849bebc98..e7fc1354c1f 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -31,6 +31,7 @@ OBJS = \
pgstat_shmem.o \
pgstat_slru.o \
pgstat_subscription.o \
+ pgstat_waitevent.o \
pgstat_wal.o \
pgstat_xact.o \
wait_event.o \
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index d8e56b49c24..8b9b4b4bdb2 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -16,6 +16,7 @@ backend_sources += files(
'pgstat_shmem.c',
'pgstat_slru.c',
'pgstat_subscription.c',
+ 'pgstat_waitevent.c',
'pgstat_wal.c',
'pgstat_xact.c',
)
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 8b57845e870..68c3eb3d894 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -479,6 +479,24 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.reset_all_cb = pgstat_wal_reset_all_cb,
.snapshot_cb = pgstat_wal_snapshot_cb,
},
+
+ [PGSTAT_KIND_WAIT_EVENT] = {
+ .name = "wait_event",
+
+ .fixed_amount = true,
+ .write_to_file = true,
+
+ .snapshot_ctl_off = offsetof(PgStat_Snapshot, wait_event),
+ .shared_ctl_off = offsetof(PgStat_ShmemControl, wait_event),
+ .shared_data_off = offsetof(PgStatShared_WaitEvent, stats),
+ .shared_data_len = sizeof(((PgStatShared_WaitEvent *) 0)->stats),
+
+ .flush_static_cb = pgstat_wait_event_flush_cb,
+ .have_static_pending_cb = pgstat_wait_event_have_pending_cb,
+ .init_shmem_cb = pgstat_wait_event_init_shmem_cb,
+ .reset_all_cb = pgstat_wait_event_reset_all_cb,
+ .snapshot_cb = pgstat_wait_event_snapshot_cb,
+ },
};
/*
diff --git a/src/backend/utils/activity/pgstat_waitevent.c b/src/backend/utils/activity/pgstat_waitevent.c
new file mode 100644
index 00000000000..a884784e43d
--- /dev/null
+++ b/src/backend/utils/activity/pgstat_waitevent.c
@@ -0,0 +1,232 @@
+/* -------------------------------------------------------------------------
+ *
+ * pgstat_waitevent.c
+ * Implementation of wait event statistics.
+ *
+ * This file contains the implementation of wait event statistics. It is kept
+ * separate from pgstat.c to enforce the line between the statistics access /
+ * storage implementation and the details about individual types of
+ * statistics.
+ *
+ * Copyright (c) 2001-2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/activity/pgstat_waitevent.c
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "utils/pgstat_internal.h"
+
+bool have_wait_event_stats = false;
+
+static PgStat_PendingWaitevent PendingWaitEventStats;
+
+/*
+ * Support function for the SQL-callable pgstat* functions. Returns
+ * a pointer to the wait events statistics struct.
+ */
+PgStat_WaitEvent *
+pgstat_fetch_stat_wait_event(void)
+{
+ pgstat_snapshot_fixed(PGSTAT_KIND_WAIT_EVENT);
+
+ return &pgStatLocal.snapshot.wait_event;
+}
+
+/*
+ * Returns a pointer to the first counter for a specific class.
+ */
+static PgStat_Counter *
+waitEventGetClassCounters(int64 *waitEventStats, int classId)
+{
+ int offset = WaitClassTable[classId].offSet;
+
+ return &waitEventStats[offset];
+}
+
+/*
+ * Returns a pointer to the counter for a specific wait event.
+ */
+static PgStat_Counter *
+waitEventGetCounter(int64 *waitEventStats, int classId, int eventId)
+{
+ int64 *classCounters;
+
+ Assert(classId >= 0 && classId < NB_WAITCLASSTABLE_ENTRIES);
+ Assert(eventId >= 0 && eventId < WaitClassTable[classId].numberOfEvents);
+
+ classCounters = waitEventGetClassCounters(waitEventStats, classId);
+
+ return &classCounters[eventId];
+}
+
+/*
+ * Increment a wait event stat counter.
+ */
+inline void
+waitEventIncrementCounter(uint32 wait_event_info)
+{
+ DecodedWaitInfo waitInfo;
+ PgStat_Counter *counter;
+ uint32 classId;
+ uint16 eventId;
+
+ classId = *my_wait_event_info & WAIT_EVENT_CLASS_MASK;
+ eventId = *my_wait_event_info & WAIT_EVENT_ID_MASK;
+
+ if (classId == 0 && eventId == 0)
+ return;
+
+ /* Don't take into account user defined LWLock in the stats */
+ if (classId == PG_WAIT_LWLOCK && eventId >= LWTRANCHE_FIRST_USER_DEFINED)
+ return;
+
+ /* Don't take into account custom wait event extension in the stats */
+ if (classId == PG_WAIT_EXTENSION && eventId >= WAIT_EVENT_CUSTOM_INITIAL_ID)
+ return;
+
+ /* Don't take account PG_WAIT_INJECTIONPOINT */
+ if (classId == PG_WAIT_INJECTIONPOINT)
+ return;
+
+ WAIT_EVENT_INFO_DECODE(waitInfo, wait_event_info);
+
+ counter = waitEventGetCounter(PendingWaitEventStats.counts, waitInfo.classId,
+ waitInfo.eventId);
+
+ (*counter)++;
+
+ have_wait_event_stats = true;
+}
+
+const char *
+get_wait_event_name_from_index(int index)
+{
+ /* Iterate through the WaitClassTable */
+ for (int classIdx = 0; classIdx < NB_WAITCLASSTABLE_ENTRIES; classIdx++)
+ {
+ int classOffset = WaitClassTable[classIdx].offSet;
+ int classSize = WaitClassTable[classIdx].numberOfEvents;
+
+ /* Skip empty entries */
+ if (WaitClassTable[classIdx].numberOfEvents == 0)
+ continue;
+
+ /* Check if the index falls within this class section */
+ if (index >= classOffset && index < classOffset + classSize)
+ {
+ /* Calculate the event ID within this class */
+ int eventId = index - classOffset;
+
+ return WaitClassTable[classIdx].eventNames[eventId];
+ }
+ }
+
+ Assert(false);
+ return "unknown";
+}
+
+/*
+ * Flush out locally pending wait event statistics
+ *
+ * If no stats have been recorded, this function returns false.
+ *
+ * If nowait is true, this function returns true if the lock could not be
+ * acquired. Otherwise, return false.
+ */
+bool
+pgstat_wait_event_flush_cb(bool nowait)
+{
+ PgStatShared_WaitEvent *stats_shmem = &pgStatLocal.shmem->wait_event;
+ int i;
+
+ if (!have_wait_event_stats)
+ return false;
+
+ if (!nowait)
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
+ return true;
+
+ for (i = 0; i < NB_WAITCLASSTABLE_SIZE; i++)
+ {
+ PgStat_WaitEvent *sharedent = &stats_shmem->stats;
+
+ sharedent->counts[i] += PendingWaitEventStats.counts[i];
+ }
+
+ /* done, clear the pending entry */
+ MemSet(PendingWaitEventStats.counts, 0, sizeof(PendingWaitEventStats.counts));
+
+ LWLockRelease(&stats_shmem->lock);
+
+ have_wait_event_stats = false;
+
+ return false;
+}
+
+void
+pgstat_wait_event_init_shmem_cb(void *stats)
+{
+ PgStatShared_WaitEvent *stat_shmem = (PgStatShared_WaitEvent *) stats;
+
+ LWLockInitialize(&stat_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
+void
+pgstat_wait_event_reset_all_cb(TimestampTz ts)
+{
+ for (int i = 0; i < NB_WAITCLASSTABLE_SIZE; i++)
+ {
+ LWLock *stats_lock = &pgStatLocal.shmem->wait_event.lock;
+ PgStat_Counter *counters = &pgStatLocal.shmem->wait_event.stats.counts[i];
+
+ LWLockAcquire(stats_lock, LW_EXCLUSIVE);
+
+ /*
+ * Use the lock in the first wait event to protect the reset timestamp
+ * as well.
+ */
+ if (i == 0)
+ pgStatLocal.shmem->wait_event.stats.stat_reset_timestamp = ts;
+
+ memset(counters, 0, sizeof(*counters));
+ LWLockRelease(stats_lock);
+ }
+}
+
+void
+pgstat_wait_event_snapshot_cb(void)
+{
+ for (int i = 0; i < NB_WAITCLASSTABLE_SIZE; i++)
+ {
+ LWLock *stats_lock = &pgStatLocal.shmem->wait_event.lock;
+ PgStat_Counter *sh_counters = &pgStatLocal.shmem->wait_event.stats.counts[i];
+ PgStat_Counter *counters_snap = &pgStatLocal.snapshot.wait_event.counts[i];
+
+ LWLockAcquire(stats_lock, LW_SHARED);
+
+ /*
+ * Use the lock in the first wait event to protect the reset timestamp
+ * as well.
+ */
+ if (i == 0)
+ pgStatLocal.snapshot.wait_event.stat_reset_timestamp =
+ pgStatLocal.shmem->wait_event.stats.stat_reset_timestamp;
+
+ /* using struct assignment due to better type safety */
+ *counters_snap = *sh_counters;
+ LWLockRelease(stats_lock);
+ }
+}
+
+/*
+ * Check if there any wait event stats waiting for flush.
+ */
+bool
+pgstat_wait_event_have_pending_cb(void)
+{
+ return have_wait_event_stats;
+}
diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c
index eba7d338c1f..613935f22a2 100644
--- a/src/backend/utils/activity/wait_event.c
+++ b/src/backend/utils/activity/wait_event.c
@@ -87,9 +87,6 @@ typedef struct WaitEventCustomCounterData
/* pointer to the shared memory */
static WaitEventCustomCounterData *WaitEventCustomCounter;
-/* first event ID of custom wait events */
-#define WAIT_EVENT_CUSTOM_INITIAL_ID 1
-
static uint32 WaitEventCustomNew(uint32 classId, const char *wait_event_name);
static const char *GetWaitEventCustomIdentifier(uint32 wait_event_info);
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 378f2f2c2ba..4b249fbbb73 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -339,6 +339,17 @@ typedef struct PgStat_IO
PgStat_BktypeIO stats[BACKEND_NUM_TYPES];
} PgStat_IO;
+typedef struct PgStat_WaitEvent
+{
+ TimestampTz stat_reset_timestamp;
+ PgStat_Counter counts[NB_WAITCLASSTABLE_SIZE];
+} PgStat_WaitEvent;
+
+typedef struct PgStat_PendingWaitEvent
+{
+ PgStat_Counter counts[NB_WAITCLASSTABLE_SIZE];
+} PgStat_PendingWaitevent;
+
typedef struct PgStat_StatDBEntry
{
PgStat_Counter xact_commit;
@@ -782,6 +793,10 @@ struct xl_xact_stats_item;
extern int pgstat_get_transactional_drops(bool isCommit, struct xl_xact_stats_item **items);
extern void pgstat_execute_transactional_drops(int ndrops, struct xl_xact_stats_item *items, bool is_redo);
+/*
+ * Functions in pgstat_waitevent.c
+ */
+extern PgStat_WaitEvent *pgstat_fetch_stat_wait_event(void);
/*
* Functions in pgstat_wal.c
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index d5557e6e998..33dc6e7ae05 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -403,6 +403,14 @@ typedef struct PgStatShared_IO
PgStat_IO stats;
} PgStatShared_IO;
+/* Shared-memory ready PgStat_WaitEvent */
+typedef struct PgStatShared_WaitEvent
+{
+ /* lock protects ->stats */
+ LWLock lock;
+ PgStat_WaitEvent stats;
+} PgStatShared_WaitEvent;
+
typedef struct PgStatShared_SLRU
{
/* lock protects ->stats */
@@ -501,6 +509,7 @@ typedef struct PgStat_ShmemControl
PgStatShared_IO io;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
+ PgStatShared_WaitEvent wait_event;
/*
* Custom stats data with fixed-numbered objects, indexed by (PgStat_Kind
@@ -535,6 +544,8 @@ typedef struct PgStat_Snapshot
PgStat_WalStats wal;
+ PgStat_WaitEvent wait_event;
+
/*
* Data in snapshot for custom fixed-numbered statistics, indexed by
* (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in
@@ -780,6 +791,15 @@ extern void pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 obji
extern PGDLLIMPORT PgStat_LocalState pgStatLocal;
+/*
+ * Functions in pgstat_waitevent.c
+ */
+
+extern bool pgstat_wait_event_flush_cb(bool nowait);
+extern void pgstat_wait_event_init_shmem_cb(void *stats);
+extern void pgstat_wait_event_reset_all_cb(TimestampTz ts);
+extern void pgstat_wait_event_snapshot_cb(void);
+extern bool pgstat_wait_event_have_pending_cb(void);
/*
* Implementation of inline functions declared above.
diff --git a/src/include/utils/pgstat_kind.h b/src/include/utils/pgstat_kind.h
index f44169fd5a3..848966111c6 100644
--- a/src/include/utils/pgstat_kind.h
+++ b/src/include/utils/pgstat_kind.h
@@ -38,9 +38,10 @@
#define PGSTAT_KIND_IO 10
#define PGSTAT_KIND_SLRU 11
#define PGSTAT_KIND_WAL 12
+#define PGSTAT_KIND_WAIT_EVENT 13
#define PGSTAT_KIND_BUILTIN_MIN PGSTAT_KIND_DATABASE
-#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_WAL
+#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_WAIT_EVENT
#define PGSTAT_KIND_BUILTIN_SIZE (PGSTAT_KIND_BUILTIN_MAX + 1)
/* Custom stats kinds */
diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h
index f5815b4994a..45f2347b91b 100644
--- a/src/include/utils/wait_event.h
+++ b/src/include/utils/wait_event.h
@@ -10,6 +10,8 @@
#ifndef WAIT_EVENT_H
#define WAIT_EVENT_H
+#include "storage/lwlock.h"
+
/* enums for wait events */
#include "utils/wait_event_types.h"
@@ -19,9 +21,14 @@ static inline void pgstat_report_wait_start(uint32 wait_event_info);
static inline void pgstat_report_wait_end(void);
extern void pgstat_set_wait_event_storage(uint32 *wait_event_info);
extern void pgstat_reset_wait_event_storage(void);
+extern void waitEventIncrementCounter(uint32 wait_event_info);
+extern const char *get_wait_event_name_from_index(int index);
extern PGDLLIMPORT uint32 *my_wait_event_info;
+extern PGDLLIMPORT bool have_wait_event_stats;
+/* first event ID of custom wait events */
+#define WAIT_EVENT_CUSTOM_INITIAL_ID 1
/*
* Wait Events - Extension, InjectionPoint
@@ -84,6 +91,9 @@ pgstat_report_wait_start(uint32 wait_event_info)
static inline void
pgstat_report_wait_end(void)
{
+ /* Increment the wait event counter */
+ waitEventIncrementCounter(*(volatile uint32 *) my_wait_event_info);
+
/* see pgstat_report_wait_start() */
*(volatile uint32 *) my_wait_event_info = 0;
}
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 32d6e718adc..c45fd61098b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2215,6 +2215,7 @@ PgStatShared_Relation
PgStatShared_ReplSlot
PgStatShared_SLRU
PgStatShared_Subscription
+PgStatShared_WaitEvent
PgStatShared_Wal
PgStat_ArchiverStats
PgStat_Backend
@@ -2235,6 +2236,7 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
+PgStat_PendingWaitevent
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
@@ -2250,6 +2252,7 @@ PgStat_SubXactStatus
PgStat_TableCounts
PgStat_TableStatus
PgStat_TableXactStatus
+PgStat_WaitEvent
PgStat_WalCounters
PgStat_WalStats
PgXmlErrorContext
--
2.34.1
>From 295c963b9dc7e94fc4967600eea8238fcb017340 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot...@gmail.com>
Date: Fri, 20 Jun 2025 12:11:21 +0000
Subject: [PATCH v1 3/4] Add the pg_stat_wait_event view
This new view reports wait events statistics (for non custom wait event).
This commit also adds documentation and regression tests.
XXX: Bump catalog version
---
doc/src/sgml/monitoring.sgml | 84 ++++++++++++++++++++++++++++
src/backend/catalog/system_views.sql | 8 +++
src/backend/utils/adt/pgstatfuncs.c | 51 +++++++++++++++++
src/include/catalog/pg_proc.dat | 9 +++
src/test/regress/expected/rules.out | 5 ++
src/test/regress/expected/stats.out | 48 ++++++++++++++++
src/test/regress/sql/stats.sql | 22 ++++++++
7 files changed, 227 insertions(+)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 4265a22d4de..4299dd3ffb2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -518,6 +518,14 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_wait_event</structname><indexterm><primary>pg_stat_wait_event</primary></indexterm></entry>
+ <entry>One row per each non custom wait event, showing statistics about wait event activity. See
+ <link linkend="monitoring-pg-stat-wait-event-view">
+ <structname>pg_stat_wait_event</structname></link> for details.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_wal</structname><indexterm><primary>pg_stat_wal</primary></indexterm></entry>
<entry>One row only, showing statistics about WAL activity. See
@@ -4822,6 +4830,76 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</sect2>
+ <sect2 id="monitoring-pg-stat-wait-event-view">
+ <title><structname>pg_stat_wait_event</structname></title>
+
+ <indexterm>
+ <primary>pg_stat_wait_event</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_stat_wait_event</structname> view will contain
+ one row for each non custom wait event, showing statistics about that wait
+ event.
+ </para>
+
+ <table id="pg-stat-wait-event-view" xreflabel="pg_stat_wait_event">
+ <title><structname>pg_stat_wait_event</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>type</structfield> <type>text</type>
+ </para>
+ <para>
+ Wait event type
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>name</structfield> <type>text</type>
+ </para>
+ <para>
+ Wait event name
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>counts</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times waiting on this wait event
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>stats_reset</structfield> <type>timestamp with time zone</type>
+ </para>
+ <para>
+ Time at which these statistics were last reset
+ </para></entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+
+ </sect2>
+
<sect2 id="monitoring-stats-functions">
<title>Statistics Functions</title>
@@ -5054,6 +5132,12 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structname>pg_stat_slru</structname> view.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>wait_event</literal>: Reset all the counters shown in the
+ <structname>pg_stat_wait_event</structname> view.
+ </para>
+ </listitem>
<listitem>
<para>
<literal>wal</literal>: Reset all the counters shown in the
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 08f780a2e63..b3e93723964 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -942,6 +942,14 @@ CREATE VIEW pg_stat_slru AS
s.stats_reset
FROM pg_stat_get_slru() s;
+CREATE VIEW pg_stat_wait_event AS
+ SELECT
+ s.type,
+ s.name,
+ s.counts,
+ s.stats_reset
+ FROM pg_stat_get_wait_event() s;
+
CREATE VIEW pg_stat_wal_receiver AS
SELECT
s.pid,
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 1c12ddbae49..cfa6aefd95e 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1743,6 +1743,54 @@ pg_stat_get_slru(PG_FUNCTION_ARGS)
return (Datum) 0;
}
+/*
+ * Returns statistics of wait events.
+ */
+Datum
+pg_stat_get_wait_event(PG_FUNCTION_ARGS)
+{
+#define PG_STAT_GET_WAIT_EVENTS_COLS 4
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ int i,
+ j;
+ PgStat_WaitEvent *stats;
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ /* request wait event stats from the cumulative stats system */
+ stats = pgstat_fetch_stat_wait_event();
+
+ for (i = 0; i < NB_WAITCLASSTABLE_ENTRIES; i++)
+ {
+ /* for each row */
+ Datum values[PG_STAT_GET_WAIT_EVENTS_COLS] = {0};
+ bool nulls[PG_STAT_GET_WAIT_EVENTS_COLS] = {0};
+ WaitClassTableEntry *class = &WaitClassTable[i];
+ int numWaitEvents;
+
+ numWaitEvents = class->numberOfEvents;
+
+ for (j = 0; j < numWaitEvents; j++)
+ {
+ const char *name;
+
+ name = get_wait_event_name_from_index(class->offSet + j);
+
+ if (!name)
+ continue;
+
+ values[0] = PointerGetDatum(cstring_to_text(class->className));
+ values[1] = PointerGetDatum(cstring_to_text(name));
+ values[2] = Int64GetDatum(stats->counts[class->offSet + j]);
+ values[3] = TimestampTzGetDatum(stats->stat_reset_timestamp);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
+ }
+ }
+
+ return (Datum) 0;
+}
+
#define PG_STAT_GET_XACT_RELENTRY_INT64(stat) \
Datum \
CppConcat(pg_stat_get_xact_,stat)(PG_FUNCTION_ARGS) \
@@ -1882,6 +1930,7 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
pgstat_reset_of_kind(PGSTAT_KIND_IO);
XLogPrefetchResetStats();
pgstat_reset_of_kind(PGSTAT_KIND_SLRU);
+ pgstat_reset_of_kind(PGSTAT_KIND_WAIT_EVENT);
pgstat_reset_of_kind(PGSTAT_KIND_WAL);
PG_RETURN_VOID();
@@ -1901,6 +1950,8 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS)
XLogPrefetchResetStats();
else if (strcmp(target, "slru") == 0)
pgstat_reset_of_kind(PGSTAT_KIND_SLRU);
+ else if (strcmp(target, "wait_event") == 0)
+ pgstat_reset_of_kind(PGSTAT_KIND_WAIT_EVENT);
else if (strcmp(target, "wal") == 0)
pgstat_reset_of_kind(PGSTAT_KIND_WAL);
else
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index fb4f7f50350..6ecea50a88c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6041,6 +6041,15 @@
proargnames => '{name,blks_zeroed,blks_hit,blks_read,blks_written,blks_exists,flushes,truncates,stats_reset}',
prosrc => 'pg_stat_get_slru' },
+{ oid => '8962', descr => 'statistics: information about wait events',
+ proname => 'pg_stat_get_wait_event', prorows => '300', proisstrict => 'f',
+ proretset => 't', provolatile => 's', proparallel => 'r',
+ prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,text,int8,timestamptz}',
+ proargmodes => '{o,o,o,o}',
+ proargnames => '{type,name,counts,stats_reset}',
+ prosrc => 'pg_stat_get_wait_event' },
+
{ oid => '2978', descr => 'statistics: number of function calls',
proname => 'pg_stat_get_function_calls', provolatile => 's',
proparallel => 'r', prorettype => 'int8', proargtypes => 'oid',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 6cf828ca8d0..e5f8938a176 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2277,6 +2277,11 @@ pg_stat_user_tables| SELECT relid,
total_autoanalyze_time
FROM pg_stat_all_tables
WHERE ((schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (schemaname !~ '^pg_toast'::text));
+pg_stat_wait_event| SELECT type,
+ name,
+ counts,
+ stats_reset
+ FROM pg_stat_get_wait_event() s(type, name, counts, stats_reset);
pg_stat_wal| SELECT wal_records,
wal_fpi,
wal_bytes,
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index 776f1ad0e53..26d72dda3f1 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -236,6 +236,54 @@ FROM prevstats AS pr;
(1 row)
COMMIT;
+----
+-- Basic tests for wait events statistics
+---
+-- ensure there is no wait events missing in pg_stat_wait_event
+select count(1) > 0 from pg_stat_wait_event
+ where name not in (select name from pg_wait_events
+ where type <> 'InjectionPoint' or type <> 'Extension')
+ and type <> 'Extension';
+ ?column?
+----------
+ f
+(1 row)
+
+-- Test that reset_shared with wait_event specified as the stats type works
+SELECT count(1) AS counts_wait_event FROM pg_stat_wait_event WHERE counts > 0 \gset
+SELECT :counts_wait_event > 0;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT pg_stat_reset_shared('wait_event');
+ pg_stat_reset_shared
+----------------------
+
+(1 row)
+
+SELECT count(1) < :counts_wait_event FROM pg_stat_wait_event WHERE counts > 0;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Test wait event counters are incremented
+CREATE TABLE wait_event_stats_test(id serial);
+INSERT INTO wait_event_stats_test DEFAULT VALUES;
+SELECT pg_stat_force_next_flush();
+ pg_stat_force_next_flush
+--------------------------
+
+(1 row)
+
+SELECT counts > 0 FROM pg_stat_wait_event WHERE name = 'WalWrite' and type = 'IO';
+ ?column?
+----------
+ t
+(1 row)
+
----
-- Basic tests for track_functions
---
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index 232ab8db8fa..0ea887a45fd 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -132,6 +132,28 @@ FROM prevstats AS pr;
COMMIT;
+----
+-- Basic tests for wait events statistics
+---
+
+-- ensure there is no wait events missing in pg_stat_wait_event
+select count(1) > 0 from pg_stat_wait_event
+ where name not in (select name from pg_wait_events
+ where type <> 'InjectionPoint' or type <> 'Extension')
+ and type <> 'Extension';
+
+-- Test that reset_shared with wait_event specified as the stats type works
+SELECT count(1) AS counts_wait_event FROM pg_stat_wait_event WHERE counts > 0 \gset
+SELECT :counts_wait_event > 0;
+SELECT pg_stat_reset_shared('wait_event');
+SELECT count(1) < :counts_wait_event FROM pg_stat_wait_event WHERE counts > 0;
+
+-- Test wait event counters are incremented
+CREATE TABLE wait_event_stats_test(id serial);
+INSERT INTO wait_event_stats_test DEFAULT VALUES;
+SELECT pg_stat_force_next_flush();
+SELECT counts > 0 FROM pg_stat_wait_event WHERE name = 'WalWrite' and type = 'IO';
+
----
-- Basic tests for track_functions
---
--
2.34.1
>From 8d9d8109bc94de487a9e0cdb31b4090bd25b5393 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot...@gmail.com>
Date: Fri, 27 Jun 2025 05:55:43 +0000
Subject: [PATCH v1 4/4] switching PGSTAT_KIND_WAIT_EVENT to variable sized
It might be better for PGSTAT_KIND_WAIT_EVENT to be a variable sized stats kind.
That way:
* It would be easier to add custom wait events if we want to
* It would be possible to add a new wait events without having to change the
stats file format
It uses the uint32 as the hash key while the hash key is defined as uint64: that
should not be an issue but this patch does explicit casting though.
That said it might be better to use all the 64 bits (means not have the half full
of zeroes) for the hash key (better hashing distribution?) then we could imagine
using something like:
((uint64) wait_event_info) | (((uint64) wait_event_info) << 32)
for the hash key.
---
.../activity/generate-wait_event_types.pl | 17 ++-
src/backend/utils/activity/pgstat.c | 35 +++--
src/backend/utils/activity/pgstat_waitevent.c | 123 ++++++++----------
src/backend/utils/adt/pgstatfuncs.c | 9 +-
src/include/pgstat.h | 32 +++--
src/include/utils/pgstat_internal.h | 22 ++--
src/include/utils/pgstat_kind.h | 16 +--
7 files changed, 129 insertions(+), 125 deletions(-)
diff --git a/src/backend/utils/activity/generate-wait_event_types.pl b/src/backend/utils/activity/generate-wait_event_types.pl
index c18693aa68b..623a7aa5e85 100644
--- a/src/backend/utils/activity/generate-wait_event_types.pl
+++ b/src/backend/utils/activity/generate-wait_event_types.pl
@@ -241,13 +241,22 @@ if ($gen_code)
';
+ my $wait_event_class_shift = 0;
+ my $temp_mask = $wait_event_class_mask;
+ while (($temp_mask & 1) == 0 && $temp_mask != 0)
+ {
+ $wait_event_class_shift++;
+ $temp_mask >>= 1;
+ }
+
printf $h $header_comment, 'wait_event_types.h';
printf $h "#ifndef WAIT_EVENT_TYPES_H\n";
printf $h "#define WAIT_EVENT_TYPES_H\n\n";
printf $h "#define WAIT_EVENT_CLASS_MASK 0x%08X\n",
$wait_event_class_mask;
- printf $h "#define WAIT_EVENT_ID_MASK 0x%08X\n\n",
- $wait_event_id_mask;
+ printf $h "#define WAIT_EVENT_ID_MASK 0x%08X\n", $wait_event_id_mask;
+ printf $h "#define WAIT_EVENT_CLASS_SHIFT %d\n\n",
+ $wait_event_class_shift;
printf $h "#include \"utils/wait_classes.h\"\n\n";
printf $c $header_comment, 'pgstat_wait_event.c';
@@ -363,6 +372,10 @@ typedef struct DecodedWaitInfo
d.classId = ((i) & WAIT_EVENT_CLASS_MASK) / (WAIT_EVENT_CLASS_MASK & (-WAIT_EVENT_CLASS_MASK)), \\
d.eventId = (i) & WAIT_EVENT_ID_MASK
+/* To encode wait_event_info from classId and eventId as integers */
+#define ENCODE_WAIT_EVENT_INFO(classId, eventId) \\
+ (((classId) << WAIT_EVENT_CLASS_SHIFT) | ((eventId) & WAIT_EVENT_ID_MASK))
+
/* To map wait event classes into the WaitClassTable */
typedef struct
{
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 68c3eb3d894..ca7a1d1b23e 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -375,6 +375,23 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.reset_timestamp_cb = pgstat_backend_reset_timestamp_cb,
},
+ [PGSTAT_KIND_WAIT_EVENT] = {
+ .name = "wait_event",
+
+ .fixed_amount = false,
+ .write_to_file = true,
+
+ .accessed_across_databases = true,
+
+ .shared_size = sizeof(PgStatShared_WaitEvent),
+ .shared_data_off = offsetof(PgStatShared_WaitEvent, stats),
+ .shared_data_len = sizeof(((PgStatShared_WaitEvent *) 0)->stats),
+
+ .have_static_pending_cb = pgstat_wait_event_have_pending_cb,
+ .flush_static_cb = pgstat_wait_event_flush_cb,
+ .reset_timestamp_cb = pgstat_wait_event_reset_timestamp_cb,
+ },
+
/* stats for fixed-numbered (mostly 1) objects */
[PGSTAT_KIND_ARCHIVER] = {
@@ -479,24 +496,6 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.reset_all_cb = pgstat_wal_reset_all_cb,
.snapshot_cb = pgstat_wal_snapshot_cb,
},
-
- [PGSTAT_KIND_WAIT_EVENT] = {
- .name = "wait_event",
-
- .fixed_amount = true,
- .write_to_file = true,
-
- .snapshot_ctl_off = offsetof(PgStat_Snapshot, wait_event),
- .shared_ctl_off = offsetof(PgStat_ShmemControl, wait_event),
- .shared_data_off = offsetof(PgStatShared_WaitEvent, stats),
- .shared_data_len = sizeof(((PgStatShared_WaitEvent *) 0)->stats),
-
- .flush_static_cb = pgstat_wait_event_flush_cb,
- .have_static_pending_cb = pgstat_wait_event_have_pending_cb,
- .init_shmem_cb = pgstat_wait_event_init_shmem_cb,
- .reset_all_cb = pgstat_wait_event_reset_all_cb,
- .snapshot_cb = pgstat_wait_event_snapshot_cb,
- },
};
/*
diff --git a/src/backend/utils/activity/pgstat_waitevent.c b/src/backend/utils/activity/pgstat_waitevent.c
index a884784e43d..0655e2bdfe2 100644
--- a/src/backend/utils/activity/pgstat_waitevent.c
+++ b/src/backend/utils/activity/pgstat_waitevent.c
@@ -28,11 +28,14 @@ static PgStat_PendingWaitevent PendingWaitEventStats;
* a pointer to the wait events statistics struct.
*/
PgStat_WaitEvent *
-pgstat_fetch_stat_wait_event(void)
+pgstat_fetch_stat_wait_event(uint32 wait_event_info)
{
- pgstat_snapshot_fixed(PGSTAT_KIND_WAIT_EVENT);
+ PgStat_WaitEvent *wait_event_entry;
- return &pgStatLocal.snapshot.wait_event;
+ wait_event_entry = (PgStat_WaitEvent *) pgstat_fetch_entry(PGSTAT_KIND_WAIT_EVENT,
+ InvalidOid, (uint64) wait_event_info);
+
+ return wait_event_entry;
}
/*
@@ -131,95 +134,81 @@ get_wait_event_name_from_index(int index)
/*
* Flush out locally pending wait event statistics
*
- * If no stats have been recorded, this function returns false.
- *
- * If nowait is true, this function returns true if the lock could not be
- * acquired. Otherwise, return false.
+ * Returns true if some statistics could not be flushed due to lock contention.
*/
+
bool
pgstat_wait_event_flush_cb(bool nowait)
{
- PgStatShared_WaitEvent *stats_shmem = &pgStatLocal.shmem->wait_event;
- int i;
+ PgStat_EntryRef *entry_ref;
+ bool could_not_be_flushed = false;
if (!have_wait_event_stats)
return false;
- if (!nowait)
- LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
- else if (!LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
- return true;
-
- for (i = 0; i < NB_WAITCLASSTABLE_SIZE; i++)
+ for (int classIdx = 0; classIdx < NB_WAITCLASSTABLE_ENTRIES; classIdx++)
{
- PgStat_WaitEvent *sharedent = &stats_shmem->stats;
+ WaitClassTableEntry *class;
+ int classOffset;
+ int classSize;
- sharedent->counts[i] += PendingWaitEventStats.counts[i];
- }
+ /* Skip empty entries */
+ if (WaitClassTable[classIdx].numberOfEvents == 0)
+ continue;
- /* done, clear the pending entry */
- MemSet(PendingWaitEventStats.counts, 0, sizeof(PendingWaitEventStats.counts));
+ class = &WaitClassTable[classIdx];
- LWLockRelease(&stats_shmem->lock);
+ classOffset = class->offSet;
+ classSize = class->numberOfEvents;
- have_wait_event_stats = false;
+ for (int eventId = 0; eventId < classSize; eventId++)
+ {
+ const char *name;
+ PgStatShared_WaitEvent *shwaiteventent;
+ PgStat_Counter *shstat;
+ PgStat_Counter pending_counter;
+ uint32 wait_event_info;
- return false;
-}
+ name = get_wait_event_name_from_index(classOffset + eventId);
-void
-pgstat_wait_event_init_shmem_cb(void *stats)
-{
- PgStatShared_WaitEvent *stat_shmem = (PgStatShared_WaitEvent *) stats;
+ if (!name)
+ continue;
- LWLockInitialize(&stat_shmem->lock, LWTRANCHE_PGSTATS_DATA);
-}
+ /* Build the wait_event_info */
+ wait_event_info = ENCODE_WAIT_EVENT_INFO(classIdx, eventId);
-void
-pgstat_wait_event_reset_all_cb(TimestampTz ts)
-{
- for (int i = 0; i < NB_WAITCLASSTABLE_SIZE; i++)
- {
- LWLock *stats_lock = &pgStatLocal.shmem->wait_event.lock;
- PgStat_Counter *counters = &pgStatLocal.shmem->wait_event.stats.counts[i];
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_WAIT_EVENT,
+ InvalidOid, (uint64) wait_event_info, nowait);
+
+ if (!entry_ref)
+ {
+ could_not_be_flushed = true;
+ continue;
+ }
- LWLockAcquire(stats_lock, LW_EXCLUSIVE);
+ shwaiteventent = (PgStatShared_WaitEvent *) entry_ref->shared_stats;
+ shstat = &shwaiteventent->stats.counts;
+ pending_counter = PendingWaitEventStats.counts[classOffset + eventId];
- /*
- * Use the lock in the first wait event to protect the reset timestamp
- * as well.
- */
- if (i == 0)
- pgStatLocal.shmem->wait_event.stats.stat_reset_timestamp = ts;
+ *shstat += pending_counter;
- memset(counters, 0, sizeof(*counters));
- LWLockRelease(stats_lock);
+ pgstat_unlock_entry(entry_ref);
+ }
}
+
+ /* done, clear the pending entry */
+ MemSet(PendingWaitEventStats.counts, 0, sizeof(PendingWaitEventStats.counts));
+
+ if (!could_not_be_flushed)
+ have_wait_event_stats = false;
+
+ return could_not_be_flushed;
}
void
-pgstat_wait_event_snapshot_cb(void)
+pgstat_wait_event_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts)
{
- for (int i = 0; i < NB_WAITCLASSTABLE_SIZE; i++)
- {
- LWLock *stats_lock = &pgStatLocal.shmem->wait_event.lock;
- PgStat_Counter *sh_counters = &pgStatLocal.shmem->wait_event.stats.counts[i];
- PgStat_Counter *counters_snap = &pgStatLocal.snapshot.wait_event.counts[i];
-
- LWLockAcquire(stats_lock, LW_SHARED);
-
- /*
- * Use the lock in the first wait event to protect the reset timestamp
- * as well.
- */
- if (i == 0)
- pgStatLocal.snapshot.wait_event.stat_reset_timestamp =
- pgStatLocal.shmem->wait_event.stats.stat_reset_timestamp;
-
- /* using struct assignment due to better type safety */
- *counters_snap = *sh_counters;
- LWLockRelease(stats_lock);
- }
+ ((PgStatShared_WaitEvent *) header)->stats.stat_reset_timestamp = ts;
}
/*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index cfa6aefd95e..5060090ee21 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1757,9 +1757,6 @@ pg_stat_get_wait_event(PG_FUNCTION_ARGS)
InitMaterializedSRF(fcinfo, 0);
- /* request wait event stats from the cumulative stats system */
- stats = pgstat_fetch_stat_wait_event();
-
for (i = 0; i < NB_WAITCLASSTABLE_ENTRIES; i++)
{
/* for each row */
@@ -1773,15 +1770,19 @@ pg_stat_get_wait_event(PG_FUNCTION_ARGS)
for (j = 0; j < numWaitEvents; j++)
{
const char *name;
+ uint32 wait_event_info;
name = get_wait_event_name_from_index(class->offSet + j);
if (!name)
continue;
+ wait_event_info = ENCODE_WAIT_EVENT_INFO(i, j);
+ stats = pgstat_fetch_stat_wait_event(wait_event_info);
+
values[0] = PointerGetDatum(cstring_to_text(class->className));
values[1] = PointerGetDatum(cstring_to_text(name));
- values[2] = Int64GetDatum(stats->counts[class->offSet + j]);
+ values[2] = Int64GetDatum(stats->counts);
values[3] = TimestampTzGetDatum(stats->stat_reset_timestamp);
tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 4b249fbbb73..f1883e1ec83 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -339,17 +339,6 @@ typedef struct PgStat_IO
PgStat_BktypeIO stats[BACKEND_NUM_TYPES];
} PgStat_IO;
-typedef struct PgStat_WaitEvent
-{
- TimestampTz stat_reset_timestamp;
- PgStat_Counter counts[NB_WAITCLASSTABLE_SIZE];
-} PgStat_WaitEvent;
-
-typedef struct PgStat_PendingWaitEvent
-{
- PgStat_Counter counts[NB_WAITCLASSTABLE_SIZE];
-} PgStat_PendingWaitevent;
-
typedef struct PgStat_StatDBEntry
{
PgStat_Counter xact_commit;
@@ -515,6 +504,25 @@ typedef struct PgStat_BackendPending
PgStat_PendingIO pending_io;
} PgStat_BackendPending;
+/* -------
+ * PgStat_WaitEvent Wait events statistics
+ * -------
+ */
+typedef struct PgStat_WaitEvent
+{
+ TimestampTz stat_reset_timestamp;
+ PgStat_Counter counts;
+} PgStat_WaitEvent;
+
+/* ---------
+ * PgStat_PendingWaitEvent Non-flushed wait events stats.
+ * ---------
+ */
+typedef struct PgStat_PendingWaitEvent
+{
+ PgStat_Counter counts[NB_WAITCLASSTABLE_SIZE];
+} PgStat_PendingWaitevent;
+
/*
* Functions in pgstat.c
*/
@@ -796,7 +804,7 @@ extern void pgstat_execute_transactional_drops(int ndrops, struct xl_xact_stats_
/*
* Functions in pgstat_waitevent.c
*/
-extern PgStat_WaitEvent *pgstat_fetch_stat_wait_event(void);
+extern PgStat_WaitEvent *pgstat_fetch_stat_wait_event(uint32 wait_event_info);
/*
* Functions in pgstat_wal.c
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 33dc6e7ae05..4c3ff60b8a9 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -403,14 +403,6 @@ typedef struct PgStatShared_IO
PgStat_IO stats;
} PgStatShared_IO;
-/* Shared-memory ready PgStat_WaitEvent */
-typedef struct PgStatShared_WaitEvent
-{
- /* lock protects ->stats */
- LWLock lock;
- PgStat_WaitEvent stats;
-} PgStatShared_WaitEvent;
-
typedef struct PgStatShared_SLRU
{
/* lock protects ->stats */
@@ -471,6 +463,12 @@ typedef struct PgStatShared_Backend
PgStat_Backend stats;
} PgStatShared_Backend;
+typedef struct PgStatShared_WaitEvent
+{
+ PgStatShared_Common header;
+ PgStat_WaitEvent stats;
+} PgStatShared_WaitEvent;
+
/*
* Central shared memory entry for the cumulative stats system.
*
@@ -509,7 +507,6 @@ typedef struct PgStat_ShmemControl
PgStatShared_IO io;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
- PgStatShared_WaitEvent wait_event;
/*
* Custom stats data with fixed-numbered objects, indexed by (PgStat_Kind
@@ -544,8 +541,6 @@ typedef struct PgStat_Snapshot
PgStat_WalStats wal;
- PgStat_WaitEvent wait_event;
-
/*
* Data in snapshot for custom fixed-numbered statistics, indexed by
* (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in
@@ -796,9 +791,8 @@ extern PGDLLIMPORT PgStat_LocalState pgStatLocal;
*/
extern bool pgstat_wait_event_flush_cb(bool nowait);
-extern void pgstat_wait_event_init_shmem_cb(void *stats);
-extern void pgstat_wait_event_reset_all_cb(TimestampTz ts);
-extern void pgstat_wait_event_snapshot_cb(void);
+extern void pgstat_wait_event_reset_timestamp_cb(PgStatShared_Common *header,
+ TimestampTz ts);
extern bool pgstat_wait_event_have_pending_cb(void);
/*
diff --git a/src/include/utils/pgstat_kind.h b/src/include/utils/pgstat_kind.h
index 848966111c6..90ae7d49325 100644
--- a/src/include/utils/pgstat_kind.h
+++ b/src/include/utils/pgstat_kind.h
@@ -30,18 +30,18 @@
#define PGSTAT_KIND_REPLSLOT 4 /* per-slot statistics */
#define PGSTAT_KIND_SUBSCRIPTION 5 /* per-subscription statistics */
#define PGSTAT_KIND_BACKEND 6 /* per-backend statistics */
+#define PGSTAT_KIND_WAIT_EVENT 7 /* wait events statistics */
/* stats for fixed-numbered objects */
-#define PGSTAT_KIND_ARCHIVER 7
-#define PGSTAT_KIND_BGWRITER 8
-#define PGSTAT_KIND_CHECKPOINTER 9
-#define PGSTAT_KIND_IO 10
-#define PGSTAT_KIND_SLRU 11
-#define PGSTAT_KIND_WAL 12
-#define PGSTAT_KIND_WAIT_EVENT 13
+#define PGSTAT_KIND_ARCHIVER 8
+#define PGSTAT_KIND_BGWRITER 9
+#define PGSTAT_KIND_CHECKPOINTER 10
+#define PGSTAT_KIND_IO 11
+#define PGSTAT_KIND_SLRU 12
+#define PGSTAT_KIND_WAL 13
#define PGSTAT_KIND_BUILTIN_MIN PGSTAT_KIND_DATABASE
-#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_WAIT_EVENT
+#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_WAL
#define PGSTAT_KIND_BUILTIN_SIZE (PGSTAT_KIND_BUILTIN_MAX + 1)
/* Custom stats kinds */
--
2.34.1
