Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Tom Lane]

Hamid Akhtar  writes:
> I wasn't aware of the meson.build file. Attached is the latest version of
> the patch that contains the updated meson.build.

Pushed with minor corrections, plus one major one: you missed the
point of aeaaf520f, that pageinspect functions that touch relations
need to be parallel-restricted not parallel-safe.

regards, tom lane

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Muhammad Usama]

The following review has been posted through the commitfest application:
make installcheck-world:  tested, passed
Implements feature:   tested, passed
Spec compliant:   tested, passed
Documentation:not tested

Looks good to me

The new status of this patch is: Ready for Committer

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Hamid Akhtar]

On Wed, 7 Dec 2022 at 13:01, Alvaro Herrera  wrote:

> On 2022-Dec-06, Andres Freund wrote:
>
> > Hi,
> >
> > On 2022-11-25 02:45:01 +0500, Hamid Akhtar wrote:
> > > Rebasing the patch to the tip of the master branch.
> >
> > This doesn't pass tests on cfbot. Looks like possibly some files are
> missing?
>
> The .sql file is there all right, but meson.build is not altered to be
> aware of it.


I wasn't aware of the meson.build file. Attached is the latest version of
the patch that contains the updated meson.build.

-- 
Hamid Akhtar,
Percona LLC, www.percona.com


pageinspect_btree_multipagestats-v7.patch
Description: Binary data

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Alvaro Herrera]

On 2022-Dec-06, Andres Freund wrote:

> Hi,
> 
> On 2022-11-25 02:45:01 +0500, Hamid Akhtar wrote:
> > Rebasing the patch to the tip of the master branch.
> 
> This doesn't pass tests on cfbot. Looks like possibly some files are missing?

The .sql file is there all right, but meson.build is not altered to be
aware of it.

-- 
Álvaro Herrera PostgreSQL Developer  —  https://www.EnterpriseDB.com/

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Andres Freund]

Hi,

On 2022-11-25 02:45:01 +0500, Hamid Akhtar wrote:
> Rebasing the patch to the tip of the master branch.

This doesn't pass tests on cfbot. Looks like possibly some files are missing?

https://api.cirrus-ci.com/v1/artifact/task/4916614353649664/testrun/build/testrun/pageinspect/regress/regression.diffs

diff -U3 /tmp/cirrus-ci-build/contrib/pageinspect/expected/page.out 
/tmp/cirrus-ci-build/build/testrun/pageinspect/regress/results/page.out
--- /tmp/cirrus-ci-build/contrib/pageinspect/expected/page.out  2022-12-06 
20:07:47.691479000 +
+++ /tmp/cirrus-ci-build/build/testrun/pageinspect/regress/results/page.out 
2022-12-06 20:11:42.955606000 +
@@ -1,4 +1,5 @@
 CREATE EXTENSION pageinspect;
+ERROR:  extension "pageinspect" has no installation script nor update path for 
version "1.12"
 -- Use a temp table so that effects of VACUUM are predictable
 CREATE TEMP TABLE test1 (a int, b int);
 INSERT INTO test1 VALUES (16777217, 131584);
@@ -6,236 +7,203 @@

Greetings,

Andres Freund

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Hamid Akhtar]

On Mon, 21 Nov 2022 at 17:34, Naeem Akhter  wrote:

> The following review has been posted through the commitfest application:
> make installcheck-world:  tested, passed
> Implements feature:   tested, passed
> Spec compliant:   not tested
> Documentation:not tested
>
> i've tested and verified the documentation.


Rebasing the patch to the tip of the master branch.


pageinspect_btree_multipagestats-v6.patch
Description: Binary data

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Naeem Akhter]

The following review has been posted through the commitfest application:
make installcheck-world:  tested, passed
Implements feature:   tested, passed
Spec compliant:   not tested
Documentation:not tested

i've tested and verified the documentation.

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Hamid Akhtar]

On Wed, 12 Oct 2022 at 10:51, Michael Paquier  wrote:

> On Mon, Sep 12, 2022 at 03:18:59PM -0400, Tom Lane wrote:
> > I spun up a 32-bit VM, since that had been on my to-do list anyway,
> > and it looks like I was right:
>
> This feedback has not been addressed and the thread is idle four
> weeks, so I have marked this CF entry as RwF.  Please feel free to
> resubmit once a new version of the patch is available.
>

Attaching the version 5 of the patch that addresses all 3 points raised by
Tom Lane earlier in the thread.
(1) Documentation is added.
(2) Passing "-1" for the number of blocks required now returns all the
remaining index pages after the starting block.
(3) The newly added test cases work for both 32-bit and 64-bit systems.



> --
> Michael
>


pageinspect_btree_multipagestats-v5.patch
Description: Binary data

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Michael Paquier]

On Mon, Sep 12, 2022 at 03:18:59PM -0400, Tom Lane wrote:
> I spun up a 32-bit VM, since that had been on my to-do list anyway,
> and it looks like I was right:

This feedback has not been addressed and the thread is idle four
weeks, so I have marked this CF entry as RwF.  Please feel free to
resubmit once a new version of the patch is available.
--
Michael


signature.asc
Description: PGP signature

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Tom Lane]

I wrote:
> 3. I think it's highly likely that the new test case is not portable.
> In particular a machine with MAXALIGN 4 would be likely to put a
> different number of tuples per page, or do the page split differently
> so that the page with fewer index tuples isn't page 3.  Unfortunately
> I don't seem to have a working setup like that right at the moment
> to verify; but I'd counsel trying this inside a VM or something to
> see if it's actually likely to survive on the buildfarm.

I spun up a 32-bit VM, since that had been on my to-do list anyway,
and it looks like I was right:

diff -U3 /usr/home/tgl/pgsql/contrib/pageinspect/expected/btree.out 
/usr/home/tgl/pgsql/contrib/pageinspect/results/btree.out
--- /usr/home/tgl/pgsql/contrib/pageinspect/expected/btree.out  2022-09-12 
15:15:40.432135000 -0400
+++ /usr/home/tgl/pgsql/contrib/pageinspect/results/btree.out   2022-09-12 
15:15:54.481549000 -0400
@@ -49,11 +49,11 @@
 -[ RECORD 1 ]-+-
 blkno | 1
 type  | l
-live_items| 367
+live_items| 458
 dead_items| 0
-avg_item_size | 16
+avg_item_size | 12
 page_size | 8192
-free_size | 808
+free_size | 820
 btpo_prev | 0
 btpo_next | 2
 btpo_level| 0
@@ -61,11 +61,11 @@
... etc etc ...

regards, tom lane

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Tom Lane]

Hamid Akhtar  writes:
> Attached is the rebased version of the patch
> (pageinspect_btree_multipagestats_03.patch) for the master branch.

I looked through this and cleaned up the code a little (attached).
There are still some issues before it could be considered committable:

1. Where's the documentation update?

2. As this stands, there's no nice way to ask for "all the pages".
If you specify a page count that's even one too large, you get an
error.  I think there's room for an easier-to-use way to do that.
We could say that the thing just silently stops at the last page,
so that you just need to write a large page count.  Or maybe it'd
be better to define a zero or negative page count as "all the rest",
while still insisting that a positive count refer to real pages.

3. I think it's highly likely that the new test case is not portable.
In particular a machine with MAXALIGN 4 would be likely to put a
different number of tuples per page, or do the page split differently
so that the page with fewer index tuples isn't page 3.  Unfortunately
I don't seem to have a working setup like that right at the moment
to verify; but I'd counsel trying this inside a VM or something to
see if it's actually likely to survive on the buildfarm.  I'm not
sure, but making the indexed column be int8 instead of int4 might
reduce the risks here.

regards, tom lane

diff --git a/contrib/pageinspect/Makefile b/contrib/pageinspect/Makefile
index 5c0736564a..069683caf6 100644
--- a/contrib/pageinspect/Makefile
+++ b/contrib/pageinspect/Makefile
@@ -13,7 +13,9 @@ OBJS = \
 	rawpage.o
 
 EXTENSION = pageinspect
-DATA =  pageinspect--1.9--1.10.sql pageinspect--1.8--1.9.sql \
+DATA = \
+	pageinspect--1.10--1.11.sql \
+	pageinspect--1.9--1.10.sql pageinspect--1.8--1.9.sql \
 	pageinspect--1.7--1.8.sql pageinspect--1.6--1.7.sql \
 	pageinspect--1.5.sql pageinspect--1.5--1.6.sql \
 	pageinspect--1.4--1.5.sql pageinspect--1.3--1.4.sql \
diff --git a/contrib/pageinspect/btreefuncs.c b/contrib/pageinspect/btreefuncs.c
index 9375d55e14..f52a6ac87e 100644
--- a/contrib/pageinspect/btreefuncs.c
+++ b/contrib/pageinspect/btreefuncs.c
@@ -46,6 +46,7 @@ PG_FUNCTION_INFO_V1(bt_page_items);
 PG_FUNCTION_INFO_V1(bt_page_items_bytea);
 PG_FUNCTION_INFO_V1(bt_page_stats_1_9);
 PG_FUNCTION_INFO_V1(bt_page_stats);
+PG_FUNCTION_INFO_V1(bt_multi_page_stats);
 
 #define IS_INDEX(r) ((r)->rd_rel->relkind == RELKIND_INDEX)
 #define IS_BTREE(r) ((r)->rd_rel->relam == BTREE_AM_OID)
@@ -80,6 +81,28 @@ typedef struct BTPageStat
 	BTCycleId	btpo_cycleid;
 } BTPageStat;
 
+/*
+ * cross-call data structure for SRF for page stats
+ */
+typedef struct ua_page_stats
+{
+	Oid			relid;
+	int64		blkno;
+	int64		blk_count;
+} ua_page_stats;
+
+/*
+ * cross-call data structure for SRF for page items
+ */
+typedef struct ua_page_items
+{
+	Page		page;
+	OffsetNumber offset;
+	bool		leafpage;
+	bool		rightmost;
+	TupleDesc	tupd;
+} ua_page_items;
+
 
 /* -
  * GetBTPageStatistics()
@@ -177,34 +200,15 @@ GetBTPageStatistics(BlockNumber blkno, Buffer buffer, BTPageStat *stat)
 }
 
 /* ---
- * bt_page_stats()
+ * bt_index_block_validate()
  *
- * Usage: SELECT * FROM bt_page_stats('t1_pkey', 1);
+ * Validate index type is btree and block number
+ * is within a valid block number range.
  * ---
  */
-static Datum
-bt_page_stats_internal(PG_FUNCTION_ARGS, enum pageinspect_version ext_version)
+static void
+bt_index_block_validate(Relation rel, int64 blkno)
 {
-	text	   *relname = PG_GETARG_TEXT_PP(0);
-	int64		blkno = (ext_version == PAGEINSPECT_V1_8 ? PG_GETARG_UINT32(1) : PG_GETARG_INT64(1));
-	Buffer		buffer;
-	Relation	rel;
-	RangeVar   *relrv;
-	Datum		result;
-	HeapTuple	tuple;
-	TupleDesc	tupleDesc;
-	int			j;
-	char	   *values[11];
-	BTPageStat	stat;
-
-	if (!superuser())
-		ereport(ERROR,
-(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
- errmsg("must be superuser to use pageinspect functions")));
-
-	relrv = makeRangeVarFromNameList(textToQualifiedNameList(relname));
-	rel = relation_openrv(relrv, AccessShareLock);
-
 	if (!IS_INDEX(rel) || !IS_BTREE(rel))
 		ereport(ERROR,
 (errcode(ERRCODE_WRONG_OBJECT_TYPE),
@@ -232,6 +236,39 @@ bt_page_stats_internal(PG_FUNCTION_ARGS, enum pageinspect_version ext_version)
  errmsg("block 0 is a meta page")));
 
 	CHECK_RELATION_BLOCK_RANGE(rel, blkno);
+}
+
+/* ---
+ * bt_page_stats()
+ *
+ * Usage: SELECT * FROM bt_page_stats('t1_pkey', 1);
+ * Arguments are index relation name and block number
+ * ---
+ */
+static Datum
+bt_page_stats_internal(PG_FUNCTION_ARGS, enum pageinspect_version ext_version)
+{
+	text	   *relname = PG_GETARG_TEXT_PP(0);
+	int64		blkno = (ext_version == PAGEINSPECT_V1_8 ? PG_GETARG_UINT32(1) : PG_GETARG_INT64(1));
+	Buffer		buffer;
+	Relation	rel;
+	

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Hamid Akhtar]

On Tue, 6 Sept 2022 at 11:25, Ibrar Ahmed  wrote:

>
>
> On Mon, Aug 1, 2022 at 11:29 PM Naeem Akhter 
> wrote:
>
>> The following review has been posted through the commitfest application:
>> make installcheck-world:  tested, passed
>> Implements feature:   tested, passed
>> Spec compliant:   not tested
>> Documentation:not tested
>>
>> Looks good to me.
>>
>> The new status of this patch is: Ready for Committer
>>
>
> The patch has a compilation error on the latest code base, please rebase
> your patch.
>
> [03:08:46.087] /tmp/cirrus-ci-build/contrib/cube/cubeparse.h:77:27: error:
> ‘result’ was not declared in this scope
> [03:08:46.087] 77 | int cube_yyparse (NDBOX **result, Size scanbuflen);
> [03:08:46.087] | ^~
> [03:08:46.087] /tmp/cirrus-ci-build/contrib/cube/cubeparse.h:77:40: error:
> expected primary-expression before ‘scanbuflen’
> [03:08:46.087] 77 | int cube_yyparse (NDBOX **result, Size scanbuflen);
> ...
>
> --
> Ibrar Ahmed
>

The compilation and regression are working fine. I have verified it against
the tip of the master branch [commit: 57796a0f].

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Ibrar Ahmed]

On Mon, Aug 1, 2022 at 11:29 PM Naeem Akhter 
wrote:

> The following review has been posted through the commitfest application:
> make installcheck-world:  tested, passed
> Implements feature:   tested, passed
> Spec compliant:   not tested
> Documentation:not tested
>
> Looks good to me.
>
> The new status of this patch is: Ready for Committer
>

The patch has a compilation error on the latest code base, please rebase
your patch.

[03:08:46.087] /tmp/cirrus-ci-build/contrib/cube/cubeparse.h:77:27: error:
‘result’ was not declared in this scope
[03:08:46.087] 77 | int cube_yyparse (NDBOX **result, Size scanbuflen);
[03:08:46.087] | ^~
[03:08:46.087] /tmp/cirrus-ci-build/contrib/cube/cubeparse.h:77:40: error:
expected primary-expression before ‘scanbuflen’
[03:08:46.087] 77 | int cube_yyparse (NDBOX **result, Size scanbuflen);
...

-- 
Ibrar Ahmed

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Naeem Akhter]

The following review has been posted through the commitfest application:
make installcheck-world:  tested, passed
Implements feature:   tested, passed
Spec compliant:   not tested
Documentation:not tested

Looks good to me.

The new status of this patch is: Ready for Committer

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Hamid Akhtar]

On Sun, 31 Jul 2022 at 18:00, Hamid Akhtar  wrote:

>
>
> On Thu, 28 Jul 2022 at 00:36, Tom Lane  wrote:
>
>> Hamid Akhtar  writes:
>> > That attached patch is based on the master branch. It makes the
>> following
>> > changes to the pageinspect contrib module:
>> > - Updates bt_page_stats_internal function to accept 3 arguments instead
>> of
>> > 2.
>> > - The function now uses SRF macros to return a set rather than a single
>> > row. The function call now requires specifying column names.
>>
>> FWIW, I think you'd be way better off changing the function name, say
>> to bt_multi_page_stats().  Overloading the name this way is going to
>> lead to great confusion, e.g. somebody who fat-fingers the number of
>> output arguments in a JDBC call could see confusing results due to
>> invoking the wrong one of the two functions.  Also, I'm not quite sure
>> what you mean by "The function call now requires specifying column
>> names", but it doesn't sound like an acceptable restriction from a
>> compatibility standpoint.  If a different name dodges that issue then
>> it's clearly a better way.
>>
>> regards, tom lane
>>
>
> Attached please find the latest version of the patch;
> pageinspect_btree_multipagestats_02.patch.
>
> It no longer modifies the existing bt_page_stats function. Instead, it
> introduces a new function bt_multi_page_stats as you had suggested. The
> function expects three arguments where the first argument is the index name
> followed by block number and number of blocks to be returned.
>
> Please ignore this statement. It was a typo.
> "The function call now requires specifying column names"
>

Attached is the rebased version of the patch
(pageinspect_btree_multipagestats_03.patch) for the master branch.


>
>
diff --git a/contrib/pageinspect/Makefile b/contrib/pageinspect/Makefile
index 5c07365..31d7b1c 100644
--- a/contrib/pageinspect/Makefile
+++ b/contrib/pageinspect/Makefile
@@ -13,12 +13,12 @@ OBJS = \
 	rawpage.o
 
 EXTENSION = pageinspect
-DATA =  pageinspect--1.9--1.10.sql pageinspect--1.8--1.9.sql \
-	pageinspect--1.7--1.8.sql pageinspect--1.6--1.7.sql \
-	pageinspect--1.5.sql pageinspect--1.5--1.6.sql \
-	pageinspect--1.4--1.5.sql pageinspect--1.3--1.4.sql \
-	pageinspect--1.2--1.3.sql pageinspect--1.1--1.2.sql \
-	pageinspect--1.0--1.1.sql
+DATA =  pageinspect--1.10--1.11.sql pageinspect--1.9--1.10.sql \
+	pageinspect--1.8--1.9.sql pageinspect--1.7--1.8.sql \
+	pageinspect--1.6--1.7.sql pageinspect--1.5.sql \
+	pageinspect--1.5--1.6.sql pageinspect--1.4--1.5.sql \
+	pageinspect--1.3--1.4.sql pageinspect--1.2--1.3.sql \
+	pageinspect--1.1--1.2.sql pageinspect--1.0--1.1.sql
 PGFILEDESC = "pageinspect - functions to inspect contents of database pages"
 
 REGRESS = page btree brin gin gist hash checksum oldextversions
diff --git a/contrib/pageinspect/btreefuncs.c b/contrib/pageinspect/btreefuncs.c
index 9375d55..f7ec95c 100644
--- a/contrib/pageinspect/btreefuncs.c
+++ b/contrib/pageinspect/btreefuncs.c
@@ -46,6 +46,7 @@ PG_FUNCTION_INFO_V1(bt_page_items);
 PG_FUNCTION_INFO_V1(bt_page_items_bytea);
 PG_FUNCTION_INFO_V1(bt_page_stats_1_9);
 PG_FUNCTION_INFO_V1(bt_page_stats);
+PG_FUNCTION_INFO_V1(bt_multi_page_stats);
 
 #define IS_INDEX(r) ((r)->rd_rel->relkind == RELKIND_INDEX)
 #define IS_BTREE(r) ((r)->rd_rel->relam == BTREE_AM_OID)
@@ -80,6 +81,26 @@ typedef struct BTPageStat
 	BTCycleId	btpo_cycleid;
 } BTPageStat;
 
+/*
+ * cross-call data structure for SRF for page stats
+ */
+typedef struct ua_page_stats
+{
+	uint64		blkno;
+	uint64		blk_count;
+}			ua_page_stats;
+
+/*
+ * cross-call data structure for SRF for page items
+ */
+typedef struct ua_page_items
+{
+	Page		page;
+	OffsetNumber offset;
+	bool		leafpage;
+	bool		rightmost;
+	TupleDesc	tupd;
+}			ua_page_items;
 
 /* -
  * GetBTPageStatistics()
@@ -177,34 +198,15 @@ GetBTPageStatistics(BlockNumber blkno, Buffer buffer, BTPageStat *stat)
 }
 
 /* ---
- * bt_page_stats()
+ * bt_index_block_validate()
  *
- * Usage: SELECT * FROM bt_page_stats('t1_pkey', 1);
+ * Validate index type is btree and block number
+ * is within a valid block number range.
  * ---
  */
-static Datum
-bt_page_stats_internal(PG_FUNCTION_ARGS, enum pageinspect_version ext_version)
+static void
+bt_index_block_validate(Relation rel, int64 blkno)
 {
-	text	   *relname = PG_GETARG_TEXT_PP(0);
-	int64		blkno = (ext_version == PAGEINSPECT_V1_8 ? PG_GETARG_UINT32(1) : PG_GETARG_INT64(1));
-	Buffer		buffer;
-	Relation	rel;
-	RangeVar   *relrv;
-	Datum		result;
-	HeapTuple	tuple;
-	TupleDesc	tupleDesc;
-	int			j;
-	char	   *values[11];
-	BTPageStat	stat;
-
-	if (!superuser())
-		ereport(ERROR,
-(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
- errmsg("must be superuser to use pageinspect functions")));
-
-	relrv = makeRangeVarFromNameList(textToQualifiedNameList(relname));
-	rel = 

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Hamid Akhtar]

On Thu, 28 Jul 2022 at 00:36, Tom Lane  wrote:

> Hamid Akhtar  writes:
> > That attached patch is based on the master branch. It makes the following
> > changes to the pageinspect contrib module:
> > - Updates bt_page_stats_internal function to accept 3 arguments instead
> of
> > 2.
> > - The function now uses SRF macros to return a set rather than a single
> > row. The function call now requires specifying column names.
>
> FWIW, I think you'd be way better off changing the function name, say
> to bt_multi_page_stats().  Overloading the name this way is going to
> lead to great confusion, e.g. somebody who fat-fingers the number of
> output arguments in a JDBC call could see confusing results due to
> invoking the wrong one of the two functions.  Also, I'm not quite sure
> what you mean by "The function call now requires specifying column
> names", but it doesn't sound like an acceptable restriction from a
> compatibility standpoint.  If a different name dodges that issue then
> it's clearly a better way.
>
> regards, tom lane
>

Attached please find the latest version of the patch;
pageinspect_btree_multipagestats_02.patch.

It no longer modifies the existing bt_page_stats function. Instead, it
introduces a new function bt_multi_page_stats as you had suggested. The
function expects three arguments where the first argument is the index name
followed by block number and number of blocks to be returned.

Please ignore this statement. It was a typo.
"The function call now requires specifying column names"
diff --git a/contrib/pageinspect/Makefile b/contrib/pageinspect/Makefile
index 5c07365..31d7b1c 100644
--- a/contrib/pageinspect/Makefile
+++ b/contrib/pageinspect/Makefile
@@ -13,12 +13,12 @@ OBJS = \
 	rawpage.o
 
 EXTENSION = pageinspect
-DATA =  pageinspect--1.9--1.10.sql pageinspect--1.8--1.9.sql \
-	pageinspect--1.7--1.8.sql pageinspect--1.6--1.7.sql \
-	pageinspect--1.5.sql pageinspect--1.5--1.6.sql \
-	pageinspect--1.4--1.5.sql pageinspect--1.3--1.4.sql \
-	pageinspect--1.2--1.3.sql pageinspect--1.1--1.2.sql \
-	pageinspect--1.0--1.1.sql
+DATA =  pageinspect--1.10--1.11.sql pageinspect--1.9--1.10.sql \
+	pageinspect--1.8--1.9.sql pageinspect--1.7--1.8.sql \
+	pageinspect--1.6--1.7.sql pageinspect--1.5.sql \
+	pageinspect--1.5--1.6.sql pageinspect--1.4--1.5.sql \
+	pageinspect--1.3--1.4.sql pageinspect--1.2--1.3.sql \
+	pageinspect--1.1--1.2.sql pageinspect--1.0--1.1.sql
 PGFILEDESC = "pageinspect - functions to inspect contents of database pages"
 
 REGRESS = page btree brin gin gist hash checksum oldextversions
diff --git a/contrib/pageinspect/btreefuncs.c b/contrib/pageinspect/btreefuncs.c
index 62f2c1b..5dd216e 100644
--- a/contrib/pageinspect/btreefuncs.c
+++ b/contrib/pageinspect/btreefuncs.c
@@ -46,6 +46,7 @@ PG_FUNCTION_INFO_V1(bt_page_items);
 PG_FUNCTION_INFO_V1(bt_page_items_bytea);
 PG_FUNCTION_INFO_V1(bt_page_stats_1_9);
 PG_FUNCTION_INFO_V1(bt_page_stats);
+PG_FUNCTION_INFO_V1(bt_multi_page_stats);
 
 #define IS_INDEX(r) ((r)->rd_rel->relkind == RELKIND_INDEX)
 #define IS_BTREE(r) ((r)->rd_rel->relam == BTREE_AM_OID)
@@ -80,6 +81,26 @@ typedef struct BTPageStat
 	BTCycleId	btpo_cycleid;
 } BTPageStat;
 
+/*
+ * cross-call data structure for SRF for page stats
+ */
+typedef struct ua_page_stats
+{
+	uint64		blkno;
+	uint64		blk_count;
+}			ua_page_stats;
+
+/*
+ * cross-call data structure for SRF for page items
+ */
+typedef struct ua_page_items
+{
+	Page		page;
+	OffsetNumber offset;
+	bool		leafpage;
+	bool		rightmost;
+	TupleDesc	tupd;
+}			ua_page_items;
 
 /* -
  * GetBTPageStatistics()
@@ -177,34 +198,15 @@ GetBTPageStatistics(BlockNumber blkno, Buffer buffer, BTPageStat *stat)
 }
 
 /* ---
- * bt_page_stats()
+ * bt_index_block_validate()
  *
- * Usage: SELECT * FROM bt_page_stats('t1_pkey', 1);
+ * Validate index type is btree and block number
+ * is within a valid block number range.
  * ---
  */
-static Datum
-bt_page_stats_internal(PG_FUNCTION_ARGS, enum pageinspect_version ext_version)
+static void
+bt_index_block_validate(Relation rel, int64 blkno)
 {
-	text	   *relname = PG_GETARG_TEXT_PP(0);
-	int64		blkno = (ext_version == PAGEINSPECT_V1_8 ? PG_GETARG_UINT32(1) : PG_GETARG_INT64(1));
-	Buffer		buffer;
-	Relation	rel;
-	RangeVar   *relrv;
-	Datum		result;
-	HeapTuple	tuple;
-	TupleDesc	tupleDesc;
-	int			j;
-	char	   *values[11];
-	BTPageStat	stat;
-
-	if (!superuser())
-		ereport(ERROR,
-(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
- errmsg("must be superuser to use pageinspect functions")));
-
-	relrv = makeRangeVarFromNameList(textToQualifiedNameList(relname));
-	rel = relation_openrv(relrv, AccessShareLock);
-
 	if (!IS_INDEX(rel) || !IS_BTREE(rel))
 		ereport(ERROR,
 (errcode(ERRCODE_WRONG_OBJECT_TYPE),
@@ -232,6 +234,38 @@ bt_page_stats_internal(PG_FUNCTION_ARGS, enum pageinspect_version 

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Tom Lane]

Hamid Akhtar  writes:
> That attached patch is based on the master branch. It makes the following
> changes to the pageinspect contrib module:
> - Updates bt_page_stats_internal function to accept 3 arguments instead of
> 2.
> - The function now uses SRF macros to return a set rather than a single
> row. The function call now requires specifying column names.

FWIW, I think you'd be way better off changing the function name, say
to bt_multi_page_stats().  Overloading the name this way is going to
lead to great confusion, e.g. somebody who fat-fingers the number of
output arguments in a JDBC call could see confusing results due to
invoking the wrong one of the two functions.  Also, I'm not quite sure
what you mean by "The function call now requires specifying column
names", but it doesn't sound like an acceptable restriction from a
compatibility standpoint.  If a different name dodges that issue then
it's clearly a better way.

regards, tom lane

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Hamid Akhtar]

On Fri, 1 Jul 2022 at 13:01, Drouvot, Bertrand  wrote:

> Hi,
> On 6/30/22 10:24 AM, Hamid Akhtar wrote:
>
> On Mon, 27 Jun 2022 at 15:52, Bharath Rupireddy <
> bharath.rupireddyforpostg...@gmail.com> wrote:
>
>> On Mon, Jun 27, 2022 at 1:40 PM Drouvot, Bertrand 
>> wrote:
>> >
>> > Hi,
>> >
>> > On 6/27/22 9:31 AM, Hamid Akhtar wrote:
>> >
>> >
>> > Hello Hackers,
>> >
>> > While working on one of my blogs on the B-Tree indexes, I needed to
>> look at a range of B-Tree page statistics. So the goto solution was to use
>> pageinspect. However, reviewing stats for multiple pages meant issuing
>> multiple queries.
>>
>> +1 to improve the API.
>>
> I think it makes sense too.
>
> But what about the other pageinspect's functions that also use a single
> blkno as parameter? Should not the patch also takes care of them?
>
> I've started working on that. But it's going to be a much bigger change
with a lot of code refactoring.

So, taking this one step at a time, IMHO, this patch is good to be reviewed
now.


> Regards,
>
> Bertrand
>

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Hamid Akhtar]

On Thu, 30 Jun 2022 at 14:27, Bharath Rupireddy <
bharath.rupireddyforpostg...@gmail.com> wrote:

> On Thu, Jun 30, 2022 at 1:54 PM Hamid Akhtar 
> wrote:
> >
> >> Do we have any
> >> difference in the execution times for the above query vs the new
> >> function introduced in the v1 patch? If there's not much difference, I
> >> would suggest adding an SQL function around the generate_series
> >> approach in the pageinspect extension for better and easier usability.
> >
> >
> > Based on some basic SQL execution time comparison of the two approaches,
> I see that the API change, on average, is around 40% faster than the SQL.
> >
> > CREATE TABLE test2 AS (SELECT generate_series(1, 500) AS col1);
> > CREATE INDEX test2_col1_idx ON test2(col1);
> >
> > EXPLAIN ANALYZE
> > SELECT * FROM bt_page_stats('test2_col1_idx', 1, 5000);
> >
> > EXPLAIN ANALYZE
> > SELECT * FROM GENERATE_SERIES(1, 5000) blkno,
> bt_page_stats('test2_col1_idx',blkno::int);
> >
> > For me, the API change returns back the data in around 74ms whereas the
> SQL returns it in 102ms. So considering this and as you mentioned, the
> alternative may not be that obvious to everyone, it is a fair improvement.
>
> I'm wondering what happens with a bit of huge data and different test
> cases each test case executed, say, 2 or 3 times.
>
> If the difference in execution times is always present, then the API
> approach or changing the core function would make more sense.
>

Technically, AFAIK, the performance difference will always be there.
Firstly, in the API change, there is no additional overhead of the
generate_series function. Additionally, with API change, looping over the
pages has a smaller overhead when compared with the overhead of the SQL
approach.


>
> Regards,
> Bharath Rupireddy.
>

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Bharath Rupireddy]

On Thu, Jun 30, 2022 at 1:54 PM Hamid Akhtar  wrote:
>
>> Do we have any
>> difference in the execution times for the above query vs the new
>> function introduced in the v1 patch? If there's not much difference, I
>> would suggest adding an SQL function around the generate_series
>> approach in the pageinspect extension for better and easier usability.
>
>
> Based on some basic SQL execution time comparison of the two approaches, I 
> see that the API change, on average, is around 40% faster than the SQL.
>
> CREATE TABLE test2 AS (SELECT generate_series(1, 500) AS col1);
> CREATE INDEX test2_col1_idx ON test2(col1);
>
> EXPLAIN ANALYZE
> SELECT * FROM bt_page_stats('test2_col1_idx', 1, 5000);
>
> EXPLAIN ANALYZE
> SELECT * FROM GENERATE_SERIES(1, 5000) blkno, 
> bt_page_stats('test2_col1_idx',blkno::int);
>
> For me, the API change returns back the data in around 74ms whereas the SQL 
> returns it in 102ms. So considering this and as you mentioned, the 
> alternative may not be that obvious to everyone, it is a fair improvement.

I'm wondering what happens with a bit of huge data and different test
cases each test case executed, say, 2 or 3 times.

If the difference in execution times is always present, then the API
approach or changing the core function would make more sense.

Regards,
Bharath Rupireddy.

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Hamid Akhtar]

On Mon, 27 Jun 2022 at 15:52, Bharath Rupireddy <
bharath.rupireddyforpostg...@gmail.com> wrote:

> On Mon, Jun 27, 2022 at 1:40 PM Drouvot, Bertrand 
> wrote:
> >
> > Hi,
> >
> > On 6/27/22 9:31 AM, Hamid Akhtar wrote:
> >
> >
> > Hello Hackers,
> >
> > While working on one of my blogs on the B-Tree indexes, I needed to look
> at a range of B-Tree page statistics. So the goto solution was to use
> pageinspect. However, reviewing stats for multiple pages meant issuing
> multiple queries.
>
> +1 to improve the API.
>
> > I felt that there's an opportunity for improvement in the extension by
> extending the API to output the statistics for multiple pages with a single
> query.
> >
> > That attached patch is based on the master branch. It makes the
> following changes to the pageinspect contrib module:
> > - Updates bt_page_stats_internal function to accept 3 arguments instead
> of 2.
> > - The function now uses SRF macros to return a set rather than a single
> row. The function call now requires specifying column names.
> >
> > The extension version is bumped to 1.11 (PAGEINSPECT_V1_11).
> > To maintain backward compatibility, for versions below 1.11, the
> multi-call mechanism is ended to keep the old behavior consistent.
> >
> > Regression test cases for the module are updated as well as part of this
> change. Here is a subset of queries that are added to the btree.sql test
> case file for pageinspect.
> >
> > 
> > CREATE TABLE test2 AS (SELECT generate_series(1, 5000) AS col1);
> > CREATE INDEX test2_col1_idx ON test2(col1);
> > SELECT * FROM bt_page_stats('test2_col1_idx', 1, 2);
> >
> > For example, this could be written as:
> >
> > select * from
> > generate_series(1, 2) blkno ,
> > bt_page_stats('test2_col1_idx',blkno::int);
> >
> > Or, if one wants to inspect to whole relation, something like:
> >
> > select * from
> > generate_series(1, pg_relation_size('test2_col1_idx'::regclass::text) /
> 8192 - 1) blkno ,
> > bt_page_stats('test2_col1_idx',blkno::int);
>
> Good one. But not all may know the alternatives.


+1


> Do we have any
> difference in the execution times for the above query vs the new
> function introduced in the v1 patch? If there's not much difference, I
> would suggest adding an SQL function around the generate_series
> approach in the pageinspect extension for better and easier usability.
>

Based on some basic SQL execution time comparison of the two approaches, I
see that the API change, on average, is around 40% faster than the SQL.

CREATE TABLE test2 AS (SELECT generate_series(1, 500) AS col1);
CREATE INDEX test2_col1_idx ON test2(col1);

EXPLAIN ANALYZE
SELECT * FROM bt_page_stats('test2_col1_idx', 1, 5000);

EXPLAIN ANALYZE
SELECT * FROM GENERATE_SERIES(1, 5000) blkno,
bt_page_stats('test2_col1_idx',blkno::int);

For me, the API change returns back the data in around 74ms whereas the SQL
returns it in 102ms. So considering this and as you mentioned, the
alternative may not be that obvious to everyone, it is a fair improvement.


>
> Regards,
> Bharath Rupireddy.
>

Re: Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Bharath Rupireddy]

On Mon, Jun 27, 2022 at 1:40 PM Drouvot, Bertrand  wrote:
>
> Hi,
>
> On 6/27/22 9:31 AM, Hamid Akhtar wrote:
>
>
> Hello Hackers,
>
> While working on one of my blogs on the B-Tree indexes, I needed to look at a 
> range of B-Tree page statistics. So the goto solution was to use pageinspect. 
> However, reviewing stats for multiple pages meant issuing multiple queries.

+1 to improve the API.

> I felt that there's an opportunity for improvement in the extension by 
> extending the API to output the statistics for multiple pages with a single 
> query.
>
> That attached patch is based on the master branch. It makes the following 
> changes to the pageinspect contrib module:
> - Updates bt_page_stats_internal function to accept 3 arguments instead of 2.
> - The function now uses SRF macros to return a set rather than a single row. 
> The function call now requires specifying column names.
>
> The extension version is bumped to 1.11 (PAGEINSPECT_V1_11).
> To maintain backward compatibility, for versions below 1.11, the multi-call 
> mechanism is ended to keep the old behavior consistent.
>
> Regression test cases for the module are updated as well as part of this 
> change. Here is a subset of queries that are added to the btree.sql test case 
> file for pageinspect.
>
> 
> CREATE TABLE test2 AS (SELECT generate_series(1, 5000) AS col1);
> CREATE INDEX test2_col1_idx ON test2(col1);
> SELECT * FROM bt_page_stats('test2_col1_idx', 1, 2);
>
> For example, this could be written as:
>
> select * from
> generate_series(1, 2) blkno ,
> bt_page_stats('test2_col1_idx',blkno::int);
>
> Or, if one wants to inspect to whole relation, something like:
>
> select * from
> generate_series(1, pg_relation_size('test2_col1_idx'::regclass::text) / 8192 
> - 1) blkno ,
> bt_page_stats('test2_col1_idx',blkno::int);

Good one. But not all may know the alternatives. Do we have any
difference in the execution times for the above query vs the new
function introduced in the v1 patch? If there's not much difference, I
would suggest adding an SQL function around the generate_series
approach in the pageinspect extension for better and easier usability.

Regards,
Bharath Rupireddy.

Allow pageinspect's bt_page_stats function to return a set of rows instead of a single row

[Hamid Akhtar]

Hello Hackers,

While working on one of my blogs on the B-Tree indexes
,
I needed to look at a range of B-Tree page statistics. So the goto solution
was to use pageinspect. However, reviewing stats for multiple pages meant
issuing multiple queries. I felt that there's an opportunity for
improvement in the extension by extending the API to output the statistics
for multiple pages with a single query.

That attached patch is based on the master branch. It makes the following
changes to the pageinspect contrib module:
- Updates bt_page_stats_internal function to accept 3 arguments instead of
2.
- The function now uses SRF macros to return a set rather than a single
row. The function call now requires specifying column names.

The extension version is bumped to 1.11 (PAGEINSPECT_V1_11).
To maintain backward compatibility, for versions below 1.11, the multi-call
mechanism is ended to keep the old behavior consistent.

Regression test cases for the module are updated as well as part of this
change. Here is a subset of queries that are added to the btree.sql test
case file for pageinspect.


CREATE TABLE test2 AS (SELECT generate_series(1, 5000) AS col1);
CREATE INDEX test2_col1_idx ON test2(col1);
SELECT * FROM bt_page_stats('test2_col1_idx', 1, 2);
SELECT * FROM bt_page_stats('test2_col1_idx', 1, 0);
SELECT * FROM bt_page_stats('test2_col1_idx', 0, 1);
SELECT * FROM bt_page_stats('test2_col1_idx', 1, -1);
DROP TABLE test2;


Regards.

--
Hamid Akhtar,
Percona LLC,
URL : www.percona.com
CELL：+923335449950
SKYPE: engineeredvirus
diff --git a/contrib/pageinspect/Makefile b/contrib/pageinspect/Makefile
index 5c0736564a..31d7b1c581 100644
--- a/contrib/pageinspect/Makefile
+++ b/contrib/pageinspect/Makefile
@@ -13,12 +13,12 @@ OBJS = \
 	rawpage.o
 
 EXTENSION = pageinspect
-DATA =  pageinspect--1.9--1.10.sql pageinspect--1.8--1.9.sql \
-	pageinspect--1.7--1.8.sql pageinspect--1.6--1.7.sql \
-	pageinspect--1.5.sql pageinspect--1.5--1.6.sql \
-	pageinspect--1.4--1.5.sql pageinspect--1.3--1.4.sql \
-	pageinspect--1.2--1.3.sql pageinspect--1.1--1.2.sql \
-	pageinspect--1.0--1.1.sql
+DATA =  pageinspect--1.10--1.11.sql pageinspect--1.9--1.10.sql \
+	pageinspect--1.8--1.9.sql pageinspect--1.7--1.8.sql \
+	pageinspect--1.6--1.7.sql pageinspect--1.5.sql \
+	pageinspect--1.5--1.6.sql pageinspect--1.4--1.5.sql \
+	pageinspect--1.3--1.4.sql pageinspect--1.2--1.3.sql \
+	pageinspect--1.1--1.2.sql pageinspect--1.0--1.1.sql
 PGFILEDESC = "pageinspect - functions to inspect contents of database pages"
 
 REGRESS = page btree brin gin gist hash checksum oldextversions
diff --git a/contrib/pageinspect/btreefuncs.c b/contrib/pageinspect/btreefuncs.c
index 62f2c1b315..85127fcfdb 100644
--- a/contrib/pageinspect/btreefuncs.c
+++ b/contrib/pageinspect/btreefuncs.c
@@ -45,6 +45,7 @@ PG_FUNCTION_INFO_V1(bt_page_items_1_9);
 PG_FUNCTION_INFO_V1(bt_page_items);
 PG_FUNCTION_INFO_V1(bt_page_items_bytea);
 PG_FUNCTION_INFO_V1(bt_page_stats_1_9);
+PG_FUNCTION_INFO_V1(bt_page_stats_1_11);
 PG_FUNCTION_INFO_V1(bt_page_stats);
 
 #define IS_INDEX(r) ((r)->rd_rel->relkind == RELKIND_INDEX)
@@ -80,6 +81,26 @@ typedef struct BTPageStat
 	BTCycleId	btpo_cycleid;
 } BTPageStat;
 
+/*
+ * cross-call data structure for SRF for page stats
+ */
+typedef struct ua_page_stats
+{
+	uint64		blkno;
+	uint64		blk_count;
+}			ua_page_stats;
+
+/*
+ * cross-call data structure for SRF for page items
+ */
+typedef struct ua_page_items
+{
+	Page		page;
+	OffsetNumber offset;
+	bool		leafpage;
+	bool		rightmost;
+	TupleDesc	tupd;
+}			ua_page_items;
 
 /* -
  * GetBTPageStatistics()
@@ -176,35 +197,17 @@ GetBTPageStatistics(BlockNumber blkno, Buffer buffer, BTPageStat *stat)
 		stat->avg_item_size = 0;
 }
 
+
 /* ---
- * bt_page_stats()
+ * bt_index_block_validate()
  *
- * Usage: SELECT * FROM bt_page_stats('t1_pkey', 1);
+ * Validate index type is btree and block number
+ * is within a valid block number range.
  * ---
  */
-static Datum
-bt_page_stats_internal(PG_FUNCTION_ARGS, enum pageinspect_version ext_version)
+static void
+bt_index_block_validate(Relation rel, int64 blkno)
 {
-	text	   *relname = PG_GETARG_TEXT_PP(0);
-	int64		blkno = (ext_version == PAGEINSPECT_V1_8 ? PG_GETARG_UINT32(1) : PG_GETARG_INT64(1));
-	Buffer		buffer;
-	Relation	rel;
-	RangeVar   *relrv;
-	Datum		result;
-	HeapTuple	tuple;
-	TupleDesc	tupleDesc;
-	int			j;
-	char	   *values[11];
-	BTPageStat	stat;
-
-	if (!superuser())
-		ereport(ERROR,
-(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
- errmsg("must be superuser to use pageinspect functions")));
-
-	relrv = makeRangeVarFromNameList(textToQualifiedNameList(relname));
-	rel = relation_openrv(relrv, AccessShareLock);
-
 	if (!IS_INDEX(rel) || !IS_BTREE(rel))
 		ereport(ERROR,