Re: [HACKERS] Unusable SP-GiST index

[Tom Lane]

I wrote:
> After looking a bit at gist and sp-gist, neither of them would find that
> terribly convenient; they really want to create one blob of memory per
> index entry so as to not complicate storage management too much.  But
> they'd be fine with making that blob be a HeapTuple not IndexTuple.
> So maybe the right approach is to expand the existing API to allow the
> AM to return *either* a heap or index tuple; that could be made to not
> be an API break.

Here's a draft patch along those lines.  With this approach, btree doesn't
need to be touched at all, since what it's returning certainly is an
IndexTuple anyway.  I fixed both SPGIST and GIST to use HeapTuple return
format.  It's not very clear to me whether GIST has a similar hazard with
very large return values, but it might, and it's simple enough to change.

regards, tom lane

diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 40f201b..d4af010 100644
*** a/doc/src/sgml/indexam.sgml
--- b/doc/src/sgml/indexam.sgml
*** amgettuple (IndexScanDesc scan,
*** 535,549 

 If the index supports index-only
 scans (i.e., amcanreturn returns TRUE for it),
!then on success the AM must also check
!scan-xs_want_itup, and if that is true it must return
!the original indexed data for the index entry, in the form of an
 IndexTuple pointer stored at scan-xs_itup,
!with tuple descriptor scan-xs_itupdesc.
!(Management of the data referenced by the pointer is the access method's
 responsibility.  The data must remain good at least until the next
 amgettuple, amrescan, or amendscan
!call for the scan.)

  

--- 535,553 

 If the index supports index-only
 scans (i.e., amcanreturn returns TRUE for it),
!then on success the AM must also check scan-xs_want_itup,
!and if that is true it must return the originally indexed data for the
!index entry.  The data can be returned in the form of an
 IndexTuple pointer stored at scan-xs_itup,
!with tuple descriptor scan-xs_itupdesc; or in the form of
!a HeapTuple pointer stored at scan-xs_hitup,
!with tuple descriptor scan-xs_hitupdesc.  (The latter
!format should be used when reconstructing data that might possibly not fit
!into an IndexTuple.)  In either case,
!management of the data referenced by the pointer is the access method's
 responsibility.  The data must remain good at least until the next
 amgettuple, amrescan, or amendscan
!call for the scan.

  

diff --git a/src/backend/access/gist/gistget.c b/src/backend/access/gist/gistget.c
index eea366b..122dc38 100644
*** a/src/backend/access/gist/gistget.c
--- b/src/backend/access/gist/gistget.c
*** gistScanPage(IndexScanDesc scan, GISTSea
*** 441,452 
  			so->pageData[so->nPageData].offnum = i;
  
  			/*
! 			 * In an index-only scan, also fetch the data from the tuple.
  			 */
  			if (scan->xs_want_itup)
  			{
  oldcxt = MemoryContextSwitchTo(so->pageDataCxt);
! so->pageData[so->nPageData].ftup =
  	gistFetchTuple(giststate, r, it);
  MemoryContextSwitchTo(oldcxt);
  			}
--- 441,453 
  			so->pageData[so->nPageData].offnum = i;
  
  			/*
! 			 * In an index-only scan, also fetch the data from the tuple.  The
! 			 * reconstructed tuples are stored in pageDataCxt.
  			 */
  			if (scan->xs_want_itup)
  			{
  oldcxt = MemoryContextSwitchTo(so->pageDataCxt);
! so->pageData[so->nPageData].recontup =
  	gistFetchTuple(giststate, r, it);
  MemoryContextSwitchTo(oldcxt);
  			}
*** gistScanPage(IndexScanDesc scan, GISTSea
*** 478,484 
   * In an index-only scan, also fetch the data from the tuple.
   */
  if (scan->xs_want_itup)
! 	item->data.heap.ftup = gistFetchTuple(giststate, r, it);
  			}
  			else
  			{
--- 479,485 
   * In an index-only scan, also fetch the data from the tuple.
   */
  if (scan->xs_want_itup)
! 	item->data.heap.recontup = gistFetchTuple(giststate, r, it);
  			}
  			else
  			{
*** getNextNearest(IndexScanDesc scan)
*** 540,550 
  	bool		res = false;
  	int			i;
  
! 	if (scan->xs_itup)
  	{
  		/* free previously returned tuple */
! 		pfree(scan->xs_itup);
! 		scan->xs_itup = NULL;
  	}
  
  	do
--- 541,551 
  	bool		res = false;
  	int			i;
  
! 	if (scan->xs_hitup)
  	{
  		/* free previously returned tuple */
! 		pfree(scan->xs_hitup);
! 		scan->xs_hitup = NULL;
  	}
  
  	do
*** getNextNearest(IndexScanDesc scan)
*** 601,607 
  
  			/* in an index-only scan, also return the reconstructed tuple. */
  			if (scan->xs_want_itup)
! scan->xs_itup = item->data.heap.ftup;
  			res = true;
  		}
  		else
--- 602,608 
  
  			/* in an index-only scan, also return the reconstructed tuple. */
  			if (scan->xs_want_itup)
! scan->xs_hitup = item->data.heap.recontup;
  			res = true;
  		}
  		else
*** 

Re: [HACKERS] Unusable SP-GiST index

[Tom Lane]

I wrote:
> Maybe we should redefine the API as involving a TupleTableSlot that
> the AM is supposed to fill --- basically, moving StoreIndexTuple
> out of the common code in nodeIndexonlyscan.c and requiring the AM
> to do that work.  The possible breakage of third-party code is a
> bit annoying, but there can't be all that many third-party AMs
> out there yet.

After looking a bit at gist and sp-gist, neither of them would find that
terribly convenient; they really want to create one blob of memory per
index entry so as to not complicate storage management too much.  But
they'd be fine with making that blob be a HeapTuple not IndexTuple.
So maybe the right approach is to expand the existing API to allow the
AM to return *either* a heap or index tuple; that could be made to not
be an API break.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Unusable SP-GiST index

[Tom Lane]

Vik Fearing  writes:
> While trying to find a case where spgist wins over btree for text, I
> came across the following behavior which I would consider a bug:

> CREATE TABLE texts (value text);
> INSERT INTO texts SELECT repeat('a', (2^20)::integer);
> CREATE INDEX ON texts USING spgist (value);
> SET enable_seqscan = off;
> TABLE texts;

> That produces:
> ERROR:  index row requires 12024 bytes, maximum size is 8191

Hmm ... it's not really SP-GiST's fault.  This query is trying to do
an index-only scan, and the API defined for that requires the index
to hand back an IndexTuple, which is of (very) limited size.
SP-GiST is capable of dealing with values much larger than one page,
but there's no way to hand them back through that API.

Maybe we should redefine the API as involving a TupleTableSlot that
the AM is supposed to fill --- basically, moving StoreIndexTuple
out of the common code in nodeIndexonlyscan.c and requiring the AM
to do that work.  The possible breakage of third-party code is a
bit annoying, but there can't be all that many third-party AMs
out there yet.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Unusable SP-GiST index

[Vik Fearing]

While trying to find a case where spgist wins over btree for text, I
came across the following behavior which I would consider a bug:

CREATE TABLE texts (value text);
INSERT INTO texts SELECT repeat('a', (2^20)::integer);
CREATE INDEX ON texts USING spgist (value);
SET enable_seqscan = off;
TABLE texts;

That produces:

ERROR:  index row requires 12024 bytes, maximum size is 8191

It seems to me the index should not be allowed to be created if it won't
be usable.
-- 
Vik Fearing  +33 6 46 75 15 36
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers