[HACKERS] New version of contrib-intarray is ready !
Hi,
New version of contrib-intarray is available from
http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz
From README.intarray:
March 19, 2001
1. Added support for toastable keys
2. Improved split algorithm for intbig (selection speedup is about 30%)
Regards,
Oleg
On Mon, 19 Mar 2001, Oleg Bartunov wrote:
I just returned from vacation and identified the problem.
We'll fix it.
Regards,
Oleg
On Sun, 18 Mar 2001, Tom Lane wrote:
I wrote:
I did this, also reinstalled the include-file changes I had made, and
then spent several fruitless hours trying to find why the "intbig" index
operators fail selftest here (on HP-PA). I suppose it's a portability
problem, since presumably they pass for Oleg ... but I don't see it.
Further experimentation shows that intbig fails selftest on ALL
platforms under 7.1. I see the problem: the intarray operators are
mostly unprepared to cope with TOASTed input arrays. In particular,
_intbig_union() generates an erroneous "null" result for a compressed
input array, leading to completely incorrect GiST index trees in the
self-test example.
A somewhat-related error in this code is that some routines feel free
to scribble on their input. This is tres uncool, because they may be
scribbling on disk buffers. Example:
regression=# create table foo(f1 int4[]);
CREATE
regression=# insert into foo values ('{10,1,2,1,4}');
INSERT 150265 1
regression=# select * from foo;
f1
--
{10,1,2,1,4}
(1 row)
regression=# select * from foo where f1 '{4}';
f1
--
{1,1,2,4,10}
(1 row)
regression=# select * from foo;
f1
--
{1,1,2,4,10}
(1 row)
And you thought SELECT was a read-only operation ...
I do not have time to work on this stuff now, but as it stands the
contrib/intarray code is unusable in 7.1. Unless Oleg can find the
time to fix these issues before release, I will recommend that we
not ship contrib/intarray in 7.1.
regards, tom lane
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/users-lounge/docs/faq.html
Regards,
Oleg
_
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/users-lounge/docs/faq.html
Regards,
Oleg
_
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83
---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] new version of contrib-intarray
Bruce Momjian writes: Seems I am no longer capable of understanding the affects of such changes as this: ! override CPPFLAGS := -I$(srcdir) $(CPPFLAGS) -DPGSQL71 ! override CPPFLAGS += -I$(srcdir) -DPGSQL71 Having $(srcdir) before $(CPPFLAGS) must be significant. The CVS log message says: : Make sure -L and -I's for our source tree are always before system include : or library directories on the command line. That's all this does. -- Peter Eisentraut [EMAIL PROTECTED] http://yi.org/peter-e/ ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] new version of contrib-intarray
Sorry, I have again messed up this Makefile, and I am glad Tom has put things back. Seems I am no longer capable of understanding the affects of such changes as this: ! override CPPFLAGS := -I$(srcdir) $(CPPFLAGS) -DPGSQL71 ! override CPPFLAGS += -I$(srcdir) -DPGSQL71 Having $(srcdir) before $(CPPFLAGS) must be significant. I am going to make a proposal that we partition off certain areas to be handled by certain people so I don't mess things up again. Look for my email in a few minutes. Bruce Momjian writes: I see change of += in CFLAGS (harmless), Not. movement of #include postgres.h, and removal of // comments, which don't appear anymore in the code. I only saw that the Makefile is back to how it looked at rev 1.1 before I did some work on it. AFAICT the Makefile should be reverted back to the previous revision, since the code change does not require any changes to the Makefile. -- Peter Eisentraut [EMAIL PROTECTED] http://yi.org/peter-e/ -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] new version of contrib-intarray
Bruce Momjian writes: Seems I am no longer capable of understanding the affects of such changes as this: ! override CPPFLAGS := -I$(srcdir) $(CPPFLAGS) -DPGSQL71 ! override CPPFLAGS += -I$(srcdir) -DPGSQL71 Having $(srcdir) before $(CPPFLAGS) must be significant. The CVS log message says: : Make sure -L and -I's for our source tree are always before system include : or library directories on the command line. That's all this does. Makes sense. See my new email about patches. I think I need to rely more on you and others to help me evaluate these changes. We have much more expertice in the group than I can hope to comprehend. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] new version of contrib-intarray
Bruce Momjian writes: I see change of += in CFLAGS (harmless), Not. movement of #include postgres.h, and removal of // comments, which don't appear anymore in the code. I only saw that the Makefile is back to how it looked at rev 1.1 before I did some work on it. AFAICT the Makefile should be reverted back to the previous revision, since the code change does not require any changes to the Makefile. -- Peter Eisentraut [EMAIL PROTECTED] http://yi.org/peter-e/ ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] new version of contrib-intarray
Peter Eisentraut [EMAIL PROTECTED] writes: I only saw that the Makefile is back to how it looked at rev 1.1 before I did some work on it. AFAICT the Makefile should be reverted back to the previous revision, since the code change does not require any changes to the Makefile. I did this, also reinstalled the include-file changes I had made, and then spent several fruitless hours trying to find why the "intbig" index operators fail selftest here (on HP-PA). I suppose it's a portability problem, since presumably they pass for Oleg ... but I don't see it. Who else finds that the new contrib/intarray code passes or fails its selftest, and on what platforms? regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] new version of contrib-intarray
I wrote:
I did this, also reinstalled the include-file changes I had made, and
then spent several fruitless hours trying to find why the "intbig" index
operators fail selftest here (on HP-PA). I suppose it's a portability
problem, since presumably they pass for Oleg ... but I don't see it.
Further experimentation shows that intbig fails selftest on ALL
platforms under 7.1. I see the problem: the intarray operators are
mostly unprepared to cope with TOASTed input arrays. In particular,
_intbig_union() generates an erroneous "null" result for a compressed
input array, leading to completely incorrect GiST index trees in the
self-test example.
A somewhat-related error in this code is that some routines feel free
to scribble on their input. This is tres uncool, because they may be
scribbling on disk buffers. Example:
regression=# create table foo(f1 int4[]);
CREATE
regression=# insert into foo values ('{10,1,2,1,4}');
INSERT 150265 1
regression=# select * from foo;
f1
--
{10,1,2,1,4}
(1 row)
regression=# select * from foo where f1 '{4}';
f1
--
{1,1,2,4,10}
(1 row)
regression=# select * from foo;
f1
--
{1,1,2,4,10}
(1 row)
And you thought SELECT was a read-only operation ...
I do not have time to work on this stuff now, but as it stands the
contrib/intarray code is unusable in 7.1. Unless Oleg can find the
time to fix these issues before release, I will recommend that we
not ship contrib/intarray in 7.1.
regards, tom lane
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] new version of contrib-intarray
I just returned from vacation and identified the problem.
We'll fix it.
Regards,
Oleg
On Sun, 18 Mar 2001, Tom Lane wrote:
I wrote:
I did this, also reinstalled the include-file changes I had made, and
then spent several fruitless hours trying to find why the "intbig" index
operators fail selftest here (on HP-PA). I suppose it's a portability
problem, since presumably they pass for Oleg ... but I don't see it.
Further experimentation shows that intbig fails selftest on ALL
platforms under 7.1. I see the problem: the intarray operators are
mostly unprepared to cope with TOASTed input arrays. In particular,
_intbig_union() generates an erroneous "null" result for a compressed
input array, leading to completely incorrect GiST index trees in the
self-test example.
A somewhat-related error in this code is that some routines feel free
to scribble on their input. This is tres uncool, because they may be
scribbling on disk buffers. Example:
regression=# create table foo(f1 int4[]);
CREATE
regression=# insert into foo values ('{10,1,2,1,4}');
INSERT 150265 1
regression=# select * from foo;
f1
--
{10,1,2,1,4}
(1 row)
regression=# select * from foo where f1 '{4}';
f1
--
{1,1,2,4,10}
(1 row)
regression=# select * from foo;
f1
--
{1,1,2,4,10}
(1 row)
And you thought SELECT was a read-only operation ...
I do not have time to work on this stuff now, but as it stands the
contrib/intarray code is unusable in 7.1. Unless Oleg can find the
time to fix these issues before release, I will recommend that we
not ship contrib/intarray in 7.1.
regards, tom lane
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/users-lounge/docs/faq.html
Regards,
Oleg
_
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] new version of contrib-intarray
Seems we have an older version in CVS. I will update it now. I assume /contrib is available for changes up until release, as usual. Mark, we prepared new version of contrib-intarray - index support for 1-D integer arrays using GiST. Changes: - Improved regression test - Current implementation provides index support for one-dimensional array of int4's - gist__int_ops, suitable for small and medium size of arrays, and gist__intbig_ops for indexing large arrays (we use superimposed signature with length of 4096 bits to represent sets, see Sven Helmer,1997). Archive is available from http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz Regards, Oleg _ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] new version of contrib-intarray
Installed in CVS. Thanks. Mark, we prepared new version of contrib-intarray - index support for 1-D integer arrays using GiST. Changes: - Improved regression test - Current implementation provides index support for one-dimensional array of int4's - gist__int_ops, suitable for small and medium size of arrays, and gist__intbig_ops for indexing large arrays (we use superimposed signature with length of 4096 bits to represent sets, see Sven Helmer,1997). Archive is available from http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz Regards, Oleg _ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] new version of contrib-intarray
Bruce Momjian writes: Installed in CVS. Thanks. You overwrote the changes that other people have made meanwhile. Mark, we prepared new version of contrib-intarray - index support for 1-D integer arrays using GiST. Changes: - Improved regression test - Current implementation provides index support for one-dimensional array of int4's - gist__int_ops, suitable for small and medium size of arrays, and gist__intbig_ops for indexing large arrays (we use superimposed signature with length of 4096 bits to represent sets, see Sven Helmer,1997). Archive is available from http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz Regards, Oleg _ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 -- Peter Eisentraut [EMAIL PROTECTED] http://yi.org/peter-e/ ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] new version of contrib-intarray
I checked README.intarray to see what the most recent date was, and it
was Jan 10, so I knew that this version was newer. I then did a diff -c
against the current CVS and I didn't see anything unusual in the
changes.
Attached is the CVS diff command line showing me all the changes made:
cvs diff -c -D '2001-01-13 00:00:00' -D'2001-03-16 00:00:00' .
I see change of += in CFLAGS (harmless), movement of #include
postgres.h, and removal of // comments, which don't appear anymore in
the code.
Do you see anything else? This one was easy to track because it was
installed only recently, but other /contrib stuff is much tougher
because you never know what the original install date was.
I usually only look at Makefile changes in /contrib because that is
where most of the customization is, and I don't see any changes made to
Makefile by the patch. It doesn't even touch the CFLAGS += change.
--
Bruce Momjian| http://candle.pha.pa.us
[EMAIL PROTECTED] | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup.| Drexel Hill, Pennsylvania 19026
? Makefile.703
Index: Makefile
===
RCS file: /home/projects/pgsql/cvsroot/pgsql/contrib/intarray/Makefile,v
retrieving revision 1.2
retrieving revision 1.3
diff -c -r1.2 -r1.3
*** Makefile2001/01/13 02:18:31 1.2
--- Makefile2001/02/20 19:20:27 1.3
***
*** 1,4
! # $Header: /home/projects/pgsql/cvsroot/pgsql/contrib/intarray/Makefile,v 1.2
2001/01/13 02:18:31 petere Exp $
subdir = contrib/intarray
top_builddir = ../..
--- 1,4
! # $Header: /home/projects/pgsql/cvsroot/pgsql/contrib/intarray/Makefile,v 1.3
2001/02/20 19:20:27 petere Exp $
subdir = contrib/intarray
top_builddir = ../..
***
*** 12,18
SO_MAJOR_VERSION= 1
SO_MINOR_VERSION= 0
! override CPPFLAGS += -I$(srcdir) -DPGSQL71
OBJS= _int.o
--- 12,18
SO_MAJOR_VERSION= 1
SO_MINOR_VERSION= 0
! override CPPFLAGS := -I$(srcdir) $(CPPFLAGS) -DPGSQL71
OBJS= _int.o
Index: _int.c
===
RCS file: /home/projects/pgsql/cvsroot/pgsql/contrib/intarray/_int.c,v
retrieving revision 1.1
retrieving revision 1.3
diff -c -r1.1 -r1.3
*** _int.c 2001/01/12 00:16:23 1.1
--- _int.c 2001/02/12 18:30:52 1.3
***
*** 4,14
format for these routines is dictated by Postgres architecture.
**/
! #include stdio.h
#include float.h
#include string.h
- #include "postgres.h"
#include "access/gist.h"
#include "access/itup.h"
#include "access/rtree.h"
--- 4,14
format for these routines is dictated by Postgres architecture.
**/
! #include "postgres.h"
!
#include float.h
#include string.h
#include "access/gist.h"
#include "access/itup.h"
#include "access/rtree.h"
***
*** 194,200
#ifdef GIST_DEBUG
elog(NOTICE, "COMP IN: %d leaf; %d rel; %d page; %d offset; %d bytes; %d elems",
entry-leafkey, (int)entry-rel, (int)entry-page, (int)entry-offset,
(int)entry-bytes, len);
! //printarr( r, len );
#endif
if ( len = 2*MAXNUMRANGE ) { /*compress*/
--- 194,200
#ifdef GIST_DEBUG
elog(NOTICE, "COMP IN: %d leaf; %d rel; %d page; %d offset; %d bytes; %d elems",
entry-leafkey, (int)entry-rel, (int)entry-page, (int)entry-offset,
(int)entry-bytes, len);
! /* printarr( r, len ); */
#endif
if ( len = 2*MAXNUMRANGE ) { /*compress*/
***
*** 260,266
#ifdef GIST_DEBUG
elog(NOTICE, "DECOMP IN: %d leaf; %d rel; %d page; %d offset; %d bytes; %d
elems", entry-leafkey, (int)entry-rel, (int)entry-page, (int)entry-offset,
(int)entry-bytes, lenin);
! //printarr( in, lenin );
#endif
lenr = internal_size(din, lenin);
--- 260,266
#ifdef GIST_DEBUG
elog(NOTICE, "DECOMP IN: %d leaf; %d rel; %d page; %d offset; %d bytes; %d
elems", entry-leafkey, (int)entry-rel, (int)entry-page, (int)entry-offset,
(int)entry-bytes, lenin);
! /* printarr( in, lenin ); */
#endif
lenr = internal_size(din, lenin);
***
*** 653,659
int i,j;
#ifdef GIST_DEBUG
! //elog(NOTICE, "inner_union %d %d", ARRISNULL( a ) , ARRISNULL( b ) );
#endif
if ( ARRISNULL( a ) ARRISNULL( b ) ) return new_intArrayType(0);
--- 653,659
int i,j;
#ifdef GIST_DEBUG
! /* elog(NOTICE, "inner_union %d %d", ARRISNULL( a ) , ARRISNULL( b ) ); */
#endif
if ( ARRISNULL( a ) ARRISNULL( b ) ) return new_intArrayType(0);
***
*** 709,715
int i,j;
#ifdef GIST_DEBUG
! //elog(NOTICE, "inner_inter %d %d", ARRISNULL( a ),
Re: [HACKERS] new version of contrib-intarray
Oleg Bartunov [EMAIL PROTECTED] writes: gist__int_ops| 1007 gist__intbig_ops | 1007 we want gist__int_ops to be default index opclass. If we delete gist__intbig_ops entry from opclass, then we couldn't use gist__intbig_ops ! Put in gist__intbig_ops with zero for the default type. You should never have more than one entry in pg_opclass claiming to be the default for a given type OID. regards, tom lane
Re: [HACKERS] new version of contrib-intarray
Oleg Bartunov [EMAIL PROTECTED] writes: btw, is there way to specify default ops for index ? Sure, that's what pg_opclass is for. Just insert the opclass name and the OID of the type you want it to be the default index opclass for. regards, tom lane
Re: [HACKERS] new version of contrib-intarray
Oleg, do you want this in /contrib for 7.1? Mark, we prepared new version of contrib-intarray - index support for 1-D integer arrays using GiST. Changes: - Improved regression test - Current implementation provides index support for one-dimensional array of int4's - gist__int_ops, suitable for small and medium size of arrays, and gist__intbig_ops for indexing large arrays (we use superimposed signature with length of 4096 bits to represent sets, see Sven Helmer,1997). Archive is available from http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz Regards, Oleg _ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026
Re: [HACKERS] new version of contrib-intarray
On Sat, 27 Jan 2001, Bruce Momjian wrote: Oleg, do you want this in /contrib for 7.1? yes, if it's possible. btw, is there way to specify default ops for index ? We have two methods of index creation for intarrays and would like to define which should be used by default Mark, we prepared new version of contrib-intarray - index support for 1-D integer arrays using GiST. Changes: - Improved regression test - Current implementation provides index support for one-dimensional array of int4's - gist__int_ops, suitable for small and medium size of arrays, and gist__intbig_ops for indexing large arrays (we use superimposed signature with length of 4096 bits to represent sets, see Sven Helmer,1997). Archive is available from http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz Regards, Oleg _ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 Regards, Oleg _ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83
[HACKERS] new version of contrib-intarray
Mark, we prepared new version of contrib-intarray - index support for 1-D integer arrays using GiST. Changes: - Improved regression test - Current implementation provides index support for one-dimensional array of int4's - gist__int_ops, suitable for small and medium size of arrays, and gist__intbig_ops for indexing large arrays (we use superimposed signature with length of 4096 bits to represent sets, see Sven Helmer,1997). Archive is available from http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz Regards, Oleg _ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83
