Re: [HACKERS] GIST and TOAST

2007-03-07 Thread Teodor Sigaev
I'm already started, don't worry about that. Cube is broken since TOAST implemented :) Gregory Stark wrote: "Teodor Sigaev" <[EMAIL PROTECTED]> writes: input value. As I remember, only R-Tree emulation over boxes, contrib/seg and contrib/cube have simple compress method. Hm, if they just ret

Re: [HACKERS] GIST and TOAST

2007-03-07 Thread Gregory Stark
"Teodor Sigaev" <[EMAIL PROTECTED]> writes: >>> input value. As I remember, only R-Tree emulation over boxes, contrib/seg >>> and >>> contrib/cube have simple compress method. >> Hm, if they just return the original datum without detoasting it then it >> could >> be an issue. I'll check. > seg a

Re: [HACKERS] GIST and TOAST

2007-03-06 Thread Gregory Stark
"Teodor Sigaev" <[EMAIL PROTECTED]> writes: >> It doesn't actually crash, it just fails CHECKARRVALID. I added an assertion >> in there to cause it to generate a core dump. > > Wow, catch that, see attached patch. > > g_int_decompress doesn't returns detoasted array in case it was empty. > Previou

Re: [HACKERS] GIST and TOAST

2007-03-06 Thread Teodor Sigaev
It doesn't actually crash, it just fails CHECKARRVALID. I added an assertion in there to cause it to generate a core dump. Wow, catch that, see attached patch. g_int_decompress doesn't returns detoasted array in case it was empty. Previously it was safe because empty array never has been toast

Re: [HACKERS] GIST and TOAST

2007-03-06 Thread Gregory Stark
"Teodor Sigaev" <[EMAIL PROTECTED]> writes: >> And it's not clear _int_gist.c has been running with toasted array values for >> years because it's limited to arrays of 100 integers (or perhaps 200 >> integers, >> there's a factor of 2 in the test). That's not enough to trigger toasting >> unless

Re: [HACKERS] GIST and TOAST

2007-03-06 Thread Teodor Sigaev
input value. As I remember, only R-Tree emulation over boxes, contrib/seg and contrib/cube have simple compress method. Hm, if they just return the original datum without detoasting it then it could be an issue. I'll check. seg and box aren't a varlena types, but cube is and it seems broken :(.

Re: [HACKERS] GIST and TOAST

2007-03-06 Thread Gregory Stark
>> Does every data type define a compress/decompress method? Even if it's not a >> data type that normally gets very large? > > Yes, any GiST opclass should have such methods. In trivial case it just > returns > input value. As I remember, only R-Tree emulation over boxes, contrib/seg and > cont

Re: [HACKERS] GIST and TOAST

2007-03-06 Thread Teodor Sigaev
And it's not clear _int_gist.c has been running with toasted array values for years because it's limited to arrays of 100 integers (or perhaps 200 integers, there's a factor of 2 in the test). That's not enough to trigger toasting unless there are other large columns in the same table. That's was

Re: [HACKERS] GIST and TOAST

2007-03-06 Thread Teodor Sigaev
So when does index_form_tuple get called? The single call of index_form_tuple in GiST core is in gistFormTuple which initially compress any indexed value with a help of their compress methods. Only tuples formed by gistFormTuple could be inserted in index. Does every data type define a comp

Re: [HACKERS] GIST and TOAST

2007-03-06 Thread Gregory Stark
"Gregory Stark" <[EMAIL PROTECTED]> writes: > "Andrew - Supernews" <[EMAIL PROTECTED]> writes: > >> So I think you've mis-analyzed the problem. That's especially true since >> you are claiming that the existing code is already buggy when in fact no >> such bugs have been reported (and clearly int

Re: [HACKERS] GIST and TOAST

2007-03-06 Thread Gregory Stark
"Teodor Sigaev" <[EMAIL PROTECTED]> writes: > I'm afraid that we have some lack of understanding. Flow of operation with > indexed tuple in gist is: > - read tuple > - get n-th attribute with a help of index_getattr > - call user-defined decompress method which should, at least, detoast value >

Re: [HACKERS] GIST and TOAST

2007-03-06 Thread Gregory Stark
"Andrew - Supernews" <[EMAIL PROTECTED]> writes: > The places in the intarray code that you tried to "fix" in your patch at > the start of this thread are not dealing with data that came from a tuple, > but from data that came from a decompress method. It's expected that the > decompress method do

Re: [HACKERS] GIST and TOAST

2007-03-06 Thread Teodor Sigaev
The problem is that this is the only place in the code where we make wholesale assumptions that a datum that comes from a tuple (heap tuple or index tuple) isn't toasted. There are other places but they're always flagged with big comments explaining *why* the datum can't be toasted and they're m

Re: [HACKERS] GIST and TOAST

2007-03-06 Thread Andrew - Supernews
On 2007-03-06, Gregory Stark <[EMAIL PROTECTED]> wrote: > "Teodor Sigaev" <[EMAIL PROTECTED]> writes: >>> A closer reading, however, shows that at least for cases like intarray, >>> btree_gist, etc., the detoasting of an index value is being done in the >>> gist decompress function, so the value se

Re: [HACKERS] GIST and TOAST

2007-03-06 Thread Gregory Stark
"Teodor Sigaev" <[EMAIL PROTECTED]> writes: >> A closer reading, however, shows that at least for cases like intarray, >> btree_gist, etc., the detoasting of an index value is being done in the >> gist decompress function, so the value seen via GISTENTRY in the other >> functions should already ha

Re: [HACKERS] GIST and TOAST

2007-03-05 Thread Teodor Sigaev
A closer reading, however, shows that at least for cases like intarray, btree_gist, etc., the detoasting of an index value is being done in the gist decompress function, so the value seen via GISTENTRY in the other functions should already have been detoasted once. Right, any stored value form i

Re: [HACKERS] GIST and TOAST

2007-03-02 Thread Andrew - Supernews
On 2007-03-02, Tom Lane <[EMAIL PROTECTED]> wrote: > Andrew - Supernews <[EMAIL PROTECTED]> writes: >> On 2007-03-02, Gregory Stark <[EMAIL PROTECTED]> wrote: >>> I think these are actual bugs. If you happened to provide a large enough >>> datum >>> to the gist code it would cause the same problem

Re: [HACKERS] GIST and TOAST

2007-03-02 Thread Tom Lane
Andrew - Supernews <[EMAIL PROTECTED]> writes: > On 2007-03-02, Gregory Stark <[EMAIL PROTECTED]> wrote: >> I think these are actual bugs. If you happened to provide a large enough >> datum >> to the gist code it would cause the same problem I'm seeing. The packed >> varlena patch just makes it ea

Re: [HACKERS] GIST and TOAST

2007-03-02 Thread Andrew - Supernews
On 2007-03-02, Gregory Stark <[EMAIL PROTECTED]> wrote: > I think these are actual bugs. If you happened to provide a large enough datum > to the gist code it would cause the same problem I'm seeing. The packed > varlena patch just makes it easier to trigger. Are you taking into account the fact t

[HACKERS] GIST and TOAST

2007-03-02 Thread Gregory Stark
I have a problem getting packed varlenas to work with GIST indexes. Namely, the GIST code doesn't call pg_detoast_datum() enough. Instead of using the DatumGetFoo() macros it uses DatumGetPointer() and calls PG_DETOAST_DATUM only when it thinks it'll be necessary. I've temporarily made the packed