Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread David E. Wheeler
On Jan 18, 2011, at 3:46 PM, Tom Lane wrote: > The above is "using" the index, but only as a guide to where the rows > satisfying the partial-index predicate are --- note the lack of any > index condition in the indexscan node. That's because the query_int > query is not in fact compatible with t

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread Tom Lane
I wrote: > No, I see no reason to think that has much to do with it. I'm wondering > if your table is itself a bit bloated ... Actually ... I notice you did not show EXPLAIN ANALYZE output for your tests. Now I'm wondering whether you tested the right thing at all. I got burnt that way too. Obs

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread David E. Wheeler
On Jan 18, 2011, at 3:08 PM, Tom Lane wrote: >> Shall I send you data with the other two columns?: > > No, I see no reason to think that has much to do with it. I'm wondering > if your table is itself a bit bloated ... Nope. Just ran the bloat query from check_postgres.pl. Bloat is 0. Not surp

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread Tom Lane
"David E. Wheeler" writes: > On Jan 18, 2011, at 1:58 PM, Tom Lane wrote: >> I'm noticing also that I get different rowcounts than you do, although >> possibly that has something to do with the partial-index conditions, >> which I'm not trying to duplicate here (all rows in my table pass those >>

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread David E. Wheeler
On Jan 18, 2011, at 1:58 PM, Tom Lane wrote: > I'm noticing also that I get different rowcounts than you do, although > possibly that has something to do with the partial-index conditions, > which I'm not trying to duplicate here (all rows in my table pass those > two tests). Shall I send you dat

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread Tom Lane
"David E. Wheeler" writes: > On Jan 18, 2011, at 1:39 PM, Tom Lane wrote: >> At the moment my opinion is that gist__int_ops is too broken to be >> usable, and it's also too uncommented to be fixable by anyone other >> than the original author. > That seems to jibe with your comments from last yea

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread David E. Wheeler
On Jan 18, 2011, at 1:39 PM, Tom Lane wrote: > At the moment my opinion is that gist__int_ops is too broken to be > usable, and it's also too uncommented to be fixable by anyone other > than the original author. That seems to jibe with your comments from last year: http://archives.postgresql.o

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread Tom Lane
"David E. Wheeler" writes: > These numbers are a bit crazy-making, but the upshot is that Gist is > slow out of the gate, but with data cached, it's pretty speedy. With > indexscan and bitmapscan disabled, these queries all took 300-400 > ms. So GIN was never better performing than a table scan.

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread Tom Lane
"David E. Wheeler" writes: > One of the reasons our client wants GIN for the integer[] column so bad is > because recreating the GiST integer[] index is quite painful. Before I duped > the table, I was just dropping and recreating the index on the original > table. It was great to create the GI

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-16 Thread David E. Wheeler
On Jan 14, 2011, at 11:37 PM, David E. Wheeler wrote: >> Hard to comment on any of this without a concrete example (including >> data) to look at. Given the bugs we've recently found in the picksplit >> algorithms for other contrib modules, I wouldn't be too surprised if the >> sucky GiST perform

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-14 Thread David E. Wheeler
On Jan 14, 2011, at 5:54 PM, Tom Lane wrote: > "David E. Wheeler" writes: >> So some questions: > >> * Is something seriously wrong with GiST index creation on integer[] columns? > >> * Why does GIN performance appear to be no better than table scans on >> integer[] columns? > >> * Why does i

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-14 Thread Tom Lane
"David E. Wheeler" writes: > So some questions: > * Is something seriously wrong with GiST index creation on integer[] columns? > * Why does GIN performance appear to be no better than table scans on > integer[] columns? > * Why does it take 3-4x longer to create the GIN than the GiST index on

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-14 Thread David E . Wheeler
On Jan 8, 2011, at 9:41 PM, Tom Lane wrote: > "David E. Wheeler" writes: >> On Jan 7, 2011, at 4:19 PM, Tom Lane wrote: >>> Well, actually, I just committed it. If you want to test, feel free. >>> Note that right now only the anyarray && <@ @> operators are genuinely >>> fixed ... I plan to hack

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-13 Thread Tom Lane
Bruce Momjian writes: > What does pg_upgrade need to do about this for 9.1? Nothing. An existing GIN index can still do all the same queries it could do before. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to y

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-13 Thread Bruce Momjian
Robert Haas wrote: > On Tue, Jan 4, 2011 at 4:49 PM, Tom Lane wrote: > > Robert Haas writes: > >> On Tue, Jan 4, 2011 at 4:09 PM, Tom Lane wrote: > >>> * Existing GIN indexes are upwards compatible so far as on-disk storage > >>> goes, but they will of course be missing entries for empty, null,

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-13 Thread Dimitri Fontaine
Tom Lane writes: > "David E. Wheeler" writes: >> On Jan 12, 2011, at 4:35 PM, Tom Lane wrote: >>> No, what we need is a decent extension package manager ;-) > >> Yeah. Maybe you can do that this weekend? Or, I dunno, while you “sleep” >> tonight? > > Supposedly it's in the queue for the upcoming

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-12 Thread Tom Lane
"David E. Wheeler" writes: > On Jan 12, 2011, at 4:35 PM, Tom Lane wrote: >> No, what we need is a decent extension package manager ;-) > Yeah. Maybe you can do that this weekend? Or, I dunno, while you “sleep” > tonight? Supposedly it's in the queue for the upcoming CF :-)

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-12 Thread David E. Wheeler
On Jan 12, 2011, at 4:35 PM, Tom Lane wrote: >> Did a signature change or something? > > Yeah. I think if you just load up the current contrib/intarray first, > you'll be fine (ignore all the object-already-exists errors). Oh, from 9.1devel? yeah, okay. Will do that tomorrow (finishing the curr

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-12 Thread Tom Lane
"David E. Wheeler" writes: > Thanks, working on it now. I'm restoring a dump from 8.4, but got these erors: > pg_restore: [archiver (db)] Error from TOC entry 3227; 2616 46485 OPERATOR > CLASS gin__int_ops postgres > pg_restore: [archiver (db)] could not execute query: ERROR: function > ginarr

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-12 Thread David E. Wheeler
Tom, On Jan 8, 2011, at 9:41 PM, Tom Lane wrote: > "David E. Wheeler" writes: >> On Jan 7, 2011, at 4:19 PM, Tom Lane wrote: >>> Well, actually, I just committed it. If you want to test, feel free. >>> Note that right now only the anyarray && <@ @> operators are genuinely >>> fixed ... I plan t

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-08 Thread Tom Lane
"David E. Wheeler" writes: > On Jan 7, 2011, at 4:19 PM, Tom Lane wrote: >> Well, actually, I just committed it. If you want to test, feel free. >> Note that right now only the anyarray && <@ @> operators are genuinely >> fixed ... I plan to hack on tsearch and contrib pretty soon though. > Hrm,

Re: contrib/intarray (was Re: [HACKERS] Fixing GIN for empty/null/full-scan cases)

2011-01-08 Thread Tom Lane
"David E. Wheeler" writes: > On Jan 8, 2011, at 1:59 PM, Tom Lane wrote: >> There seem to be three ways >> in which intarray is simpler/faster than the generic operators: >> >> * restricted to integer arrays >> * restricted to 1-D arrays >> * doesn't allow nulls in the arrays > My understanding

Re: contrib/intarray (was Re: [HACKERS] Fixing GIN for empty/null/full-scan cases)

2011-01-08 Thread David E. Wheeler
On Jan 8, 2011, at 1:59 PM, Tom Lane wrote: >> Hrm, the queries I wrote for this sort of thing use intarray: > > I'm going to work on contrib/intarray first (before tsearch etc) > so that you can do whatever testing you want sooner. No, of course not. > One of the things that first got me annoy

contrib/intarray (was Re: [HACKERS] Fixing GIN for empty/null/full-scan cases)

2011-01-08 Thread Tom Lane
"David E. Wheeler" writes: > On Jan 7, 2011, at 4:19 PM, Tom Lane wrote: >> Well, actually, I just committed it. If you want to test, feel free. >> Note that right now only the anyarray && <@ @> operators are genuinely >> fixed ... I plan to hack on tsearch and contrib pretty soon though. > Hrm,

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-07 Thread Tom Lane
"David E. Wheeler" writes: > On Jan 7, 2011, at 4:19 PM, Tom Lane wrote: >> Well, actually, I just committed it. If you want to test, feel free. >> Note that right now only the anyarray && <@ @> operators are genuinely >> fixed ... I plan to hack on tsearch and contrib pretty soon though. > Hrm,

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-07 Thread David E. Wheeler
On Jan 7, 2011, at 4:19 PM, Tom Lane wrote: > Well, actually, I just committed it. If you want to test, feel free. > Note that right now only the anyarray && <@ @> operators are genuinely > fixed ... I plan to hack on tsearch and contrib pretty soon though. Hrm, the queries I wrote for this sort

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-07 Thread Tom Lane
"David E. Wheeler" writes: > We (PGX) actually have a client who could use this. Tom, if you have patches > as you work on this (or, better, a branch in a Git repo), I could do some > testing on your client's code with it. It would involve converting from a > GiST to a GIN index and then seeing

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-07 Thread David E . Wheeler
On Jan 4, 2011, at 3:18 PM, Josh Berkus wrote: > Actually, there's been a *lot* of complaining about the GIN issues. > It's just that most of that complaining doesn't reach -hackers. > > The common pattern I've seen in our practice and on IRC is: > > 1) user has GiST indexes > 2) user tries conv

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-07 Thread Tom Lane
I wrote: > 2. Add another output bool parameter to extractQuery that it must set > true (from a default false state) if the query could match with no check > values set. This would prompt the GIN code to search for EMPTY_ITEM > placeholders, but they'd not be part of the check[] array. On further

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-06 Thread Euler Taveira de Oliveira
Em 06-01-2011 21:31, Tom Lane escreveu: I think I like option #2 better. Comments? +1. -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsq

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-04 Thread Josh Berkus
On 1/4/11 1:49 PM, Tom Lane wrote: > I don't think it's really worth the trouble. The GIN code has been > broken for these types of queries since day one, and yet we've had only > maybe half a dozen complaints about it. Moreover there's no practical > way to "avoid trying to use the index", since

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-04 Thread Robert Haas
On Tue, Jan 4, 2011 at 4:49 PM, Tom Lane wrote: > Robert Haas writes: >> On Tue, Jan 4, 2011 at 4:09 PM, Tom Lane wrote: >>> * Existing GIN indexes are upwards compatible so far as on-disk storage >>> goes, but they will of course be missing entries for empty, null, or >>> null-containing items.

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-04 Thread Tom Lane
Robert Haas writes: > On Tue, Jan 4, 2011 at 4:09 PM, Tom Lane wrote: >> * Existing GIN indexes are upwards compatible so far as on-disk storage >> goes, but they will of course be missing entries for empty, null, or >> null-containing items.  Users who want to do searches that should find >> suc

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-04 Thread Robert Haas
On Tue, Jan 4, 2011 at 4:09 PM, Tom Lane wrote: > * Existing GIN indexes are upwards compatible so far as on-disk storage > goes, but they will of course be missing entries for empty, null, or > null-containing items.  Users who want to do searches that should find > such items will need to reinde

[HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-04 Thread Tom Lane
I've been thinking about how to fix GIN's assorted corner-case problems, as has been discussed several times, most recently here: http://archives.postgresql.org/pgsql-hackers/2010-10/msg00521.php See also http://wiki.postgresql.org/wiki/Todo#GIN There are basically three related issues: 1. GIN do