Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread Tom Lane
David E. Wheeler da...@kineticode.com writes: One of the reasons our client wants GIN for the integer[] column so bad is because recreating the GiST integer[] index is quite painful. Before I duped the table, I was just dropping and recreating the index on the original table. It was great

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread Tom Lane
David E. Wheeler da...@kineticode.com writes: These numbers are a bit crazy-making, but the upshot is that Gist is slow out of the gate, but with data cached, it's pretty speedy. With indexscan and bitmapscan disabled, these queries all took 300-400 ms. So GIN was never better performing than

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread David E. Wheeler
On Jan 18, 2011, at 1:39 PM, Tom Lane wrote: At the moment my opinion is that gist__int_ops is too broken to be usable, and it's also too uncommented to be fixable by anyone other than the original author. That seems to jibe with your comments from last year:

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread Tom Lane
David E. Wheeler da...@kineticode.com writes: On Jan 18, 2011, at 1:39 PM, Tom Lane wrote: At the moment my opinion is that gist__int_ops is too broken to be usable, and it's also too uncommented to be fixable by anyone other than the original author. That seems to jibe with your comments

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread David E. Wheeler
On Jan 18, 2011, at 1:58 PM, Tom Lane wrote: I'm noticing also that I get different rowcounts than you do, although possibly that has something to do with the partial-index conditions, which I'm not trying to duplicate here (all rows in my table pass those two tests). Shall I send you data

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread Tom Lane
David E. Wheeler da...@kineticode.com writes: On Jan 18, 2011, at 1:58 PM, Tom Lane wrote: I'm noticing also that I get different rowcounts than you do, although possibly that has something to do with the partial-index conditions, which I'm not trying to duplicate here (all rows in my table

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread David E. Wheeler
On Jan 18, 2011, at 3:08 PM, Tom Lane wrote: Shall I send you data with the other two columns?: No, I see no reason to think that has much to do with it. I'm wondering if your table is itself a bit bloated ... Nope. Just ran the bloat query from check_postgres.pl. Bloat is 0. Not

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread Tom Lane
I wrote: No, I see no reason to think that has much to do with it. I'm wondering if your table is itself a bit bloated ... Actually ... I notice you did not show EXPLAIN ANALYZE output for your tests. Now I'm wondering whether you tested the right thing at all. I got burnt that way too.

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-18 Thread David E. Wheeler
On Jan 18, 2011, at 3:46 PM, Tom Lane wrote: The above is using the index, but only as a guide to where the rows satisfying the partial-index predicate are --- note the lack of any index condition in the indexscan node. That's because the query_int query is not in fact compatible with the

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-16 Thread David E. Wheeler
On Jan 14, 2011, at 11:37 PM, David E. Wheeler wrote: Hard to comment on any of this without a concrete example (including data) to look at. Given the bugs we've recently found in the picksplit algorithms for other contrib modules, I wouldn't be too surprised if the sucky GiST performance

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-14 Thread David E . Wheeler
On Jan 8, 2011, at 9:41 PM, Tom Lane wrote: David E. Wheeler da...@kineticode.com writes: On Jan 7, 2011, at 4:19 PM, Tom Lane wrote: Well, actually, I just committed it. If you want to test, feel free. Note that right now only the anyarray @ @ operators are genuinely fixed ... I plan to

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-14 Thread Tom Lane
David E. Wheeler da...@kineticode.com writes: So some questions: * Is something seriously wrong with GiST index creation on integer[] columns? * Why does GIN performance appear to be no better than table scans on integer[] columns? * Why does it take 3-4x longer to create the GIN than the

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-14 Thread David E. Wheeler
On Jan 14, 2011, at 5:54 PM, Tom Lane wrote: David E. Wheeler da...@kineticode.com writes: So some questions: * Is something seriously wrong with GiST index creation on integer[] columns? * Why does GIN performance appear to be no better than table scans on integer[] columns? * Why

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-13 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes: David E. Wheeler da...@kineticode.com writes: On Jan 12, 2011, at 4:35 PM, Tom Lane wrote: No, what we need is a decent extension package manager ;-) Yeah. Maybe you can do that this weekend? Or, I dunno, while you “sleep” tonight? Supposedly it's in

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-13 Thread Bruce Momjian
Robert Haas wrote: On Tue, Jan 4, 2011 at 4:49 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Jan 4, 2011 at 4:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: * Existing GIN indexes are upwards compatible so far as on-disk storage goes, but they will of

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-13 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: What does pg_upgrade need to do about this for 9.1? Nothing. An existing GIN index can still do all the same queries it could do before. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-12 Thread David E. Wheeler
Tom, On Jan 8, 2011, at 9:41 PM, Tom Lane wrote: David E. Wheeler da...@kineticode.com writes: On Jan 7, 2011, at 4:19 PM, Tom Lane wrote: Well, actually, I just committed it. If you want to test, feel free. Note that right now only the anyarray @ @ operators are genuinely fixed ... I

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-12 Thread Tom Lane
David E. Wheeler da...@kineticode.com writes: Thanks, working on it now. I'm restoring a dump from 8.4, but got these erors: pg_restore: [archiver (db)] Error from TOC entry 3227; 2616 46485 OPERATOR CLASS gin__int_ops postgres pg_restore: [archiver (db)] could not execute query: ERROR:

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-12 Thread David E. Wheeler
On Jan 12, 2011, at 4:35 PM, Tom Lane wrote: Did a signature change or something? Yeah. I think if you just load up the current contrib/intarray first, you'll be fine (ignore all the object-already-exists errors). Oh, from 9.1devel? yeah, okay. Will do that tomorrow (finishing the current

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-12 Thread Tom Lane
David E. Wheeler da...@kineticode.com writes: On Jan 12, 2011, at 4:35 PM, Tom Lane wrote: No, what we need is a decent extension package manager ;-) Yeah. Maybe you can do that this weekend? Or, I dunno, while you “sleep” tonight? Supposedly it's in the queue for the upcoming CF :-)

contrib/intarray (was Re: [HACKERS] Fixing GIN for empty/null/full-scan cases)

2011-01-08 Thread Tom Lane
David E. Wheeler da...@kineticode.com writes: On Jan 7, 2011, at 4:19 PM, Tom Lane wrote: Well, actually, I just committed it. If you want to test, feel free. Note that right now only the anyarray @ @ operators are genuinely fixed ... I plan to hack on tsearch and contrib pretty soon though.

Re: contrib/intarray (was Re: [HACKERS] Fixing GIN for empty/null/full-scan cases)

2011-01-08 Thread David E. Wheeler
On Jan 8, 2011, at 1:59 PM, Tom Lane wrote: Hrm, the queries I wrote for this sort of thing use intarray: I'm going to work on contrib/intarray first (before tsearch etc) so that you can do whatever testing you want sooner. No, of course not. One of the things that first got me annoyed

Re: contrib/intarray (was Re: [HACKERS] Fixing GIN for empty/null/full-scan cases)

2011-01-08 Thread Tom Lane
David E. Wheeler da...@kineticode.com writes: On Jan 8, 2011, at 1:59 PM, Tom Lane wrote: There seem to be three ways in which intarray is simpler/faster than the generic operators: * restricted to integer arrays * restricted to 1-D arrays * doesn't allow nulls in the arrays My

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-08 Thread Tom Lane
David E. Wheeler da...@kineticode.com writes: On Jan 7, 2011, at 4:19 PM, Tom Lane wrote: Well, actually, I just committed it. If you want to test, feel free. Note that right now only the anyarray @ @ operators are genuinely fixed ... I plan to hack on tsearch and contrib pretty soon though.

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-07 Thread Tom Lane
I wrote: 2. Add another output bool parameter to extractQuery that it must set true (from a default false state) if the query could match with no check values set. This would prompt the GIN code to search for EMPTY_ITEM placeholders, but they'd not be part of the check[] array. On further

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-07 Thread David E . Wheeler
On Jan 4, 2011, at 3:18 PM, Josh Berkus wrote: Actually, there's been a *lot* of complaining about the GIN issues. It's just that most of that complaining doesn't reach -hackers. The common pattern I've seen in our practice and on IRC is: 1) user has GiST indexes 2) user tries converting

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-07 Thread Tom Lane
David E. Wheeler da...@kineticode.com writes: We (PGX) actually have a client who could use this. Tom, if you have patches as you work on this (or, better, a branch in a Git repo), I could do some testing on your client's code with it. It would involve converting from a GiST to a GIN index

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-07 Thread David E. Wheeler
On Jan 7, 2011, at 4:19 PM, Tom Lane wrote: Well, actually, I just committed it. If you want to test, feel free. Note that right now only the anyarray @ @ operators are genuinely fixed ... I plan to hack on tsearch and contrib pretty soon though. Hrm, the queries I wrote for this sort of

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-07 Thread Tom Lane
David E. Wheeler da...@kineticode.com writes: On Jan 7, 2011, at 4:19 PM, Tom Lane wrote: Well, actually, I just committed it. If you want to test, feel free. Note that right now only the anyarray @ @ operators are genuinely fixed ... I plan to hack on tsearch and contrib pretty soon though.

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-06 Thread Euler Taveira de Oliveira
Em 06-01-2011 21:31, Tom Lane escreveu: I think I like option #2 better. Comments? +1. -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

[HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-04 Thread Tom Lane
I've been thinking about how to fix GIN's assorted corner-case problems, as has been discussed several times, most recently here: http://archives.postgresql.org/pgsql-hackers/2010-10/msg00521.php See also http://wiki.postgresql.org/wiki/Todo#GIN There are basically three related issues: 1. GIN

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-04 Thread Robert Haas
On Tue, Jan 4, 2011 at 4:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: * Existing GIN indexes are upwards compatible so far as on-disk storage goes, but they will of course be missing entries for empty, null, or null-containing items.  Users who want to do searches that should find such items will

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-04 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Tue, Jan 4, 2011 at 4:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: * Existing GIN indexes are upwards compatible so far as on-disk storage goes, but they will of course be missing entries for empty, null, or null-containing items.  Users who want to do

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-04 Thread Robert Haas
On Tue, Jan 4, 2011 at 4:49 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Jan 4, 2011 at 4:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: * Existing GIN indexes are upwards compatible so far as on-disk storage goes, but they will of course be missing

Re: [HACKERS] Fixing GIN for empty/null/full-scan cases

2011-01-04 Thread Josh Berkus
On 1/4/11 1:49 PM, Tom Lane wrote: I don't think it's really worth the trouble. The GIN code has been broken for these types of queries since day one, and yet we've had only maybe half a dozen complaints about it. Moreover there's no practical way to avoid trying to use the index, since in