Re: BRIN indexes (was Re: [HACKERS] Minmax indexes)

2014-09-14 Thread Emanuel Calvo
El 08/09/14 13:02, Alvaro Herrera escribió: Here's version 18. I have renamed it: These are now BRIN indexes. I have fixed numerous race conditions and deadlocks. In particular I fixed this problem you noted: Heikki Linnakangas wrote: Another race condition: If a new tuple is inserted

Re: BRIN indexes (was Re: [HACKERS] Minmax indexes)

2014-09-09 Thread Heikki Linnakangas
On 09/08/2014 07:02 PM, Alvaro Herrera wrote: Here's version 18. I have renamed it: These are now BRIN indexes. I have fixed numerous race conditions and deadlocks. In particular I fixed this problem you noted: Heikki Linnakangas wrote: Another race condition: If a new tuple is inserted to

Re: [HACKERS] Minmax indexes

2014-08-20 Thread Alvaro Herrera
Alvaro Herrera wrote: So here's v16, rebased on top of 9bac66020. As far as I am concerned, this is the last version before I start renaming everything to BRIN and then commit. FWIW in case you or others have interest, here's the diff between your patch and v16. Also, for illustrative

Re: [HACKERS] Minmax indexes

2014-08-15 Thread Heikki Linnakangas
On 08/15/2014 02:02 AM, Alvaro Herrera wrote: Alvaro Herrera wrote: Heikki Linnakangas wrote: I'm sure this still needs some cleanup, but here's the patch, based on your v14. Now that I know what this approach looks like, I still like it much better. The insert and update code is somewhat

Re: [HACKERS] Minmax indexes

2014-08-15 Thread Heikki Linnakangas
On 08/15/2014 10:26 AM, Heikki Linnakangas wrote: On 08/15/2014 02:02 AM, Alvaro Herrera wrote: Alvaro Herrera wrote: Heikki Linnakangas wrote: I'm sure this still needs some cleanup, but here's the patch, based on your v14. Now that I know what this approach looks like, I still like it

Re: [HACKERS] Minmax indexes

2014-08-15 Thread Fujii Masao
On Fri, Aug 15, 2014 at 8:02 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Alvaro Herrera wrote: Heikki Linnakangas wrote: I'm sure this still needs some cleanup, but here's the patch, based on your v14. Now that I know what this approach looks like, I still like it much better. The

Re: [HACKERS] Minmax indexes

2014-08-15 Thread Heikki Linnakangas
On 08/15/2014 02:02 AM, Alvaro Herrera wrote: Alvaro Herrera wrote: Heikki Linnakangas wrote: I'm sure this still needs some cleanup, but here's the patch, based on your v14. Now that I know what this approach looks like, I still like it much better. The insert and update code is somewhat

Re: [HACKERS] Minmax indexes

2014-08-15 Thread Alvaro Herrera
Fujii Masao wrote: I've not read the patch yet. But while testing the feature, I found that * Brin index cannot be created on CHAR(n) column. Maybe other data types have the same problem. Yeah, it's just a matter of adding an opclass for it -- pretty simple stuff really, because you

Re: [HACKERS] Minmax indexes

2014-08-12 Thread Alvaro Herrera
Heikki Linnakangas wrote: I couldn't resist starting to hack on this, and implemented the scheme I've been having in mind: 1. MMTuple contains the block number of the heap page (range) that the tuple represents. Vacuum is no longer needed to clean up old tuples; when an index tuples is

Re: [HACKERS] Minmax indexes

2014-08-10 Thread Simon Riggs
On 8 August 2014 16:03, Heikki Linnakangas hlinnakan...@vmware.com wrote: 1. MMTuple contains the block number of the heap page (range) that the tuple represents. Vacuum is no longer needed to clean up old tuples; when an index tuples is updated, the old tuple is deleted atomically with the

Re: [HACKERS] Minmax indexes

2014-08-10 Thread Simon Riggs
On 8 August 2014 10:01, Heikki Linnakangas hlinnakan...@vmware.com wrote: It's possible that two backends arrive at phase 3 at the same time, with different values. For example, backend A wants to update the minimum to contain 10, and and backend B wants to update it to 5. Now, if backend B

Re: [HACKERS] Minmax indexes

2014-08-10 Thread Simon Riggs
On 8 August 2014 16:03, Heikki Linnakangas hlinnakan...@vmware.com wrote: I couldn't resist starting to hack on this, and implemented the scheme I've been having in mind: 1. MMTuple contains the block number of the heap page (range) that the tuple represents. Vacuum is no longer needed to

Re: [HACKERS] Minmax indexes

2014-08-10 Thread Heikki Linnakangas
On 08/10/2014 12:22 PM, Simon Riggs wrote: On 8 August 2014 16:03, Heikki Linnakangas hlinnakan...@vmware.com wrote: 1. MMTuple contains the block number of the heap page (range) that the tuple represents. Vacuum is no longer needed to clean up old tuples; when an index tuples is updated, the

Re: [HACKERS] Minmax indexes

2014-08-10 Thread Heikki Linnakangas
On 08/10/2014 12:42 PM, Simon Riggs wrote: On 8 August 2014 16:03, Heikki Linnakangas hlinnakan...@vmware.com wrote: I couldn't resist starting to hack on this, and implemented the scheme I've been having in mind: 1. MMTuple contains the block number of the heap page (range) that the tuple

Re: [HACKERS] Minmax indexes

2014-08-10 Thread Claudio Freire
On Fri, Aug 8, 2014 at 6:01 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: It's possible that two backends arrive at phase 3 at the same time, with different values. For example, backend A wants to update the minimum to contain 10, and and backend B wants to update it to 5. Now, if

Re: [HACKERS] Minmax indexes

2014-08-08 Thread Heikki Linnakangas
I think there's a race condition in mminsert, if two backends insert a tuple to the same heap page range concurrently. mminsert does this: 1. Fetch the MMtuple for the page range 2. Check if any of the stored datums need updating 3. Unlock the page. 4. Lock the page again in exclusive mode. 5.

Re: [HACKERS] Minmax indexes

2014-08-08 Thread Heikki Linnakangas
Another race condition: If a new tuple is inserted to the range while summarization runs, it's possible that the new tuple isn't included in the tuple that the summarization calculated, nor does the insertion itself udpate it. 1. There is no index tuple for page range 1-10 2. Summarization

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Robert Haas
On Wed, Aug 6, 2014 at 4:06 PM, Nicolas Barbier nicolas.barb...@gmail.com wrote: 2014-08-06 Claudio Freire klaussfre...@gmail.com: So, I like blockfilter a lot. I change my vote to blockfilter ;) +1 for blockfilter, because it stresses the fact that the physical arrangement of rows in blocks

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Simon Riggs
On 7 August 2014 14:53, Robert Haas robertmh...@gmail.com wrote: On Wed, Aug 6, 2014 at 4:06 PM, Nicolas Barbier nicolas.barb...@gmail.com wrote: 2014-08-06 Claudio Freire klaussfre...@gmail.com: So, I like blockfilter a lot. I change my vote to blockfilter ;) +1 for blockfilter, because it

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Claudio Freire
On Thu, Aug 7, 2014 at 11:16 AM, Simon Riggs si...@2ndquadrant.com wrote: On 7 August 2014 14:53, Robert Haas robertmh...@gmail.com wrote: On Wed, Aug 6, 2014 at 4:06 PM, Nicolas Barbier nicolas.barb...@gmail.com wrote: 2014-08-06 Claudio Freire klaussfre...@gmail.com: So, I like blockfilter

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Alvaro Herrera
Simon Riggs wrote: On 7 August 2014 14:53, Robert Haas robertmh...@gmail.com wrote: On Wed, Aug 6, 2014 at 4:06 PM, Nicolas Barbier nicolas.barb...@gmail.com wrote: 2014-08-06 Claudio Freire klaussfre...@gmail.com: So, I like blockfilter a lot. I change my vote to blockfilter ;) +1

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Robert Haas
On Thu, Aug 7, 2014 at 10:16 AM, Simon Riggs si...@2ndquadrant.com wrote: On 7 August 2014 14:53, Robert Haas robertmh...@gmail.com wrote: On Wed, Aug 6, 2014 at 4:06 PM, Nicolas Barbier nicolas.barb...@gmail.com wrote: 2014-08-06 Claudio Freire klaussfre...@gmail.com: So, I like blockfilter

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Oleg Bartunov
+1 for BRIN ! On Thu, Aug 7, 2014 at 6:16 PM, Simon Riggs si...@2ndquadrant.com wrote: On 7 August 2014 14:53, Robert Haas robertmh...@gmail.com wrote: On Wed, Aug 6, 2014 at 4:06 PM, Nicolas Barbier nicolas.barb...@gmail.com wrote: 2014-08-06 Claudio Freire klaussfre...@gmail.com: So, I

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Nicolas Barbier
2014-08-07 Oleg Bartunov obartu...@gmail.com: +1 for BRIN ! +1, rolls off the tongue smoothly and captures the essence :-). Nicolas -- A. Because it breaks the logical sequence of discussion. Q. Why is top posting bad? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Petr Jelinek
On 07/08/14 16:16, Simon Riggs wrote: A better description would be block range index since we are indexing a range of blocks (not just one block). Perhaps a better one would be simply range index, which we could abbreviate to RIN or BRIN. +1 for block range index (BRIN) -- Petr Jelinek

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Josh Berkus
On 08/07/2014 08:38 AM, Oleg Bartunov wrote: +1 for BRIN ! On Thu, Aug 7, 2014 at 6:16 PM, Simon Riggs si...@2ndquadrant.com wrote: On 7 August 2014 14:53, Robert Haas robertmh...@gmail.com wrote: A better description would be block range index since we are indexing a range of blocks (not

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Michael Paquier
On Fri, Aug 8, 2014 at 9:47 AM, Josh Berkus j...@agliodbs.com wrote: On 08/07/2014 08:38 AM, Oleg Bartunov wrote: +1 for BRIN ! On Thu, Aug 7, 2014 at 6:16 PM, Simon Riggs si...@2ndquadrant.com wrote: On 7 August 2014 14:53, Robert Haas robertmh...@gmail.com wrote: A better description would

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Peter Geoghegan
On Thu, Aug 7, 2014 at 7:58 AM, Robert Haas robertmh...@gmail.com wrote: range index might get confused with range types; block range index seems better. I like summary, but I'm fine with block range index or block filter index, too. +1 -- Peter Geoghegan -- Sent via pgsql-hackers

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Josh Berkus
On 08/07/2014 05:52 PM, Michael Paquier wrote: On Fri, Aug 8, 2014 at 9:47 AM, Josh Berkus j...@agliodbs.com wrote: On 08/07/2014 08:38 AM, Oleg Bartunov wrote: +1 for BRIN ! On Thu, Aug 7, 2014 at 6:16 PM, Simon Riggs si...@2ndquadrant.com wrote: On 7 August 2014 14:53, Robert Haas

Re: [HACKERS] Minmax indexes

2014-08-06 Thread Robert Haas
On Tue, Aug 5, 2014 at 7:55 PM, Josh Berkus j...@agliodbs.com wrote: On 08/05/2014 04:41 PM, Alvaro Herrera wrote: I have chosen to keep the name minmax, even if the opclasses now let one implement completely different things on top of it such as geometry bounding boxes and bloom filters (aka

Re: [HACKERS] Minmax indexes

2014-08-06 Thread Claudio Freire
On Wed, Aug 6, 2014 at 1:25 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Summary seems good. If I get enough votes I can change it to that. CREATE INDEX foo ON t USING summary (cols) Summarizing seems weird on that command. Not sure about compressed range, as you would have to use an

Re: [HACKERS] Minmax indexes

2014-08-06 Thread Bruce Momjian
On Wed, Aug 6, 2014 at 01:31:14PM -0300, Claudio Freire wrote: On Wed, Aug 6, 2014 at 1:25 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: CREATE INDEX foo ON t USING crange (cols) -- misspelling of cringe? CREATE INDEX foo ON t USING comprange (cols) CREATE INDEX foo ON t USING

Re: [HACKERS] Minmax indexes

2014-08-06 Thread Claudio Freire
On Wed, Aug 6, 2014 at 1:25 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: CREATE INDEX foo ON t USING crange (cols) -- misspelling of cringe? CREATE INDEX foo ON t USING comprange (cols) CREATE INDEX foo ON t USING compressedrng (cols) -- ugh -- or use an identifier with whitespace:

Re: [HACKERS] Minmax indexes

2014-08-06 Thread Claudio Freire
On Wed, Aug 6, 2014 at 1:35 PM, Bruce Momjian br...@momjian.us wrote: On Wed, Aug 6, 2014 at 01:31:14PM -0300, Claudio Freire wrote: On Wed, Aug 6, 2014 at 1:25 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: CREATE INDEX foo ON t USING crange (cols) -- misspelling of cringe? CREATE

Re: [HACKERS] Minmax indexes

2014-08-06 Thread Claudio Freire
On Wed, Aug 6, 2014 at 1:55 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Claudio Freire wrote: On Wed, Aug 6, 2014 at 1:25 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: CREATE INDEX foo ON t USING crange (cols) -- misspelling of cringe? CREATE INDEX foo ON t USING comprange

Re: [HACKERS] Minmax indexes

2014-08-06 Thread Nicolas Barbier
2014-08-06 Claudio Freire klaussfre...@gmail.com: So, I like blockfilter a lot. I change my vote to blockfilter ;) +1 for blockfilter, because it stresses the fact that the physical arrangement of rows in blocks matters for this index. Nicolas -- A. Because it breaks the logical sequence of

Re: [HACKERS] Minmax indexes

2014-08-05 Thread Josh Berkus
On 08/05/2014 04:41 PM, Alvaro Herrera wrote: I have chosen to keep the name minmax, even if the opclasses now let one implement completely different things on top of it such as geometry bounding boxes and bloom filters (aka bitmap indexes). I don't see a need for a rename: essentially, in PR

Re: [HACKERS] Minmax indexes

2014-08-05 Thread Alvaro Herrera
FWIW I think I haven't responded appropriately to the points raised by Heikki. Basically, as I see it there are three main items: 1. the revmap physical-to-logical mapping is too complex; let's use something else. We had revmap originally in a separate fork. The current approach grew out of

Re: [HACKERS] Minmax indexes

2014-07-29 Thread Heikki Linnakangas
On 07/10/2014 12:41 AM, Alvaro Herrera wrote: Heikki Linnakangas wrote: On 06/23/2014 08:07 PM, Alvaro Herrera wrote: I feel that the below would nevertheless be simpler: I wonder if it would be simpler to just always store the revmap pages in the beginning of the index, before any other

Re: [HACKERS] Minmax indexes

2014-07-14 Thread Robert Haas
On Wed, Jul 9, 2014 at 5:16 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: The way it works now, each opclass needs to have three support procedures; I've called them getOpers, maybeUpdateValues, and compare. (I realize these names are pretty bad, and will be changing them.) getOpers is

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Fujii Masao
On Thu, Jul 10, 2014 at 6:16 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Claudio Freire wrote: An aggregate to generate a compressed set from several values A function which adds a new value to the compressed set and returns the new compressed set A function which tests if a value is

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Simon Riggs
On 9 July 2014 23:54, Peter Geoghegan p...@heroku.com wrote: On Wed, Jul 9, 2014 at 2:16 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: All this being said, I'm sticking to the name Minmax indexes. There was a poll in pgsql-advocacy

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Simon Riggs
On 10 July 2014 00:13, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Josh Berkus wrote: On 07/09/2014 02:16 PM, Alvaro Herrera wrote: The way it works now, each opclass needs to have three support procedures; I've called them getOpers, maybeUpdateValues, and compare. (I realize these

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Alvaro Herrera
Fujii Masao wrote: On Thu, Jul 10, 2014 at 6:16 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Here's a new version of this patch, which is more generic the original versions, and similar to what you describe. I've not read the discussion so far at all, but I found the problem when

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Claudio Freire
On Thu, Jul 10, 2014 at 4:20 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Claudio Freire wrote: On Wed, Jul 9, 2014 at 6:16 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Another thing I noticed is that version 8 of the patch blindly believed the pages_per_range declared in

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Greg Stark
On Fri, Jul 11, 2014 at 6:00 PM, Claudio Freire klaussfre...@gmail.com wrote: Marking as read-only is ok, or emitting a NOTICE so that if anyone changes those parameters that change the shape of the index, they know it needs a rebuild would be OK too. Both mechanisms work for me. We don't

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Claudio Freire
On Fri, Jul 11, 2014 at 3:47 PM, Greg Stark st...@mit.edu wrote: On Fri, Jul 11, 2014 at 6:00 PM, Claudio Freire klaussfre...@gmail.com wrote: Marking as read-only is ok, or emitting a NOTICE so that if anyone changes those parameters that change the shape of the index, they know it needs a

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Claudio Freire
On Wed, Jul 9, 2014 at 6:16 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Another thing I noticed is that version 8 of the patch blindly believed the pages_per_range declared in catalogs. This meant that if somebody did alter index foo set pages_per_range=123 the index would immediately

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Alvaro Herrera
Claudio Freire wrote: On Wed, Jul 9, 2014 at 6:16 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Another thing I noticed is that version 8 of the patch blindly believed the pages_per_range declared in catalogs. This meant that if somebody did alter index foo set pages_per_range=123

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Josh Berkus
On 07/10/2014 12:20 PM, Alvaro Herrera wrote: So I guess the only thing left is to issue a NOTICE when said alter takes place (I don't see that on the patch, but maybe it's there?) That's not in the patch. I don't think we have an appropriate place to emit such a notice. What do you mean by

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Jaime Casanova
On Thu, Jul 10, 2014 at 3:50 PM, Josh Berkus j...@agliodbs.com wrote: On 07/10/2014 12:20 PM, Alvaro Herrera wrote: So I guess the only thing left is to issue a NOTICE when said alter takes place (I don't see that on the patch, but maybe it's there?) That's not in the patch. I don't think we

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Alvaro Herrera
Josh Berkus wrote: On 07/10/2014 12:20 PM, Alvaro Herrera wrote: So I guess the only thing left is to issue a NOTICE when said alter takes place (I don't see that on the patch, but maybe it's there?) That's not in the patch. I don't think we have an appropriate place to emit such a

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Josh Berkus
On 07/10/2014 02:30 PM, Jaime Casanova wrote: How is this different from ALTER TABLE foo SET (FILLFACTOR=80); or from ALTER TABLE foo ALTER bar SET STORAGE EXTERNAL; ? we don't get a notice for these cases either Good idea. We should also emit notices for those. Well, maybe not for

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Jeff Janes
On Thu, Jul 10, 2014 at 2:30 PM, Jaime Casanova ja...@2ndquadrant.com wrote: On Thu, Jul 10, 2014 at 3:50 PM, Josh Berkus j...@agliodbs.com wrote: On 07/10/2014 12:20 PM, Alvaro Herrera wrote: So I guess the only thing left is to issue a NOTICE when said alter takes place (I don't see

Re: [HACKERS] Minmax indexes

2014-07-10 Thread Greg Stark
On Thu, Jul 10, 2014 at 10:29 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: What I think should happen is that if the value is changed, the index sholud be rebuilt right there. I disagree. It would be a non-orthogonal interface if ALTER TABLE sometimes causes the index to be rebuilt and

Re: [HACKERS] Minmax indexes

2014-07-09 Thread Alvaro Herrera
Heikki Linnakangas wrote: On 06/23/2014 08:07 PM, Alvaro Herrera wrote: I feel that the below would nevertheless be simpler: I wonder if it would be simpler to just always store the revmap pages in the beginning of the index, before any other pages. Finding the revmap page would then be

Re: [HACKERS] Minmax indexes

2014-07-09 Thread Josh Berkus
On 07/09/2014 02:16 PM, Alvaro Herrera wrote: The way it works now, each opclass needs to have three support procedures; I've called them getOpers, maybeUpdateValues, and compare. (I realize these names are pretty bad, and will be changing them.) I kind of like maybeUpdateValues. Very ...

Re: [HACKERS] Minmax indexes

2014-07-09 Thread Peter Geoghegan
On Wed, Jul 9, 2014 at 2:16 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: All this being said, I'm sticking to the name Minmax indexes. There was a poll in pgsql-advocacy http://www.postgresql.org/message-id/53a0b4f8.8080...@agliodbs.com about a new name, but there were no suggestions

Re: [HACKERS] Minmax indexes

2014-07-09 Thread Alvaro Herrera
Josh Berkus wrote: On 07/09/2014 02:16 PM, Alvaro Herrera wrote: The way it works now, each opclass needs to have three support procedures; I've called them getOpers, maybeUpdateValues, and compare. (I realize these names are pretty bad, and will be changing them.) I kind of like

Re: [HACKERS] Minmax indexes

2014-07-09 Thread Greg Stark
On Wed, Jul 9, 2014 at 10:16 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: there is no hardcoded assumption on the number of index values stored per heap column, so it is possible to build an opclass that stores a bounding box column for a geometry heap column, for instance. I think the

Re: [HACKERS] Minmax indexes

2014-06-23 Thread Heikki Linnakangas
Some comments, aside from the design wrt. bounding boxes etc. : On 06/15/2014 05:34 AM, Alvaro Herrera wrote: Robert Haas wrote: On Wed, Sep 25, 2013 at 4:34 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Here's an updated version of this patch, with fixes to all the bugs reported so far.

Re: [HACKERS] Minmax indexes

2014-06-23 Thread Alvaro Herrera
Heikki Linnakangas wrote: Some comments, aside from the design wrt. bounding boxes etc. : Thanks. I haven't commented on that sub-thread because I think it's possible to come up with a reasonable design that solves the issue by adding a couple of amprocs. I need to do some more thinking to

Re: [HACKERS] Minmax indexes

2014-06-23 Thread Heikki Linnakangas
On 06/23/2014 08:07 PM, Alvaro Herrera wrote: Heikki Linnakangas wrote: With 8k block size, that's just enough to cover the full range of 2^32-1 blocks that you'll need if you set mm_pages_per_range=1. Each regular revmap page can store about 8192/6 = 1365 item pointers, each array revmap page

Re: [HACKERS] Minmax indexes

2014-06-23 Thread Robert Haas
On Thu, Jun 19, 2014 at 12:32 PM, Greg Stark st...@mit.edu wrote: On Wed, Jun 18, 2014 at 4:51 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Implementing something is a good way to demonstrate how it would look like. But no, I don't insist on implementing every possible design whenever

Re: [HACKERS] Minmax indexes

2014-06-21 Thread Martijn van Oosterhout
I'm sorry if I missed something, but ISTM this is beginning to look a lot like GiST. This was pointed out by Robert Haas last year. On Wed, Jun 18, 2014 at 12:09:42PM -0300, Claudio Freire wrote: So, you have: An aggregate to generate a compressed set from several values Which GiST does by

Re: [HACKERS] Minmax indexes

2014-06-19 Thread Vik Fearing
On 06/18/2014 12:46 PM, Andres Freund wrote: Isn't 'simpler implementation' a valid reason that's already been discussed onlist? Obviously simpler implementation doesn't trump everything, but it's one valid reason... Note that I have zap to do with the design of this feature. I work for

Re: [HACKERS] Minmax indexes

2014-06-19 Thread Heikki Linnakangas
On 06/18/2014 06:09 PM, Claudio Freire wrote: On Tue, Jun 17, 2014 at 8:48 PM, Greg Stark st...@mit.edu wrote: An aggregate to generate a min/max bounding box from several values A function which takes an bounding box and a new value and returns the new bounding box A function which tests if a

Re: [HACKERS] Minmax indexes

2014-06-19 Thread Tom Lane
Vik Fearing vik.fear...@dalibo.com writes: On 06/18/2014 12:46 PM, Andres Freund wrote: So to implement a feature one now has to implement the most generic variant as a prototype first? Really? Well, there is the inventor's paradox to consider. I have not seen anyone demanding a different

Re: [HACKERS] Minmax indexes

2014-06-19 Thread Greg Stark
On Wed, Jun 18, 2014 at 4:51 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Implementing something is a good way to demonstrate how it would look like. But no, I don't insist on implementing every possible design whenever a new feature is proposed. I liked Greg's sketch of what the

Re: [HACKERS] Minmax indexes

2014-06-19 Thread Claudio Freire
On Thu, Jun 19, 2014 at 10:06 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 06/18/2014 06:09 PM, Claudio Freire wrote: On Tue, Jun 17, 2014 at 8:48 PM, Greg Stark st...@mit.edu wrote: An aggregate to generate a min/max bounding box from several values A function which takes an

Re: [HACKERS] Minmax indexes

2014-06-19 Thread Claudio Freire
On Wed, Jun 18, 2014 at 8:51 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I liked Greg's sketch of what the opclass support functions would be. It doesn't seem significantly more complicated than what's in the patch now. Which was On Tue, Jun 17, 2014 at 8:48 PM, Greg Stark

Re: [HACKERS] Minmax indexes

2014-06-18 Thread Heikki Linnakangas
On 06/17/2014 09:16 PM, Andres Freund wrote: On 2014-06-17 12:14:00 -0400, Robert Haas wrote: On Tue, Jun 17, 2014 at 12:04 PM, Andres Freund and...@2ndquadrant.com wrote: Well, I'm not the guy who does things with geometric data, but I don't want to ignore the significant percentage of our

Re: [HACKERS] Minmax indexes

2014-06-18 Thread Andres Freund
On 2014-06-18 12:18:26 +0300, Heikki Linnakangas wrote: On 06/17/2014 09:16 PM, Andres Freund wrote: Well, it might be doable to correlate them along one axis, but along both? That's more complicated... And even alongside one axis you already get into problems if your geometries are

Re: [HACKERS] Minmax indexes

2014-06-18 Thread Andres Freund
On 2014-06-17 16:48:07 -0700, Greg Stark wrote: On Tue, Jun 17, 2014 at 11:16 AM, Andres Freund and...@2ndquadrant.com wrote: Well, it might be doable to correlate them along one axis, but along both? That's more complicated... And even alongside one axis you already get into problems if

Re: [HACKERS] Minmax indexes

2014-06-18 Thread Heikki Linnakangas
On 06/18/2014 01:46 PM, Andres Freund wrote: On 2014-06-18 12:18:26 +0300, Heikki Linnakangas wrote: The main problem with using it for geometric types is that you can't easily CLUSTER the table to make the minmax index effective again. But there are ways around that. Which are? Sure you can

Re: [HACKERS] Minmax indexes

2014-06-18 Thread Robert Haas
On Tue, Jun 17, 2014 at 2:16 PM, Andres Freund and...@2ndquadrant.com wrote: But I'm not trying to say that we absolutely have to support that kind of thing; what I am trying to say is that there should be a README or a mailing list post or some such that says: We thought about how generic to

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Robert Haas
On Sat, Jun 14, 2014 at 10:34 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Robert Haas wrote: On Wed, Sep 25, 2013 at 4:34 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Here's an updated version of this patch, with fixes to all the bugs reported so far. Thanks to Thom Brown,

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Andres Freund
On 2014-06-17 10:26:11 -0400, Robert Haas wrote: On Sat, Jun 14, 2014 at 10:34 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Robert Haas wrote: On Wed, Sep 25, 2013 at 4:34 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Here's an updated version of this patch, with fixes to all

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Greg Stark
On Tue, Jun 17, 2014 at 3:31 PM, Andres Freund and...@2ndquadrant.com wrote: Is there actually a significant usecase behind that wish or just a general demand for being generic? To me it seems fairly unlikely you'd end up with something useful by doing a minmax index over bounding boxes.

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Robert Haas
On Tue, Jun 17, 2014 at 10:31 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-06-17 10:26:11 -0400, Robert Haas wrote: On Sat, Jun 14, 2014 at 10:34 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Robert Haas wrote: On Wed, Sep 25, 2013 at 4:34 PM, Alvaro Herrera

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Andres Freund
On 2014-06-17 11:48:10 -0400, Robert Haas wrote: On Tue, Jun 17, 2014 at 10:31 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-06-17 10:26:11 -0400, Robert Haas wrote: On Sat, Jun 14, 2014 at 10:34 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Robert Haas wrote: On Wed,

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Robert Haas
On Tue, Jun 17, 2014 at 12:04 PM, Andres Freund and...@2ndquadrant.com wrote: Well, I'm not the guy who does things with geometric data, but I don't want to ignore the significant percentage of our users who are. As you must surely know, the GIST implementations for geometric data types store

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Andres Freund
On 2014-06-17 12:14:00 -0400, Robert Haas wrote: On Tue, Jun 17, 2014 at 12:04 PM, Andres Freund and...@2ndquadrant.com wrote: Well, I'm not the guy who does things with geometric data, but I don't want to ignore the significant percentage of our users who are. As you must surely know,

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Claudio Freire
On Tue, Jun 17, 2014 at 1:04 PM, Andres Freund and...@2ndquadrant.com wrote: For me minmax indexes are helpful because they allow to generate *small* 'coarse' indexes over large volumes of data. From my pov that's possible possible because they don't contain item pointers for every contained

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Josh Berkus
On 06/17/2014 09:14 AM, Robert Haas wrote: Well, I don't know: suppose you're loading geospatial data showing the location of every building in some country. It might easily be the case that the data is or can be loaded in an order that provides pretty good spatial locality, leading to tight

Re: [HACKERS] Minmax indexes

2014-06-17 Thread Greg Stark
On Tue, Jun 17, 2014 at 11:16 AM, Andres Freund and...@2ndquadrant.com wrote: Well, it might be doable to correlate them along one axis, but along both? That's more complicated... And even alongside one axis you already get into problems if your geometries are irregularly sized. Asingle large

Re: [HACKERS] Minmax indexes

2014-01-24 Thread Thom Brown
On 8 November 2013 20:11, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Erik Rijkers wrote: On Thu, September 26, 2013 00:34, Erik Rijkers wrote: On Wed, September 25, 2013 22:34, Alvaro Herrera wrote: [minmax-5.patch] I have the impression it's not quite working correctly. Here's a

Re: [HACKERS] Minmax indexes

2014-01-24 Thread Alvaro Herrera
Thom Brown wrote: On 8 November 2013 20:11, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Erik Rijkers wrote: On Thu, September 26, 2013 00:34, Erik Rijkers wrote: On Wed, September 25, 2013 22:34, Alvaro Herrera wrote: [minmax-5.patch] I have the impression it's not quite

Re: [HACKERS] Minmax indexes

2014-01-24 Thread Thom Brown
On 24 January 2014 17:53, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Thom Brown wrote: On 8 November 2013 20:11, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Erik Rijkers wrote: On Thu, September 26, 2013 00:34, Erik Rijkers wrote: On Wed, September 25, 2013 22:34, Alvaro Herrera

Re: [HACKERS] Minmax indexes

2014-01-24 Thread Claudio Freire
On Fri, Jan 24, 2014 at 2:54 PM, Thom Brown t...@linux.com wrote: On 24 January 2014 17:53, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Thom Brown wrote: On 8 November 2013 20:11, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Erik Rijkers wrote: On Thu, September 26, 2013 00:34, Erik

Re: [HACKERS] Minmax indexes

2014-01-24 Thread Greg Stark
On Fri, Jan 24, 2014 at 12:58 PM, Claudio Freire klaussfre...@gmail.com wrote: What's the status? I believe I have more than a use for minmax indexes, and wouldn't mind lending a hand if it's within my grasp. I'm also interested in looking at this. Mostly because I have ideas for other

Re: [HACKERS] Minmax indexes (timings)

2013-11-15 Thread Andres Freund
On 2013-11-15 17:11:46 +0100, Erik Rijkers wrote: I've been messing with minmax indexes some more so here are some results of that. Perhaps someone finds these timings useful. Centos 5.7, 32 GB memory, 2 quadcores. '--prefix=/var/data1/pg_stuff/pg_installations/pgsql.minmax'

Re: [HACKERS] Minmax indexes (timings)

2013-11-15 Thread Kevin Grittner
Erik Rijkers e...@xs4all.nl wrote: Perhaps someone finds these timings useful. '--enable-cassert' Assertions can really distort the timings, and not always equally for all code paths.  Any chance of re-running those tests without that? -- Kevin Grittner EDB: http://www.enterprisedb.com The

Re: [HACKERS] Minmax indexes

2013-11-15 Thread Jeff Janes
On Fri, Nov 8, 2013 at 12:11 PM, Alvaro Herrera alvhe...@2ndquadrant.comwrote: Erik Rijkers wrote: On Thu, September 26, 2013 00:34, Erik Rijkers wrote: On Wed, September 25, 2013 22:34, Alvaro Herrera wrote: [minmax-5.patch] I have the impression it's not quite working

Re: [HACKERS] Minmax indexes

2013-11-11 Thread Erik Rijkers
On Fri, November 8, 2013 21:11, Alvaro Herrera wrote: Here's a version 7 of the patch, which fixes these bugs and adds opclasses for a bunch more types (timestamp, timestamptz, date, time, timetz), courtesy of Martín Marqués. It's also been rebased to apply cleanly on top of today's master

Re: [HACKERS] Minmax indexes

2013-11-11 Thread Erik Rijkers
On Mon, November 11, 2013 09:53, Erik Rijkers wrote: On Fri, November 8, 2013 21:11, Alvaro Herrera wrote: Here's a version 7 of the patch, which fixes these bugs and adds opclasses for a bunch more types (timestamp, timestamptz, date, time, timetz), courtesy of Martín Marqués. It's also

Re: [HACKERS] Minmax indexes

2013-11-11 Thread Jeff Janes
On Mon, Nov 11, 2013 at 12:53 AM, Erik Rijkers e...@xs4all.nl wrote: On Fri, November 8, 2013 21:11, Alvaro Herrera wrote: Here's a version 7 of the patch, which fixes these bugs and adds opclasses for a bunch more types (timestamp, timestamptz, date, time, timetz), courtesy of Martín

Re: [HACKERS] Minmax indexes

2013-11-08 Thread Alvaro Herrera
Robert Haas escribió: On Wed, Sep 25, 2013 at 4:34 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Here's an updated version of this patch, with fixes to all the bugs reported so far. Thanks to Thom Brown, Jaime Casanova, Erik Rijkers and Amit Kapila for the reports. I'm not very

Re: [HACKERS] Minmax indexes

2013-11-08 Thread Alvaro Herrera
Alvaro Herrera escribió: I have been playing with having the revmap in the main fork of the index rather than a separate one. ... This is not complete yet; although I have a proof-of-concept working, I still need to write XLog support code and update the pageinspect code to match. Just to

Re: [HACKERS] Minmax indexes

2013-10-01 Thread Robert Haas
On Mon, Sep 30, 2013 at 1:20 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: You can almost create a bounding box opclass in the current implementation, by mapping operator to contains and to not contains. But there's no support for creating a new, larger, bounding box on insert. It

  1   2   >