Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2013-01-18 Thread Jeff Davis
On Thu, 2013-01-17 at 21:03 +0100, Stefan Keller wrote: Hi Jeff I'm perhaps really late in this discussion but I just was made aware of that via the tweet from Josh Berkus about PostgreSQL 9.3: Current Feature Status What is the reason to digg into spatial-joins when there is PostGIS

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2013-01-18 Thread Stefan Keller
Hi Jeff 2013/1/18 Jeff Davis pg...@j-davis.com: On Thu, 2013-01-17 at 21:03 +0100, Stefan Keller wrote: Hi Jeff I'm perhaps really late in this discussion but I just was made aware of that via the tweet from Josh Berkus about PostgreSQL 9.3: Current Feature Status What is the reason to

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2013-01-18 Thread Jeff Davis
On Fri, 2013-01-18 at 12:25 +0100, Stefan Keller wrote: Sounds good. Did you already had contact e.g. with Paul (cc'ed just in case)? And will this clever index also be available within all these hundreds of PostGIS functions? Yes, I've brought the idea up to Paul before, but thank you. It's

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2013-01-17 Thread Stefan Keller
Hi Jeff 2012/4/19 Jeff Davis pg...@j-davis.com: On Wed, 2012-04-18 at 01:21 -0400, Tom Lane wrote: (...) This is just handwaving of course. I think some digging in the spatial-join literature would likely find ideas better than any of these. I will look in some more detail. The merge-like

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-19 Thread Jeff Davis
On Tue, 2012-04-17 at 14:24 -0400, Robert Haas wrote: I thought Jeff was parenthetically complaining about cases like A LEFT JOIN (B INNER JOIN C ON b.y = c.y) ON a.x b.x. That presumably would require the parameterized-path stuff to have any chance of doing partial index scans over B.

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-19 Thread Jeff Davis
On Tue, 2012-04-17 at 14:03 -0400, Robert Haas wrote: I'm actually not sure these are equivalent formulations. Suppose one side has [i,i] where i ranges from 1 to 1000 and the other side the exact same thing plus [1,1000]. That one really big range will come up second on the right

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-19 Thread Jeff Davis
On Wed, 2012-04-18 at 01:21 -0400, Tom Lane wrote: It would be a pretty weird implementation of mergejoin that could discard tuples from the middle of an input stream. Or to be more specific, it wouldn't be the mergejoin itself that could do that at all --- you'd need the input plan node to

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-17 Thread Greg Stark
On Mon, Apr 16, 2012 at 10:20 PM, Simon Riggs si...@2ndquadrant.com wrote: The thing I like most about temp indexes is that they needn't be temporary. I'd like to see something along the lines of demand-created optional indexes, that we reclaim space/maintenance overhead on according to some

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-17 Thread Robert Haas
On Mon, Apr 16, 2012 at 1:43 PM, Jeff Davis pg...@j-davis.com wrote: On Mon, 2012-04-16 at 02:52 -0400, Tom Lane wrote: Jeff Davis pg...@j-davis.com writes:  1. Order the ranges on both sides by the lower bound, then upper bound. Empty ranges can be excluded entirely.  2. Left := first

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-17 Thread Robert Haas
On Mon, Apr 16, 2012 at 7:05 PM, Tom Lane t...@sss.pgh.pa.us wrote: Hmm. This sounds like something that Tom's recent work on parameterized plans ought to have fixed, or if not, it seems closely related. Not really.  It's still going to be a nestloop, and as such not terribly well suited for

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-17 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Mon, Apr 16, 2012 at 1:43 PM, Jeff Davis pg...@j-davis.com wrote: ... Only one side really needs the mark and restore logic, but it was easier to write the pseudocode in a symmetrical way (except step 7). I'm actually not sure these are equivalent

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-16 Thread Darren Duncan
Your proposal makes me think of something similar which might be useful, INclusion constraints. As exclusion constraints might be thought of like a generalization of unique/key constraints, inclusion constraints are like a generalization of foreign key constraints. The inclusion constraints

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-16 Thread Heikki Linnakangas
On 16.04.2012 08:40, Jeff Davis wrote: Does someone know of a spatial join algorithm (without IP claims) that would be as good as this one for ranges? I'd be happy with an algorithm that's specific to ranges, too, but my gut geeling is that there has to be a lot of research on spatial join

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-16 Thread Tom Lane
Jeff Davis pg...@j-davis.com writes: Proposed solution: a modified merge join that can handle ranges. 1. Order the ranges on both sides by the lower bound, then upper bound. Empty ranges can be excluded entirely. 2. Left := first range on left, Right := first range on right 3. If Left or

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-16 Thread Simon Riggs
On Mon, Apr 16, 2012 at 7:52 AM, Tom Lane t...@sss.pgh.pa.us wrote: Dunno.  It might be easier to sell the idea of adding support for range joins in a couple of years, after we've seen how much use ranges get. Once we've started the journey towards range types we must complete it reasonably

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-16 Thread Alexander Korotkov
On Mon, Apr 16, 2012 at 10:29 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 16.04.2012 08:40, Jeff Davis wrote: Does someone know of a spatial join algorithm (without IP claims) that would be as good as this one for ranges? I'd be happy with an algorithm that's

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-16 Thread Jeff Davis
On Mon, 2012-04-16 at 02:52 -0400, Tom Lane wrote: Jeff Davis pg...@j-davis.com writes: 1. Order the ranges on both sides by the lower bound, then upper bound. Empty ranges can be excluded entirely. 2. Left := first range on left, Right := first range on right 3. If Left or Right is

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-16 Thread Jeff Davis
On Mon, 2012-04-16 at 16:22 +0400, Alexander Korotkov wrote: There is a good overview article about spatial joins. http://www.cs.umd.edu/users/hjs/pubs/jacoxtods07.pdf Thank you, that's exactly the kind of overview I was looking for. It shows that there is a lot of methods based on building

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-16 Thread Jeff Davis
On Sun, 2012-04-15 at 23:18 -0700, Darren Duncan wrote: Your proposal makes me think of something similar which might be useful, INclusion constraints. As exclusion constraints might be thought of like a generalization of unique/key constraints, inclusion constraints are like a

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-16 Thread Alexander Korotkov
On Tue, Apr 17, 2012 at 12:12 AM, Jeff Davis pg...@j-davis.com wrote: On Mon, 2012-04-16 at 16:22 +0400, Alexander Korotkov wrote: There is a good overview article about spatial joins. http://www.cs.umd.edu/users/hjs/pubs/jacoxtods07.pdf Thank you, that's exactly the kind of overview I

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-16 Thread Simon Riggs
On Mon, Apr 16, 2012 at 9:12 PM, Jeff Davis pg...@j-davis.com wrote: That had occurred to me, but I was hesitant to only use temp indexes. It still doesn't really offer a good solution when both sides of the join are relatively large (because of random I/O). Also the build speed of the index

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-16 Thread Jay Levitt
Simon Riggs wrote: I'd like to see something along the lines of demand-created optional indexes, that we reclaim space/maintenance overhead on according to some cache management scheme. More space you have, the more of the important ones hang around. The rough same idea applies to materialised

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-16 Thread Robert Haas
On Apr 16, 2012, at 1:40 AM, Jeff Davis pg...@j-davis.com wrote: See attached SQL for example. The Problem statement: slow. Nested loops are the only option, although they can benefit from an inner GiST index if available. But if the join is happening up in the plan tree somewhere, then it's

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-16 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Apr 16, 2012, at 1:40 AM, Jeff Davis pg...@j-davis.com wrote: See attached SQL for example. The Problem statement: slow. Nested loops are the only option, although they can benefit from an inner GiST index if available. But if the join is

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-16 Thread Jeff Davis
On Mon, 2012-04-16 at 22:20 +0100, Simon Riggs wrote: On Mon, Apr 16, 2012 at 9:12 PM, Jeff Davis pg...@j-davis.com wrote: That had occurred to me, but I was hesitant to only use temp indexes. It still doesn't really offer a good solution when both sides of the join are relatively large

[HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-15 Thread Jeff Davis
I hope this is not an inappropriate time for 9.3 discussions. The flip side of asking for submissions in the first couple commitfests means that I need to submit proposals now. What is a Range Join? See attached SQL for example. The key difference is that the join condition is not equality, but