Re: [HACKERS] Patch to support SEMI and ANTI join removal

2015-02-14 Thread David Rowley
On 13 February 2015 at 20:52, Michael Paquier michael.paqu...@gmail.com wrote: On Sun, Nov 23, 2014 at 8:23 PM, David Rowley dgrowle...@gmail.com wrote: As the patch stands there's still a couple of FIXMEs in there, so there's still a bit of work to do yet. Comments are welcome Hm, if

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2015-02-14 Thread David Rowley
There does not seem to be a delete button, so marking as rejected due to this now being a duplicate entry for this patch. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2015-02-13 Thread Andres Freund
On 2015-02-13 17:06:14 +0900, Michael Paquier wrote: On Fri, Feb 13, 2015 at 4:57 PM, Marko Tiikkaja ma...@joh.to wrote: On 2/13/15 8:52 AM, Michael Paquier wrote: On Sun, Nov 23, 2014 at 8:23 PM, David Rowley dgrowle...@gmail.com wrote: As the patch stands there's still a couple of

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2015-02-13 Thread Michael Paquier
On Fri, Feb 13, 2015 at 4:57 PM, Marko Tiikkaja ma...@joh.to wrote: On 2/13/15 8:52 AM, Michael Paquier wrote: On Sun, Nov 23, 2014 at 8:23 PM, David Rowley dgrowle...@gmail.com wrote: As the patch stands there's still a couple of FIXMEs in there, so there's still a bit of work to do yet.

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2015-02-13 Thread Michael Paquier
On Fri, Feb 13, 2015 at 5:12 PM, Andres Freund and...@2ndquadrant.com wrote: On 2015-02-13 17:06:14 +0900, Michael Paquier wrote: On Fri, Feb 13, 2015 at 4:57 PM, Marko Tiikkaja ma...@joh.to wrote: On 2/13/15 8:52 AM, Michael Paquier wrote: On Sun, Nov 23, 2014 at 8:23 PM, David

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2015-02-12 Thread Michael Paquier
On Sun, Nov 23, 2014 at 8:23 PM, David Rowley dgrowle...@gmail.com wrote: As the patch stands there's still a couple of FIXMEs in there, so there's still a bit of work to do yet. Comments are welcome Hm, if there is still work to do, we may as well mark this patch as rejected as-is, also

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2015-02-12 Thread Marko Tiikkaja
On 2/13/15 8:52 AM, Michael Paquier wrote: On Sun, Nov 23, 2014 at 8:23 PM, David Rowley dgrowle...@gmail.com wrote: As the patch stands there's still a couple of FIXMEs in there, so there's still a bit of work to do yet. Comments are welcome Hm, if there is still work to do, we may as well

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-11-23 Thread David Rowley
On Wed, Nov 19, 2014 at 11:49 PM, David Rowley dgrowle...@gmail.com wrote: On Sun, Nov 16, 2014 at 12:19 PM, David Rowley dgrowle...@gmail.com wrote: On Sun, Nov 16, 2014 at 10:09 AM, Simon Riggs si...@2ndquadrant.com wrote: I propose that we keep track of whether there are any

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-11-19 Thread David Rowley
On Sun, Nov 16, 2014 at 12:19 PM, David Rowley dgrowle...@gmail.com wrote: On Sun, Nov 16, 2014 at 10:09 AM, Simon Riggs si...@2ndquadrant.com wrote: I propose that we keep track of whether there are any potentially skippable joins at the top of the plan. When we begin execution we do a

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-11-15 Thread Simon Riggs
On 15 October 2014 11:03, David Rowley dgrowle...@gmail.com wrote: The explain analyze from the above query looks like: test=# explain (analyze, costs off, timing off) select count(*) from t1 inner join t2 on t1.t2_id=t2.id; QUERY PLAN

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-11-15 Thread David Rowley
On Sun, Nov 16, 2014 at 10:09 AM, Simon Riggs si...@2ndquadrant.com wrote: On 15 October 2014 11:03, David Rowley dgrowle...@gmail.com wrote: The explain analyze from the above query looks like: test=# explain (analyze, costs off, timing off) select count(*) from t1 inner join t2 on

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-15 Thread David Rowley
On Thu, Oct 9, 2014 at 12:40 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-09 00:21:44 +1300, David Rowley wrote: Ok, so I've been hacking away at this for a couple of evenings and I think I have a working prototype finally! Cool! Patch attached. So it seems it's not

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-08 Thread David Rowley
On Tue, Oct 7, 2014 at 3:46 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Oct 6, 2014 at 5:57 AM, David Rowley dgrowle...@gmail.com wrote: Can anyone shed any light on how I might determine where the scan rel is in the tree? I need to find it so I can check if the RangeTblEntry is

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-08 Thread Andres Freund
On 2014-10-09 00:21:44 +1300, David Rowley wrote: Ok, so I've been hacking away at this for a couple of evenings and I think I have a working prototype finally! Cool! So it seems it's not quite as efficient as join removal at planning time, but still a big win when it's possible to perform

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-06 Thread David Rowley
On Wed, Oct 1, 2014 at 1:34 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-01 01:03:35 +1300, David Rowley wrote: On Wed, Oct 1, 2014 at 12:01 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-09-30 23:25:45 +1300, David Rowley wrote: I've not quite gotten my

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-06 Thread Robert Haas
On Mon, Oct 6, 2014 at 5:57 AM, David Rowley dgrowle...@gmail.com wrote: Hm, right. But that doesn't seem like a fatal problem to me. The planner knows about t1/t2 and Seq(t1), Seq(t2), not just Hash(Seq(t2)). So it can tell the HashJoin node that when the 'shortcut' qualifier is true, it

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-06 Thread Andres Freund
On 2014-10-06 10:46:09 -0400, Robert Haas wrote: This seems messy, though. Can't the deferred trigger queue become non-empty at pretty much any point in time? At exactly what point are we making this decision, and how do we know the correct answer can't change after that point? What we've

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-06 Thread Robert Haas
On Mon, Oct 6, 2014 at 10:59 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-06 10:46:09 -0400, Robert Haas wrote: This seems messy, though. Can't the deferred trigger queue become non-empty at pretty much any point in time? At exactly what point are we making this decision, and

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-30 Thread David Rowley
On Tue, Sep 30, 2014 at 12:42 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-09-29 22:42:57 +1300, David Rowley wrote: I've made a change to the patch locally to ignore foreign keys that are marked as deferrable. I have serious doubts about the general usefulness if this is onlyu

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-30 Thread Andres Freund
On 2014-09-30 23:25:45 +1300, David Rowley wrote: On Tue, Sep 30, 2014 at 12:42 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-09-29 22:42:57 +1300, David Rowley wrote: I've made a change to the patch locally to ignore foreign keys that are marked as deferrable. I have

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-30 Thread Andres Freund
On 2014-10-01 01:03:35 +1300, David Rowley wrote: On Wed, Oct 1, 2014 at 12:01 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-09-30 23:25:45 +1300, David Rowley wrote: I've not quite gotten my head around how we might stop the unneeded relation from being the primary path

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-29 Thread David Rowley
On Mon, Sep 29, 2014 at 2:41 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-09-28 17:32:21 +1300, David Rowley wrote: My understanding of foreign keys is that any pending foreign key triggers will be executed just before the query completes, so we should only ever encounter

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-29 Thread Andres Freund
On 2014-09-29 22:42:57 +1300, David Rowley wrote: On Mon, Sep 29, 2014 at 2:41 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-09-28 17:32:21 +1300, David Rowley wrote: My understanding of foreign keys is that any pending foreign key triggers will be executed just before the

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-29 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-09-28 10:41:56 -0400, Tom Lane wrote: If this optimization only works in that scenario, it's dead in the water, because that assumption is unsupportable. The planner does not in general use the same query snapshot as the executor, so even

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-29 Thread Andres Freund
On 2014-09-29 10:12:25 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-09-28 10:41:56 -0400, Tom Lane wrote: If this optimization only works in that scenario, it's dead in the water, because that assumption is unsupportable. The planner does not in general use

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-29 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-09-29 10:12:25 -0400, Tom Lane wrote: I see. So why aren't we simply ignoring deferrable FKs when making the optimization? That pushes it back from depending on execution-time state (unsafe) to depending on table DDL (safe). IIRC there's

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-28 Thread Andres Freund
On 2014-09-28 17:32:21 +1300, David Rowley wrote: My understanding of foreign keys is that any pending foreign key triggers will be executed just before the query completes, so we should only ever encounter pending foreign key triggers during planning when we're planning a query that's being

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-28 Thread Tom Lane
David Rowley dgrowle...@gmail.com writes: Please correct anything that sounds wrong here, but my understanding is that we'll always plan a query right before we execute it, with the exception of PREPARE statements where PostgreSQL will cache the query plan when the prepare statement is first

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-27 Thread David Rowley
On Fri, Sep 26, 2014 at 12:36 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 09/16/2014 01:20 PM, David Rowley wrote: + /* +* We mustn't allow any joins to be removed if there are any pending +* foreign key triggers in the queue. This could happen if we are

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-25 Thread Heikki Linnakangas
On 09/16/2014 01:20 PM, David Rowley wrote: + /* +* We mustn't allow any joins to be removed if there are any pending +* foreign key triggers in the queue. This could happen if we are planning +* a query that has been executed from within a volatile function and

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-16 Thread David Rowley
On Sat, Sep 13, 2014 at 1:38 AM, Tom Lane t...@sss.pgh.pa.us wrote: David Rowley dgrowle...@gmail.com writes: On Fri, Sep 12, 2014 at 3:35 AM, Robert Haas robertmh...@gmail.com wrote: I haven't read the patch, but I think the question is why this needs to be different than what we do for

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-12 Thread David Rowley
On Fri, Sep 12, 2014 at 3:35 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Sep 11, 2014 at 7:14 AM, David Rowley dgrowle...@gmail.com wrote: 1. I don't think that I'm currently handling removing eclass members properly. So far the code just removes the Vars that belong to the

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-12 Thread David Rowley
On Fri, Sep 12, 2014 at 3:47 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Thu, Sep 11, 2014 at 7:14 AM, David Rowley dgrowle...@gmail.com wrote: 5. I've added a flag to pg_class called relhasfkey. Currently this gets set to true when a foreign key

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-12 Thread Tom Lane
David Rowley dgrowle...@gmail.com writes: On Fri, Sep 12, 2014 at 3:35 AM, Robert Haas robertmh...@gmail.com wrote: I haven't read the patch, but I think the question is why this needs to be different than what we do for left join removal. I discovered over here -

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-11 Thread David Rowley
On Thu, Aug 28, 2014 at 6:23 AM, Tom Lane t...@sss.pgh.pa.us wrote: If the majority of the added code is code that will be needed for less-bogus optimizations, it might be all right; but I'd kind of want to see the less-bogus optimizations working first. That seems fair. Likely there'd be

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-11 Thread Robert Haas
On Thu, Sep 11, 2014 at 7:14 AM, David Rowley dgrowle...@gmail.com wrote: Here's a quick demo, of the patch at work: test=# create table c (id int primary key); CREATE TABLE test=# create table b (id int primary key, c_id int not null references c(id)); CREATE TABLE test=# create table a

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-11 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, Sep 11, 2014 at 7:14 AM, David Rowley dgrowle...@gmail.com wrote: 5. I've added a flag to pg_class called relhasfkey. Currently this gets set to true when a foreign key is added, though I've added nothing to set it back to false again. I notice

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-27 Thread David Rowley
On Wed, Aug 27, 2014 at 1:40 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 08/26/2014 03:28 PM, David Rowley wrote: Any ideas or feedback on this would be welcome Before someone spends time reviewing this patch, are you sure this is worth the effort? It seems like very narrow

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-27 Thread Jim Nasby
On 8/26/14, 8:40 AM, Heikki Linnakangas wrote: Just so everyone is on the same page on what kind of queries this helps with, here are some examples from the added regression tests: -- Test join removals for semi and anti joins CREATE TEMP TABLE b (id INT NOT NULL PRIMARY KEY, val INT); CREATE

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-27 Thread Tom Lane
Jim Nasby j...@nasby.net writes: On 8/26/14, 8:40 AM, Heikki Linnakangas wrote: Just so everyone is on the same page on what kind of queries this helps with, here are some examples from the added regression tests: -- Test join removals for semi and anti joins CREATE TEMP TABLE b (id INT

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-26 Thread David Rowley
On Tue, Aug 5, 2014 at 10:35 PM, David Rowley dgrowle...@gmail.com wrote: Currently most of my changes are in analyzejoin.c, but I did also have to make changes to load the foreign key constraints so that they were available to the planner. One thing that is currently lacking, which would

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-26 Thread Heikki Linnakangas
On 08/26/2014 03:28 PM, David Rowley wrote: Any ideas or feedback on this would be welcome Before someone spends time reviewing this patch, are you sure this is worth the effort? It seems like very narrow use case to me. I understand removing LEFT and INNER joins, but the case for SEMI and

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-17 Thread David Rowley
On Sun, Aug 10, 2014 at 11:48 PM, David Rowley dgrowle...@gmail.com wrote: I've attached an updated version of the patch which fixes up some incorrect logic in the foreign key matching code, plus various comment improvements. I've made a few updated to the patch to simplify some logic in

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-10 Thread David Rowley
On Tue, Aug 5, 2014 at 10:35 PM, David Rowley dgrowle...@gmail.com wrote: The patch (attached) is also now able to detect when a NOT EXISTS clause cannot produce any records at all. I've attached an updated version of the patch which fixes up some incorrect logic in the foreign key matching

[HACKERS] Patch to support SEMI and ANTI join removal

2014-08-05 Thread David Rowley
I've been working away at allowing semi and anti joins to be added to the list of join types that our join removal code supports. The basic idea is that we can removal a semi or anti join if the left hand relation references the relation that's being semi/anti joined if the join condition matches