Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-28 Thread James Coleman
On Sat, Mar 28, 2020 at 2:54 PM Tomas Vondra wrote: > ... > >> >> >9. optimizer/path/allpaths.c get_useful_pathkeys_for_relation: > >> >> >* Considering query_pathkeys is always worth it, because it might let > >> >> >us > >> >> >* avoid a local sort. > >> >> > > >> >> >That originally was a

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-28 Thread Tomas Vondra
On Sat, Mar 28, 2020 at 10:19:04AM -0400, James Coleman wrote: On Fri, Mar 27, 2020 at 10:58 PM Tomas Vondra wrote: ... >> As a side note here, I'm wondering if this (determining useful pathkeys) >> can be made a bit smarter by looking both at query_pathkeys and pathkeys >> useful for merging,

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-28 Thread James Coleman
On Fri, Mar 27, 2020 at 10:58 PM Tomas Vondra wrote: > ... > >> As a side note here, I'm wondering if this (determining useful pathkeys) > >> can be made a bit smarter by looking both at query_pathkeys and pathkeys > >> useful for merging, similarly to what truncate_useless_pathkeys() does. > >>

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-27 Thread Tomas Vondra
On Fri, Mar 27, 2020 at 09:36:55PM -0400, James Coleman wrote: On Fri, Mar 27, 2020 at 9:19 PM Tomas Vondra wrote: On Fri, Mar 27, 2020 at 12:51:34PM -0400, James Coleman wrote: >In a previous email I'd summarized remaining TODOs I'd found. Here's >an updated listed with several resolved. >

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-27 Thread Tomas Vondra
On Fri, Mar 27, 2020 at 12:51:34PM -0400, James Coleman wrote: In a previous email I'd summarized remaining TODOs I'd found. Here's an updated listed with several resolved. Resolved: 2. Not marked in the patch, but in nodeIncrementalSort.c ExecIncrementalSort() I wonder if perhaps we should

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-24 Thread James Coleman
On Tue, Mar 24, 2020 at 8:23 PM James Coleman wrote: > While working on finding a test case to show rescan isn't implemented > properly yet, I came across a bug. At the top of > ExecInitIncrementalSort, we assert that eflags does not contain > EXEC_FLAG_REWIND. But the following query (with merge

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-24 Thread James Coleman
On Tue, Mar 24, 2020 at 7:08 PM Tomas Vondra wrote: > > On Tue, Mar 24, 2020 at 06:26:11PM -0400, James Coleman wrote: > >On Mon, Mar 23, 2020 at 11:44 PM Alvaro Herrera > > wrote: > >> > >> On 2020-Mar-23, James Coleman wrote: > >> > >> > 4. nodeIncrementalSort.c ExecReScanIncrementalSort: This

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-24 Thread Tomas Vondra
On Tue, Mar 24, 2020 at 06:26:11PM -0400, James Coleman wrote: On Mon, Mar 23, 2020 at 11:44 PM Alvaro Herrera wrote: On 2020-Mar-23, James Coleman wrote: > 4. nodeIncrementalSort.c ExecReScanIncrementalSort: This whole chunk > is suspect. I've mentioned previously I don't have a great

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-24 Thread James Coleman
On Mon, Mar 23, 2020 at 11:44 PM Alvaro Herrera wrote: > > On 2020-Mar-23, James Coleman wrote: > > > 4. nodeIncrementalSort.c ExecReScanIncrementalSort: This whole chunk > > is suspect. I've mentioned previously I don't have a great mental > > model of how rescan works and its invariants (IIRC

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-23 Thread Alvaro Herrera
On 2020-Mar-23, James Coleman wrote: > 4. nodeIncrementalSort.c ExecReScanIncrementalSort: This whole chunk > is suspect. I've mentioned previously I don't have a great mental > model of how rescan works and its invariants (IIRC someone said it was > about moving around a result set in a cursor).

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-23 Thread James Coleman
On Mon, Mar 23, 2020 at 1:05 PM Tom Lane wrote: > > Alvaro Herrera writes: > > ... all plan types that use only one child use the outer one. They > > could use either, as long as it does that consistently, I think. > > Yeah, exactly. The outer/inner terminology is really only sensible > for

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-23 Thread Alvaro Herrera
On 2020-Mar-22, James Coleman wrote: > One question I have while I work on that: I've noticed some confusion > in the patch as to whether we should refer to the node below the > incremental sort node in the plan tree (i.e., the node we get tuples > from) as the inner node or the outer node.

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-22 Thread Tomas Vondra
On Sun, Mar 22, 2020 at 10:05:50PM -0400, James Coleman wrote: On Sun, Mar 22, 2020 at 8:54 PM Andreas Karlsson wrote: On 3/23/20 1:33 AM, James Coleman wrote: > So on the face of it we have a bit of a no-win situation. The function > tuple_sort_method_name returns a const, but lappend wants

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-22 Thread James Coleman
On Fri, Mar 20, 2020 at 8:56 PM Tomas Vondra wrote: > > Hi, > > I've looked at v38 but it seems it's a bit broken by some recent explain > changes (mostly missing type in declarations). Attached is v39 fixing > those issues, and including a bunch of fixes based on a review - most of > the changes

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-22 Thread James Coleman
On Sun, Mar 22, 2020 at 8:54 PM Andreas Karlsson wrote: > > On 3/23/20 1:33 AM, James Coleman wrote: > > So on the face of it we have a bit of a no-win situation. The function > > tuple_sort_method_name returns a const, but lappend wants a non-const. > > I'm not sure what the project style

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-22 Thread Andreas Karlsson
On 3/23/20 1:33 AM, James Coleman wrote: So on the face of it we have a bit of a no-win situation. The function tuple_sort_method_name returns a const, but lappend wants a non-const. I'm not sure what the project style preference is here: we could cast the result as (char *) to drop the const

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-22 Thread James Coleman
On Sun, Mar 22, 2020 at 6:02 PM Andreas Karlsson wrote: > > On 3/21/20 1:56 AM, Tomas Vondra wrote: > > I've looked at v38 but it seems it's a bit broken by some recent explain > > changes (mostly missing type in declarations). Attached is v39 fixing > > those issues, and including a bunch of

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-22 Thread Andreas Karlsson
On 3/21/20 1:56 AM, Tomas Vondra wrote: I've looked at v38 but it seems it's a bit broken by some recent explain changes (mostly missing type in declarations). Attached is v39 fixing those issues, and including a bunch of fixes based on a review - most of the changes is in comments, so I've

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-15 Thread James Coleman
On Fri, Mar 13, 2020 at 4:22 PM Tom Lane wrote: > > Alvaro Herrera writes: > > Also, I wonder if it would be better to modify our policies so that we > > update typedefs.list more frequently. Some people include additions > > with their commits, but it's far from SOP. > > Perhaps. My own

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-15 Thread James Coleman
On Sat, Mar 14, 2020 at 10:55 PM James Coleman wrote: > > On Fri, Mar 13, 2020 at 1:06 PM James Coleman wrote: > > > > On Thu, Mar 12, 2020 at 5:53 PM Alvaro Herrera > > wrote: > > > > > > I gave this a very quick look; I don't claim to understand it or > > > anything, but I thought these

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-14 Thread James Coleman
On Fri, Mar 13, 2020 at 1:06 PM James Coleman wrote: > > On Thu, Mar 12, 2020 at 5:53 PM Alvaro Herrera > wrote: > > > > I gave this a very quick look; I don't claim to understand it or > > anything, but I thought these trivial cleanups worthwhile. The only > > non-cosmetic thing is changing

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-14 Thread Tomas Vondra
On Sat, Mar 14, 2020 at 02:41:09PM -0400, James Coleman wrote: It looks like the issue is actually into the `tuplecontext`, which is currently a child context of `sortcontext`: #3 0x558cd153b565 in AllocSetCheck (context=context@entry=0x558cd28e0b70) at aset.c:1573 1573

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-14 Thread James Coleman
On Sat, Mar 14, 2020 at 12:36 PM James Coleman wrote: > > On Sat, Mar 14, 2020 at 12:24 PM James Coleman wrote: > > > > On Sat, Mar 14, 2020 at 12:07 PM James Coleman wrote: > > > > > > On Fri, Mar 13, 2020 at 8:23 PM James Coleman wrote: > > > > > > > > On Friday, March 13, 2020, Tomas Vondra

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-14 Thread James Coleman
On Sat, Mar 14, 2020 at 12:24 PM James Coleman wrote: > > On Sat, Mar 14, 2020 at 12:07 PM James Coleman wrote: > > > > On Fri, Mar 13, 2020 at 8:23 PM James Coleman wrote: > > > > > > On Friday, March 13, 2020, Tomas Vondra > > > wrote: > > >> > > >> On Fri, Mar 13, 2020 at 04:31:16PM -0400,

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-14 Thread James Coleman
On Sat, Mar 14, 2020 at 12:07 PM James Coleman wrote: > > On Fri, Mar 13, 2020 at 8:23 PM James Coleman wrote: > > > > On Friday, March 13, 2020, Tomas Vondra > > wrote: > >> > >> On Fri, Mar 13, 2020 at 04:31:16PM -0400, James Coleman wrote: > >>> > >>> On Fri, Mar 13, 2020 at 2:23 PM James

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-14 Thread James Coleman
On Fri, Mar 13, 2020 at 8:23 PM James Coleman wrote: > > On Friday, March 13, 2020, Tomas Vondra wrote: >> >> On Fri, Mar 13, 2020 at 04:31:16PM -0400, James Coleman wrote: >>> >>> On Fri, Mar 13, 2020 at 2:23 PM James Coleman wrote: On Tue, Mar 10, 2020 at 10:44 PM Tomas Vondra

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-13 Thread James Coleman
On Friday, March 13, 2020, Tomas Vondra wrote: > On Fri, Mar 13, 2020 at 04:31:16PM -0400, James Coleman wrote: > >> On Fri, Mar 13, 2020 at 2:23 PM James Coleman wrote: >> >>> >>> On Tue, Mar 10, 2020 at 10:44 PM Tomas Vondra >>> wrote: >>> > 3) Most of the execution plans look reasonable,

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-13 Thread Tomas Vondra
On Fri, Mar 13, 2020 at 04:31:16PM -0400, James Coleman wrote: On Fri, Mar 13, 2020 at 2:23 PM James Coleman wrote: On Tue, Mar 10, 2020 at 10:44 PM Tomas Vondra wrote: > 3) Most of the execution plans look reasonable, except that some of the > plans look like this: > > >

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-13 Thread James Coleman
On Fri, Mar 13, 2020 at 2:23 PM James Coleman wrote: > > On Tue, Mar 10, 2020 at 10:44 PM Tomas Vondra > wrote: > > 3) Most of the execution plans look reasonable, except that some of the > > plans look like this: > > > > > > QUERY PLAN > >

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-13 Thread Tom Lane
Alvaro Herrera writes: > Also, I wonder if it would be better to modify our policies so that we > update typedefs.list more frequently. Some people include additions > with their commits, but it's far from SOP. Perhaps. My own workflow includes pulling down a fresh typedefs.list from the

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-13 Thread Alvaro Herrera
On 2020-Mar-13, Tom Lane wrote: > Alvaro Herrera writes: > > ... You can specify a filelist to pgindent, also. What I do is super > > low-tech: do a "git diff origin/master", copy the filelist, and then > > ^V^E to paste that list into a command line to run pgindent (editing to > > remove the

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-13 Thread Tom Lane
Alvaro Herrera writes: > ... You can specify a filelist to pgindent, also. What I do is super > low-tech: do a "git diff origin/master", copy the filelist, and then > ^V^E to paste that list into a command line to run pgindent (editing to > remove the change histogram and irrelevant files). I

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-13 Thread Alvaro Herrera
On 2020-Mar-13, James Coleman wrote: > > I don't propose to commit 0003 of course, since it's not our policy; > > that's just to allow running pgindent sanely, which gives you 0004 > > (though my local pgindent has an unrelated fix). And after that you > > notice the issue that 0005 fixes. > >

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-13 Thread James Coleman
On Tue, Mar 10, 2020 at 10:44 PM Tomas Vondra wrote: > 3) Most of the execution plans look reasonable, except that some of the > plans look like this: > > > QUERY PLAN >- > Limit > -> GroupAggregate >

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-13 Thread Tomas Vondra
On Fri, Mar 13, 2020 at 01:16:44PM -0400, James Coleman wrote: On Tue, Mar 10, 2020 at 10:44 PM Tomas Vondra wrote: ... Now, a couple comments about parts 0001 - 0003 of the patch ... 1) I see a bunch of failures in the regression test, due to minor differences in the explain output. All the

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-13 Thread Andres Freund
Hi, On 2020-03-13 13:36:44 -0400, Tom Lane wrote: > James Coleman writes: > > On Tue, Mar 10, 2020 at 10:44 PM Tomas Vondra > > wrote: > >> 1) I see a bunch of failures in the regression test, due to minor > >> differences in the explain output. All the differences are about minor > >> changes

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-13 Thread Tom Lane
James Coleman writes: > On Tue, Mar 10, 2020 at 10:44 PM Tomas Vondra > wrote: >> 1) I see a bunch of failures in the regression test, due to minor >> differences in the explain output. All the differences are about minor >> changes in memory usage, like this: >> >> - "Sort Space

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-12 Thread Justin Pryzby
Thanks for working on this. I have some minor comments. In 0005: + /* Restore the input path (we might have addes Sort on top). */ => added? There's at least two more of the same typo. + /* also ignore already sorted paths */ =>

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-12 Thread Alvaro Herrera
I gave this a very quick look; I don't claim to understand it or anything, but I thought these trivial cleanups worthwhile. The only non-cosmetic thing is changing order of arguments to the SOn_printf() calls in 0008; I think they are contrary to what the comment says. I don't propose to commit

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-07 Thread James Coleman
On Thu, Mar 5, 2020 at 6:45 PM Tom Lane wrote: > James Coleman writes: > > That's what I figured, but as I mentioned I've having trouble figuring > out > > how the fact that an analyze is in flight is determined. I assume it's > > something that lives of the EState or similar, but I'm not

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-05 Thread Tom Lane
James Coleman writes: > That's what I figured, but as I mentioned I've having trouble figuring out > how the fact that an analyze is in flight is determined. I assume it's > something that lives of the EState or similar, but I'm not seeing anything > obvious. AFAIR, it's just whether or not the

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-05 Thread James Coleman
On Thu, Mar 5, 2020 at 5:53 PM Tom Lane wrote: > James Coleman writes: > > I'm looking at this now, and realized that at least for parallel plans > the > > current patch tracks the tuplesort instrumentation whether or not an > > EXPLAIN ANALYZE is in process. > > > Is this fairly standard for

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-05 Thread Tom Lane
James Coleman writes: > I'm looking at this now, and realized that at least for parallel plans the > current patch tracks the tuplesort instrumentation whether or not an > EXPLAIN ANALYZE is in process. > Is this fairly standard for executor nodes? Or is it expected to condition > some of this

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-05 Thread James Coleman
On Tue, Jan 21, 2020 at 9:37 AM James Coleman wrote: > That being said, the patch also needs some more work on improving > EXPLAIN ANALYZE output (perhaps min/max/mean or median of > memory usage number of groups in each sort mode), and I think it's far > more feasible that I can tackle that

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-01-21 Thread James Coleman
On Tue, Jan 21, 2020 at 9:58 AM Tomas Vondra wrote: > > On Tue, Jan 21, 2020 at 09:37:01AM -0500, James Coleman wrote: > >On Tue, Jan 21, 2020 at 9:25 AM Tomas Vondra > > wrote: > >> > >> Hi, > >> > >> This patch has been marked as WoA since end of November, and there has > >> been no

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-01-21 Thread Tomas Vondra
On Tue, Jan 21, 2020 at 09:37:01AM -0500, James Coleman wrote: On Tue, Jan 21, 2020 at 9:25 AM Tomas Vondra wrote: Hi, This patch has been marked as WoA since end of November, and there has been no discussion/reviews since then :-( Based on off-list discussion with James I don't think that's

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-01-21 Thread James Coleman
On Tue, Jan 21, 2020 at 9:25 AM Tomas Vondra wrote: > > Hi, > > This patch has been marked as WoA since end of November, and there has > been no discussion/reviews since then :-( Based on off-list discussion > with James I don't think that's going to change in this CF, so I'll move > it to the

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-01-21 Thread Tomas Vondra
Hi, This patch has been marked as WoA since end of November, and there has been no discussion/reviews since then :-( Based on off-list discussion with James I don't think that's going to change in this CF, so I'll move it to the next CF. I plan to work on the planner part of this patch before

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-11-30 Thread Tomas Vondra
On Fri, Nov 29, 2019 at 03:01:46PM +0900, Michael Paquier wrote: On Sun, Sep 29, 2019 at 01:00:49AM +0200, Tomas Vondra wrote: OK. I'll try extending the set of synthetic queries in [1] to also do soemthing like this and generate similar plans. Any progress on that? Please note that the

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-11-28 Thread Michael Paquier
On Sun, Sep 29, 2019 at 01:00:49AM +0200, Tomas Vondra wrote: > OK. I'll try extending the set of synthetic queries in [1] to also do > soemthing like this and generate similar plans. Any progress on that? Please note that the latest patch does not apply anymore, so a rebase is needed. I am

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-28 Thread Tomas Vondra
On Fri, Sep 27, 2019 at 01:50:30PM -0400, James Coleman wrote: On Mon, Sep 9, 2019 at 5:55 PM Tomas Vondra wrote: The "patched" column means all developer GUCs disabled, so it's expected to produce the same plan as master (and it is). And then there's one column for each developer GUC. If the

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-28 Thread Tomas Vondra
On Fri, Sep 27, 2019 at 08:31:30PM -0400, James Coleman wrote: On Fri, Sep 13, 2019 at 10:51 AM Tomas Vondra wrote: On Thu, Sep 12, 2019 at 12:49:29PM -0300, Alvaro Herrera wrote: >On 2019-Jul-30, Tomas Vondra wrote: >> I've decided to do a couple of experiments, trying to make my mind about

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-27 Thread James Coleman
On Mon, Sep 16, 2019 at 6:32 AM Tomas Vondra wrote: > > On Sun, Sep 15, 2019 at 09:33:33PM -0400, James Coleman wrote: > >On Sat, Jul 20, 2019 at 11:25 AM Tomas Vondra > > wrote: > >> >> ... > >> >> > >> >> I think this may be a thinko, as this plan demonstrates - but I'm not > >> >> sure about

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-27 Thread James Coleman
On Mon, Sep 9, 2019 at 5:55 PM Tomas Vondra wrote: > The "patched" column means all developer GUCs disabled, so it's expected > to produce the same plan as master (and it is). And then there's one > column for each developer GUC. If the column is just TRUE it means the > GUC does not affect any

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-16 Thread Tomas Vondra
On Sun, Sep 15, 2019 at 09:33:33PM -0400, James Coleman wrote: On Sat, Jul 20, 2019 at 11:25 AM Tomas Vondra wrote: >> ... >> >> I think this may be a thinko, as this plan demonstrates - but I'm not >> sure about it. I wonder if this might be penalizing some other types of >> plans

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-13 Thread James Coleman
On Fri, Sep 13, 2019 at 10:54 AM Tomas Vondra wrote: > > On Thu, Sep 12, 2019 at 11:54:06AM -0400, James Coleman wrote: > >> OK, so we have that now. I suppose this spreadsheet now tells us which > >> places are useful and which aren't, at least for the queries that you've > >> tested. Dowe

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-13 Thread Tomas Vondra
On Thu, Sep 12, 2019 at 11:54:06AM -0400, James Coleman wrote: OK, so we have that now. I suppose this spreadsheet now tells us which places are useful and which aren't, at least for the queries that you've tested. Dowe that mean that we want to get the patch to consider adding paths only the

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-13 Thread Tomas Vondra
On Thu, Sep 12, 2019 at 12:49:29PM -0300, Alvaro Herrera wrote: On 2019-Jul-30, Tomas Vondra wrote: On Sun, Jul 21, 2019 at 01:34:22PM +0200, Tomas Vondra wrote: > > I wonder if we're approaching this wrong. Maybe we should not reverse > engineer queries for the various places, but just start

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-12 Thread James Coleman
> OK, so we have that now. I suppose this spreadsheet now tells us which > places are useful and which aren't, at least for the queries that you've > tested. Dowe that mean that we want to get the patch to consider adding > paths only the places that your spreadsheet says are useful? I'm not >

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-09 Thread Tomas Vondra
On Wed, Sep 04, 2019 at 09:17:10PM +0200, Tomas Vondra wrote: On Wed, Sep 04, 2019 at 11:37:48AM +0200, Rafia Sabih wrote: On Tue, 30 Jul 2019 at 02:17, Tomas Vondra wrote: On Sun, Jul 21, 2019 at 01:34:22PM +0200, Tomas Vondra wrote: ... I wonder if we're approaching this wrong. Maybe we

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-04 Thread Tomas Vondra
On Wed, Sep 04, 2019 at 11:37:48AM +0200, Rafia Sabih wrote: On Tue, 30 Jul 2019 at 02:17, Tomas Vondra wrote: On Sun, Jul 21, 2019 at 01:34:22PM +0200, Tomas Vondra wrote: > > ... > >I wonder if we're approaching this wrong. Maybe we should not reverse >engineer queries for the various

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-04 Thread Rafia Sabih
On Tue, 30 Jul 2019 at 02:17, Tomas Vondra wrote: > On Sun, Jul 21, 2019 at 01:34:22PM +0200, Tomas Vondra wrote: > > > > ... > > > >I wonder if we're approaching this wrong. Maybe we should not reverse > >engineer queries for the various places, but just start with a set of > >queries that we

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-29 Thread Tomas Vondra
On Sun, Jul 21, 2019 at 01:34:22PM +0200, Tomas Vondra wrote: ... I wonder if we're approaching this wrong. Maybe we should not reverse engineer queries for the various places, but just start with a set of queries that we want to optimize, and then identify which places in the planner need to

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-21 Thread Tomas Vondra
On Sat, Jul 20, 2019 at 07:37:08PM -0400, James Coleman wrote: .. Yes, you're right - an extra sort node would be necessary. But I don't think that changes the idea behind that example. The question is whether the extra sorts below the gather would be cheaper than doing a large sort on top of

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-20 Thread James Coleman
On Sat, Jul 20, 2019 at 1:00 PM Tomas Vondra wrote: > > On Sat, Jul 20, 2019 at 12:21:01PM -0400, James Coleman wrote: > >On Sat, Jul 20, 2019 at 12:12 PM James Coleman wrote: > >> > >> On Sat, Jul 20, 2019 at 11:25 AM Tomas Vondra > >> wrote: > >> ... > >> > >My current line of investigation

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-20 Thread Tomas Vondra
On Sat, Jul 20, 2019 at 12:21:01PM -0400, James Coleman wrote: On Sat, Jul 20, 2019 at 12:12 PM James Coleman wrote: On Sat, Jul 20, 2019 at 11:25 AM Tomas Vondra wrote: ... > >My current line of investigation is whether we need to do anything in > >the parallel portion of

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-20 Thread James Coleman
On Sat, Jul 20, 2019 at 12:12 PM James Coleman wrote: > > On Sat, Jul 20, 2019 at 11:25 AM Tomas Vondra > wrote: > ... > > >My current line of investigation is whether we need to do anything in > > >the parallel portion of create_ordered_paths(). I noticed that the > > >first-pass patch adding

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-20 Thread James Coleman
On Sat, Jul 20, 2019 at 11:25 AM Tomas Vondra wrote: ... > >My current line of investigation is whether we need to do anything in > >the parallel portion of create_ordered_paths(). I noticed that the > >first-pass patch adding generate_useful_gather_paths() modified that > >section but wasn't

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-20 Thread Tomas Vondra
On Sat, Jul 20, 2019 at 10:33:02AM -0400, James Coleman wrote: On Sat, Jul 20, 2019 at 9:22 AM Tomas Vondra wrote: On Fri, Jul 19, 2019 at 04:59:21PM -0400, James Coleman wrote: >On Mon, Jul 8, 2019 at 9:37 PM Tomas Vondra > wrote: >> Now, consider this example: >> >> create table t (a int,

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-20 Thread Tomas Vondra
On Fri, Jul 19, 2019 at 04:59:21PM -0400, James Coleman wrote: On Mon, Jul 8, 2019 at 9:37 PM Tomas Vondra wrote: Now, consider this example: create table t (a int, b int, c int); insert into t select mod(i,100),mod(i,100),i from generate_series(1,1000) s(i); create index on t (a);

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-19 Thread James Coleman
On Mon, Jul 8, 2019 at 9:37 PM Tomas Vondra wrote: > Now, consider this example: > > create table t (a int, b int, c int); > insert into t select mod(i,100),mod(i,100),i from > generate_series(1,1000) s(i); > create index on t (a); > analyze t; > explain select a,b,sum(c) from t

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-15 Thread Tomas Vondra
On Mon, Jul 15, 2019 at 09:25:32AM -0400, James Coleman wrote: On Sun, Jul 14, 2019 at 10:16 PM Tomas Vondra wrote: >> >> Attached is a slightly modified patch series: >> >> >> >> 1) 0001 considers incremental sort paths in various places (adds the new >> >> generate_useful_gather_paths and

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-15 Thread James Coleman
On Sun, Jul 14, 2019 at 10:16 PM Tomas Vondra wrote: > >> >> Attached is a slightly modified patch series: > >> >> > >> >> 1) 0001 considers incremental sort paths in various places (adds the new > >> >> generate_useful_gather_paths and modifies places calling > >> >> create_sort_path) > >> > >

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-14 Thread Tomas Vondra
On Sun, Jul 14, 2019 at 02:38:40PM -0400, James Coleman wrote: On Tue, Jul 9, 2019 at 10:54 AM Tomas Vondra wrote: On Tue, Jul 09, 2019 at 09:28:42AM -0400, James Coleman wrote: >On Mon, Jul 8, 2019 at 9:37 PM Tomas Vondra > wrote: >> >> On Mon, Jul 08, 2019 at 12:07:06PM -0400, James Coleman

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-14 Thread James Coleman
On Tue, Jul 9, 2019 at 10:54 AM Tomas Vondra wrote: > > On Tue, Jul 09, 2019 at 09:28:42AM -0400, James Coleman wrote: > >On Mon, Jul 8, 2019 at 9:37 PM Tomas Vondra > > wrote: > >> > >> On Mon, Jul 08, 2019 at 12:07:06PM -0400, James Coleman wrote: > >> > ... > >> > > >> >I guess I'm still not

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-09 Thread Tomas Vondra
On Tue, Jul 09, 2019 at 09:28:42AM -0400, James Coleman wrote: On Mon, Jul 8, 2019 at 9:37 PM Tomas Vondra wrote: On Mon, Jul 08, 2019 at 12:07:06PM -0400, James Coleman wrote: > ... > >I guess I'm still not following. If (2) is responsible (currently) for >adding an explicit sort, why

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-09 Thread James Coleman
On Mon, Jul 8, 2019 at 6:37 PM Alexander Korotkov wrote: > > On Thu, Jul 4, 2019 at 4:25 PM James Coleman wrote: > > Process questions: > > - Do I need to explicitly move the patch somehow to the next CF? > > We didn't manage to register it on current (July) commitfest. So, > please, register

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-09 Thread James Coleman
On Mon, Jul 8, 2019 at 9:37 PM Tomas Vondra wrote: > > On Mon, Jul 08, 2019 at 12:07:06PM -0400, James Coleman wrote: > >On Mon, Jul 8, 2019 at 10:58 AM Tomas Vondra > > wrote: > >> > >> On Mon, Jul 08, 2019 at 10:32:18AM -0400, James Coleman wrote: > >> >On Mon, Jul 8, 2019 at 9:59 AM Tomas

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-09 Thread Tomas Vondra
On Tue, Jul 09, 2019 at 03:37:03AM +0200, Tomas Vondra wrote: ... Notice that cost of the second plan is almost double the first one. That means 0004 does not even generate the first plan, i.e. there are cases where we don't try to add the explicit sort before passing the path to

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread James Coleman
Note: As I was writing this, I saw a new email come in from Tomas with a new patch series, and some similar observations. I'll look at that patch series more, but I think it's likely far more complete, so will end up going with that. I wanted to send this email anyway to at least capture the

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread Tomas Vondra
On Mon, Jul 08, 2019 at 12:07:06PM -0400, James Coleman wrote: On Mon, Jul 8, 2019 at 10:58 AM Tomas Vondra wrote: On Mon, Jul 08, 2019 at 10:32:18AM -0400, James Coleman wrote: >On Mon, Jul 8, 2019 at 9:59 AM Tomas Vondra > wrote: >> >> On Mon, Jul 08, 2019 at 09:22:39AM -0400, James Coleman

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread Alexander Korotkov
On Thu, Jul 4, 2019 at 4:25 PM James Coleman wrote: > Process questions: > - Do I need to explicitly move the patch somehow to the next CF? We didn't manage to register it on current (July) commitfest. So, please, register it on next (September) commitfest. > - Since I've basically taken over

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread Alexander Korotkov
On Mon, Jun 3, 2019 at 12:18 AM Tomas Vondra wrote: > For a moment I thought we could/should look at the histogram, becase that > could tell us if there are groups "before" the first MCV one, but I don't > think we should do that, for two reasons. Firstly, rare values may not get > to the

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread James Coleman
On Mon, Jul 8, 2019 at 10:58 AM Tomas Vondra wrote: > > On Mon, Jul 08, 2019 at 10:32:18AM -0400, James Coleman wrote: > >On Mon, Jul 8, 2019 at 9:59 AM Tomas Vondra > > wrote: > >> > >> On Mon, Jul 08, 2019 at 09:22:39AM -0400, James Coleman wrote: > >> >On Sun, Jul 7, 2019 at 5:02 PM Tomas

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread Tomas Vondra
On Mon, Jul 08, 2019 at 10:32:18AM -0400, James Coleman wrote: On Mon, Jul 8, 2019 at 9:59 AM Tomas Vondra wrote: On Mon, Jul 08, 2019 at 09:22:39AM -0400, James Coleman wrote: >On Sun, Jul 7, 2019 at 5:02 PM Tomas Vondra > wrote: >> We're running query like this: >> >> SELECT a, sum(b),

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread James Coleman
On Mon, Jul 8, 2019 at 9:59 AM Tomas Vondra wrote: > > On Mon, Jul 08, 2019 at 09:22:39AM -0400, James Coleman wrote: > >On Sun, Jul 7, 2019 at 5:02 PM Tomas Vondra > > wrote: > >> We're running query like this: > >> > >> SELECT a, sum(b), count(*) FROM pagg_tab_ml GROUP BY a HAVING avg(b) < 3

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread Tomas Vondra
On Mon, Jul 08, 2019 at 09:22:39AM -0400, James Coleman wrote: On Sun, Jul 7, 2019 at 5:02 PM Tomas Vondra wrote: We're running query like this: SELECT a, sum(b), count(*) FROM pagg_tab_ml GROUP BY a HAVING avg(b) < 3 ORDER BY 1, 2, 3 but we're trying to add the incremental sort *before*

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread James Coleman
On Sun, Jul 7, 2019 at 5:02 PM Tomas Vondra wrote: > We're running query like this: > > SELECT a, sum(b), count(*) FROM pagg_tab_ml GROUP BY a HAVING avg(b) < 3 > ORDER BY 1, 2, 3 > > but we're trying to add the incremental sort *before* the aggregation, > because the optimizer also considers

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-07 Thread Tomas Vondra
On Sun, Jul 07, 2019 at 09:01:43AM -0400, James Coleman wrote: On Sun, Jul 7, 2019 at 8:34 AM Tomas Vondra wrote: On Thu, Jul 04, 2019 at 09:29:49AM -0400, James Coleman wrote: >On Tue, Jun 25, 2019 at 7:22 PM Tomas Vondra > wrote: >> >> On Tue, Jun 25, 2019 at 04:53:40PM -0400, James Coleman

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-07 Thread James Coleman
On Sun, Jul 7, 2019 at 8:34 AM Tomas Vondra wrote: > > On Thu, Jul 04, 2019 at 09:29:49AM -0400, James Coleman wrote: > >On Tue, Jun 25, 2019 at 7:22 PM Tomas Vondra > > wrote: > >> > >> On Tue, Jun 25, 2019 at 04:53:40PM -0400, James Coleman wrote: > >> > > >> >Unrelated: if you or someone else

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-07 Thread Tomas Vondra
On Thu, Jul 04, 2019 at 09:29:49AM -0400, James Coleman wrote: On Tue, Jun 25, 2019 at 7:22 PM Tomas Vondra wrote: On Tue, Jun 25, 2019 at 04:53:40PM -0400, James Coleman wrote: > >Unrelated: if you or someone else you know that's more familiar with >the parallel code, I'd be interested in

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-04 Thread James Coleman
On Thu, Jul 4, 2019 at 10:46 AM Tomas Vondra wrote: > > On Thu, Jul 04, 2019 at 09:29:49AM -0400, James Coleman wrote: > >On Tue, Jun 25, 2019 at 7:22 PM Tomas Vondra > > wrote: > >> > >> On Tue, Jun 25, 2019 at 04:53:40PM -0400, James Coleman wrote: > >> > > >> >Unrelated: if you or someone else

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-04 Thread Tomas Vondra
On Thu, Jul 04, 2019 at 09:29:49AM -0400, James Coleman wrote: On Tue, Jun 25, 2019 at 7:22 PM Tomas Vondra wrote: On Tue, Jun 25, 2019 at 04:53:40PM -0400, James Coleman wrote: > >Unrelated: if you or someone else you know that's more familiar with >the parallel code, I'd be interested in

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-04 Thread James Coleman
On Tue, Jun 25, 2019 at 7:22 PM Tomas Vondra wrote: > > On Tue, Jun 25, 2019 at 04:53:40PM -0400, James Coleman wrote: > > > >Unrelated: if you or someone else you know that's more familiar with > >the parallel code, I'd be interested in their looking at the patch at > >some point, because I have

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-06-25 Thread Tomas Vondra
On Tue, Jun 25, 2019 at 04:53:40PM -0400, James Coleman wrote: Unrelated: if you or someone else you know that's more familiar with the parallel code, I'd be interested in their looking at the patch at some point, because I have a suspicion it might not be operating in parallel ever (either

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-06-25 Thread James Coleman
On Tue, Jun 25, 2019 at 4:32 PM Tomas Vondra wrote: > > On Tue, Jun 25, 2019 at 12:13:01PM -0700, Peter Geoghegan wrote: > >On Tue, Jun 25, 2019 at 11:03 AM James Coleman wrote: > >> No, I haven't confirmed that it's called less frequently, and I'd be > >> extremely surprised if it were given

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-06-25 Thread Tomas Vondra
On Tue, Jun 25, 2019 at 12:13:01PM -0700, Peter Geoghegan wrote: On Tue, Jun 25, 2019 at 11:03 AM James Coleman wrote: No, I haven't confirmed that it's called less frequently, and I'd be extremely surprised if it were given the diff doesn't suggest any changes to that at all. I must have

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-06-25 Thread Peter Geoghegan
On Tue, Jun 25, 2019 at 11:03 AM James Coleman wrote: > No, I haven't confirmed that it's called less frequently, and I'd be > extremely surprised if it were given the diff doesn't suggest any > changes to that at all. I must have misunderstood, then. I thought that you were suggesting that that

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-06-25 Thread James Coleman
On Tue, Jun 25, 2019 at 1:13 PM Peter Geoghegan wrote: > > On Tue, Jun 25, 2019 at 9:53 AM James Coleman wrote: > > Given the patch contents I don't see any obviously reason why either > > of those possibilities (calling comparetup_heap less often, or it's > > cheaper) are likely. Is that

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-06-25 Thread Peter Geoghegan
On Tue, Jun 25, 2019 at 9:53 AM James Coleman wrote: > Given the patch contents I don't see any obviously reason why either > of those possibilities (calling comparetup_heap less often, or it's > cheaper) are likely. Is that something we should investigate further? > Or is it just a nice happy

<    1   2   3   >