Re: [HACKERS] asynchronous execution

Hello. Fully-asynchronous executor needs that every node is stateful and suspendable at the time of requesting for the next tuples to underneath nodes. I tried pure push-base executor but failed. After the miserable patch upthread, I finally managed to make executor nodes suspendable using

Re: [HACKERS] asynchronous execution

At Thu, 31 Aug 2017 21:52:36 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170831.215236.135328985.horiguchi.kyot...@lab.ntt.co.jp> > At Thu, 03 Aug 2017 09:30:57 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI >

Re: [HACKERS] asynchronous execution

At Thu, 03 Aug 2017 09:30:57 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170803.093057.261590619.horiguchi.kyot...@lab.ntt.co.jp> > > Unfortunately, that's probably another gigantic patch (that > > should probably be written by Andres). > > Yeah,

Re: [HACKERS] asynchronous execution

Thank you for the comment. At Tue, 1 Aug 2017 16:27:41 -0400, Robert Haas wrote in > On Mon, Jul 31, 2017 at 5:42 AM, Kyotaro HORIGUCHI > wrote: > > Another is getting

Re: [HACKERS] asynchronous execution

On Mon, Jul 31, 2017 at 5:42 AM, Kyotaro HORIGUCHI wrote: > Another is getting rid of recursive call to run an execution > tree. That happens to be exactly what Andres did for expression evaluation in commit b8d7f053c5c2bf2a7e8734fe3327f6a8bc711755, and I think

Re: [HACKERS] asynchronous execution

At Fri, 28 Jul 2017 17:31:05 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170728.173105.238045591.horiguchi.kyot...@lab.ntt.co.jp> > Thank you for the comment. > > At Wed, 26 Jul 2017 17:16:43 -0400, Robert Haas wrote > in

Re: [HACKERS] asynchronous execution

Thank you for the comment. At Wed, 26 Jul 2017 17:16:43 -0400, Robert Haas wrote in > But if we do, then I fear we'll just be reintroducing the same > performance regression that we introduced by

Re: [HACKERS] asynchronous execution

On Wed, Jul 26, 2017 at 5:43 PM, Tom Lane wrote: > I have not been paying any attention to this thread whatsoever, > but I wonder if you can address your problem by building on top of > the ExecProcNode replacement that Andres is working on, >

Re: [HACKERS] asynchronous execution

Robert Haas writes: > Ostensibly, the advantage of this framework over my previous proposal > is that it avoids inserting anything into ExecProcNode(), which is > probably a good thing to avoid given how frequently ExecProcNode() is > called. Unless the parent and the

Re: [HACKERS] asynchronous execution

On Tue, Jul 25, 2017 at 5:11 AM, Kyotaro HORIGUCHI wrote: > [ new patches ] I spent some time today refreshing my memory of what's going with this thread today. Ostensibly, the advantage of this framework over my previous proposal is that it avoids inserting

Re: [HACKERS] asynchronous execution

Hello, 8bf58c0d9bd33686 badly conflicts with this patch, so I'll rebase this and added a patch to refactor the function that Anotonin pointed. This would be merged into 0002 patch. At Tue, 18 Jul 2017 16:24:52 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI

Re: [HACKERS] asynchronous execution

Hello, At Tue, 11 Jul 2017 10:28:51 +0200, Antonin Houska wrote in <6448.1499761731@localhost> > Kyotaro HORIGUCHI wrote: > > Effectively it is a waiting-queue followed by a > > completed-list. The point of the compaction is keeping the order

Re: [HACKERS] asynchronous execution

Kyotaro HORIGUCHI wrote: > > Just one idea that I had while reading the code. > > > > In ExecAsyncEventLoop you iterate estate->es_pending_async, then move the > > complete requests to the end and finaly adjust estate->es_num_pending_async > > so > > that the

Re: [HACKERS] asynchronous execution

Thank you for the thought. This is at PoC level so I'd be grateful for this kind of fundamental comments. At Wed, 28 Jun 2017 20:22:24 +0200, Antonin Houska wrote in <392.1498674144@localhost> > Kyotaro HORIGUCHI wrote: > > > The patch got

Re: [HACKERS] asynchronous execution

Hi, I've returned. At Thu, 29 Jun 2017 14:08:27 +0900, Amit Langote wrote in <63a5a01c-2967-83e0-8bbf-c981404f5...@lab.ntt.co.jp> > Hi, > > On 2017/06/29 13:45, Kyotaro HORIGUCHI wrote: > > Thank you for looking this. > > > > At Wed, 28 Jun 2017 10:23:54 +0200,

Re: [HACKERS] asynchronous execution

Hi, On 2017/06/29 13:45, Kyotaro HORIGUCHI wrote: > Thank you for looking this. > > At Wed, 28 Jun 2017 10:23:54 +0200, Antonin Houska wrote: >> Kyotaro HORIGUCHI wrote: >> >>> The patch got conflicted. This is a new version just rebased to >>> the current

Re: [HACKERS] asynchronous execution

Thank you for looking this. At Wed, 28 Jun 2017 10:23:54 +0200, Antonin Houska wrote in <4579.1498638234@localhost> > Kyotaro HORIGUCHI wrote: > > > The patch got conflicted. This is a new version just rebased to > > the current master.

Re: [HACKERS] asynchronous execution

Kyotaro HORIGUCHI wrote: > The patch got conflicted. This is a new version just rebased to > the current master. Furtuer amendment will be taken later. Just one idea that I had while reading the code. In ExecAsyncEventLoop you iterate estate->es_pending_async,

Re: [HACKERS] asynchronous execution

Kyotaro HORIGUCHI wrote: > The patch got conflicted. This is a new version just rebased to > the current master. Furtuer amendment will be taken later. Can you please explain this part of make_append() ? /* Currently async on partitioned tables is not available

Re: [HACKERS] asynchronous execution

The patch got conflicted. This is a new version just rebased to the current master. Furtuer amendment will be taken later. > The attached patch is rebased on the current master, but no > substantial changes other than disallowing partitioned tables on > async by assertion. regards, -- Kyotaro

Re: [HACKERS] asynchronous execution

At Mon, 22 May 2017 13:12:14 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170522.131214.20936668.horiguchi.kyot...@lab.ntt.co.jp> > > The attached patch is rebased on the current master, but no > > substantial changes other than disallowing

Re: [HACKERS] asynchronous execution

Hello. At Tue, 04 Apr 2017 19:25:39 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170404.192539.29699823.horiguchi.kyot...@lab.ntt.co.jp> > The attached patch is rebased on the current master, but no > substantial changes other than disallowing

Re: [HACKERS] asynchronous execution

Hello, At Sun, 2 Apr 2017 12:21:14 -0400, Corey Huinker wrote in

Re: [HACKERS] asynchronous execution

> > > I'll continue working on this from this point aiming to the next > commit fest. > > This probably will not surprise you given the many commits in the past 2 weeks, but the patches no longer apply to master: $ git apply ~/async/0001-Allow-wait-event-set-to-be-registered-to-resource-ow.patch

Re: [HACKERS] asynchronous execution

Hello. This is the final report in this CF period. At Fri, 17 Mar 2017 17:35:05 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170317.173505.152063931.horiguchi.kyot...@lab.ntt.co.jp> > Async-capable plan is generated in planner. An Append contains

Re: [HACKERS] asynchronous execution

At Thu, 16 Mar 2017 17:16:32 -0400, Corey Huinker wrote in

Re: [HACKERS] asynchronous execution

On Thu, Mar 16, 2017 at 4:17 PM, Tom Lane wrote: > Corey Huinker writes: > > I reworked the test such that all of the foreign tables inherit from the > > same parent table, and if you query that you do get async execution. But > It > > doesn't work

Re: [HACKERS] asynchronous execution

Corey Huinker writes: > I reworked the test such that all of the foreign tables inherit from the > same parent table, and if you query that you do get async execution. But It > doesn't work when just stringing together those foreign tables with UNION > ALLs. > I don't

Re: [HACKERS] asynchronous execution

On Mon, Mar 13, 2017 at 9:28 PM, Amit Langote wrote: > On 2017/03/14 10:08, Corey Huinker wrote: > >> I don't think the plan itself will change as a result of applying this > >> patch. You might however be able to observe some performance > improvement. > > > > I

Re: [HACKERS] asynchronous execution

On 2017/03/14 10:08, Corey Huinker wrote: >> I don't think the plan itself will change as a result of applying this >> patch. You might however be able to observe some performance improvement. > > I could see no performance improvement, even with 16 separate queries > combined with UNION ALL.

Re: [HACKERS] asynchronous execution

> > I don't think the plan itself will change as a result of applying this > patch. You might however be able to observe some performance improvement. > > Thanks, > Amit > I could see no performance improvement, even with 16 separate queries combined with UNION ALL. Query performance was always

Re: [HACKERS] asynchronous execution

On 2017/03/14 6:31, Corey Huinker wrote: > On Mon, Mar 13, 2017 at 1:06 AM, Corey Huinker > wrote: > >> >>> I think it will, because Append itself has been made async-capable by one >>> of the patches and UNION ALL uses Append. But as mentioned above, only >>> the

Re: [HACKERS] asynchronous execution

On Mon, Mar 13, 2017 at 1:06 AM, Corey Huinker wrote: > >> I think it will, because Append itself has been made async-capable by one >> of the patches and UNION ALL uses Append. But as mentioned above, only >> the postgres_fdw foreign tables will be able to utilize this

Re: [HACKERS] asynchronous execution

> > > I think it will, because Append itself has been made async-capable by one > of the patches and UNION ALL uses Append. But as mentioned above, only > the postgres_fdw foreign tables will be able to utilize this for now. > > Ok, I'll re-run my test from a few weeks back and see if anything

Re: [HACKERS] asynchronous execution

On 2017/03/11 8:19, Corey Huinker wrote: > > On Thu, Feb 23, 2017 at 6:59 AM, Kyotaro HORIGUCHI > > > wrote: > > 9e43e87 > > > Patch fails on current master, but correctly applies to 9e43e87. Thanks > for including

Re: [HACKERS] asynchronous execution

On Thu, Feb 23, 2017 at 6:59 AM, Kyotaro HORIGUCHI < horiguchi.kyot...@lab.ntt.co.jp> wrote: > 9e43e87 Patch fails on current master, but correctly applies to 9e43e87. Thanks for including the commit id. Regression tests pass. As with my last attempt at reviewing this patch, I'm confused

Re: [HACKERS] asynchronous execution

Hello, I totally reorganized the patch set to four pathces on the current master (9e43e87). At Wed, 22 Feb 2017 17:39:45 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170222.173945.262776579.horiguchi.kyot...@lab.ntt.co.jp> > Finally, I couldn't

Re: [HACKERS] asynchronous execution

Thank you very much for testing this! At Tue, 7 Feb 2017 13:28:42 +0900, Amit Langote wrote in <9058d70b-a6b0-8b3c-091a-fe77ed0df...@lab.ntt.co.jp> > Horiguchi-san, > > On 2017/01/31 12:45, Kyotaro HORIGUCHI wrote: > > I noticed that this patch is conflicting

Re: [HACKERS] asynchronous execution

Horiguchi-san, On 2017/01/31 12:45, Kyotaro HORIGUCHI wrote: > I noticed that this patch is conflicting with 665d1fa (Logical > replication) so I rebased this. Only executor/Makefile > conflicted. With the latest set of patches, I observe a crash due to an Assert failure: #0 0x003969632625

Re: [HACKERS] asynchronous execution

On Fri, Feb 3, 2017 at 5:04 AM, Antonin Houska wrote: > Kyotaro HORIGUCHI wrote: > > > I noticed that this patch is conflicting with 665d1fa (Logical > > replication) so I rebased this. Only executor/Makefile > > conflicted. > > I was lucky

Re: [HACKERS] asynchronous execution

Kyotaro HORIGUCHI wrote: > I noticed that this patch is conflicting with 665d1fa (Logical > replication) so I rebased this. Only executor/Makefile > conflicted. I was lucky enough to see an infinite loop when using this patch, which I fixed by this change: diff

Re: [HACKERS] asynchronous execution

Thank you. At Wed, 1 Feb 2017 14:11:58 +0900, Michael Paquier wrote in

Re: [HACKERS] asynchronous execution

On Tue, Jan 31, 2017 at 12:45 PM, Kyotaro HORIGUCHI wrote: > I noticed that this patch is conflicting with 665d1fa (Logical > replication) so I rebased this. Only executor/Makefile > conflicted. The patches still apply, moved to CF 2017-03. Be aware of that: $

Re: [HACKERS] asynchronous execution

Hello, I cannot respond until next Monday, so I move this to the next CF by myself. At Tue, 15 Nov 2016 20:25:13 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20161115.202513.268072050.horiguchi.kyot...@lab.ntt.co.jp> > Hello, this is a maintenance

Re: [HACKERS] asynchronous execution

Hi, this is the 7th patch to make instrumentation work. Explain analyze shows the following result by the previous patch set . | Aggregate (cost=820.25..820.26 rows=1 width=8) (actual time=4324.676..4324.676 | rows=1 loops=1) | -> Append (cost=0.00..791.00 rows=11701 width=4) (actual

Re: [HACKERS] asynchronous execution

This is the rebased version on the current master(-0004), and added resowner stuff (0005) and unlikely(0006). At Tue, 18 Oct 2016 10:30:51 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20161018.103051.30820907.horiguchi.kyot...@lab.ntt.co.jp> > > > -

Re: [HACKERS] asynchronous execution

Hello, this works but ExecAppend gets a bit degradation. At Mon, 03 Oct 2016 19:46:32 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20161003.194632.204401048.horiguchi.kyot...@lab.ntt.co.jp> > > Some notes: > > > > - EvalPlanQual rechecks are

Re: [HACKERS] asynchronous execution

On Tue, Oct 4, 2016 at 7:53 AM, Amit Khandekar wrote: > Also, parent pointers are not required in the new design. Thinking of > parent pointers, now it seems the event won't get bubbled up the tree > with the new design. But still, , I think it's possible to switch over >

Re: [HACKERS] asynchronous execution

On 4 October 2016 at 02:30, Robert Haas wrote: > On Wed, Sep 28, 2016 at 12:30 AM, Amit Khandekar > wrote: >> On 24 September 2016 at 06:39, Robert Haas wrote: >>> Since Kyotaro Horiguchi found that my previous design had a

Re: [HACKERS] asynchronous execution

On Wed, Sep 28, 2016 at 12:30 AM, Amit Khandekar wrote: > On 24 September 2016 at 06:39, Robert Haas wrote: >> Since Kyotaro Horiguchi found that my previous design had a >> system-wide performance impact due to the ExecProcNode changes, I >>

Re: [HACKERS] asynchronous execution

Thank you for the thought. At Fri, 23 Sep 2016 21:09:03 -0400, Robert Haas wrote in > [ Adjusting subject line to reflect the actual topic of discussion better. ] > > On Fri, Sep 23, 2016 at 9:29 AM,

Re: [HACKERS] asynchronous execution

Hello, thank you for the comment. At Wed, 28 Sep 2016 10:00:08 +0530, Amit Khandekar wrote in

Re: [HACKERS] asynchronous execution

Sorry for delayed response, I'll have enough time from now and address this. At Fri, 23 Sep 2016 21:09:03 -0400, Robert Haas wrote in > Well, I promised to post this, so here it is. It's not really

Re: [HACKERS] asynchronous execution

On 24 September 2016 at 06:39, Robert Haas wrote: > Since Kyotaro Horiguchi found that my previous design had a > system-wide performance impact due to the ExecProcNode changes, I > decided to take a different approach here: I created an async > infrastructure where both

Re: [HACKERS] asynchronous execution

[ Adjusting subject line to reflect the actual topic of discussion better. ] On Fri, Sep 23, 2016 at 9:29 AM, Robert Haas wrote: > On Fri, Sep 23, 2016 at 8:45 AM, Amit Khandekar > wrote: >> For e.g., in the above plan which you specified, suppose

Re: [HACKERS] Asynchronous execution on FDW

On Mon, Aug 10, 2015 at 3:23 AM, Heikki Linnakangas hlinn...@iki.fi wrote: I've marked this as rejected in the commitfest, because others are working on a more general solution with parallel workers. That's still work-in-progress, and it's not certain if it's going to make it into 9.6, but if

Re: [HACKERS] Asynchronous execution on FDW

* Robert Haas (robertmh...@gmail.com) wrote: On Mon, Aug 10, 2015 at 3:23 AM, Heikki Linnakangas hlinn...@iki.fi wrote: I've marked this as rejected in the commitfest, because others are working on a more general solution with parallel workers. That's still work-in-progress, and it's not

Re: [HACKERS] Asynchronous execution on FDW

I've marked this as rejected in the commitfest, because others are working on a more general solution with parallel workers. That's still work-in-progress, and it's not certain if it's going to make it into 9.6, but if it does it will largely render this obsolete. We can revisit this patch later

Re: [HACKERS] Asynchronous execution on FDW

Hello Horiguchi-san, As for ForeignScan, it is merely an API for FDW and does nothing substantial so it would have nothing special to do. As for postgres_fdw, current patch restricts one execution per one foreign server at once by itself. We would have to provide another execution

Re: [HACKERS] Asynchronous execution on FDW

Hello, At Thu, 23 Jul 2015 09:38:39 +, Kouhei Kaigai kai...@ak.jp.nec.com wrote in 9a28c8860f777e439aa12e8aea7694f80111b...@bpxm15gp.gisp.nec.co.jp I expected workloads like single shot scan on a partitioned large fact table on DWH system. Yep, if workload is expected to rescan so

Re: [HACKERS] Asynchronous execution on FDW

If we have ParallelAppend node that kicks a background worker process for each underlying child node in parallel, does ForeignScan need to do something special? Although I don't see the point of the background worker in your story but at least for ParalleMergeAppend, it would

Re: [HACKERS] Asynchronous execution on FDW

-Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Kyotaro HORIGUCHI Sent: Wednesday, July 22, 2015 4:10 PM To: robertmh...@gmail.com Cc: hlinn...@iki.fi; pgsql-hackers@postgresql.org Subject: Re: [HACKERS

Re: [HACKERS] Asynchronous execution on FDW

Hello, thank you for the comment. At Fri, 17 Jul 2015 14:34:53 -0400, Robert Haas robertmh...@gmail.com wrote in ca+tgmoaijk1svzw_gkfu+zssxcijkfelqu2aomvuphpsfw4...@mail.gmail.com On Fri, Jul 3, 2015 at 4:41 PM, Heikki Linnakangas hlinn...@iki.fi wrote: At a quick glance, I think this has all

Re: [HACKERS] Asynchronous execution on FDW

Hello, Let me ask an elemental question. If we have ParallelAppend node that kicks a background worker process for each underlying child node in parallel, does ForeignScan need to do something special? Although I don't see the point of the background worker in your story but at least for

Re: [HACKERS] Asynchronous execution on FDW

On Fri, Jul 3, 2015 at 4:41 PM, Heikki Linnakangas hlinn...@iki.fi wrote: At a quick glance, I think this has all the same problems as starting the execution at ExecInit phase. The correct way to do this is to kick off the queries in the first IterateForeignScan() call. You said that ExecProc

Re: [HACKERS] Asynchronous execution on FDW

Hi, Currently there's no means to observe what it is doing from outside, so the additional sixth patch is to output debug messages about asynchronous execution. The sixth patch did not contain one message shown in the example. Attached is the revised version. Other patches are not changed.

Re: [HACKERS] Asynchronous execution on FDW

Hello, thank you for looking this. If it is acceptable to reconstruct the executor nodes to have additional return state PREP_RUN or such (which means it needs one more call for the first tuple) , I'll modify the whole executor to handle the state in the next patch to do so. I haven't take the

Re: [HACKERS] Asynchronous execution on FDW

On 07/02/2015 08:48 AM, Kyotaro HORIGUCHI wrote: - It was a problem when to give the first kick for async exec. It is not in ExecInit phase, and ExecProc phase does not fit, too. An extra phase ExecPreProc or something is too invasive. So I tried pre-exec callback. Any init-node can

Re: [HACKERS] Asynchronous execution on FDW

Ouch! I mistakenly made two CF entries for this patch. Could someone remove this entry for me? https://commitfest.postgresql.org/5/290/ The correct entry is /5/291/ == Hello. This is the new version of FDW async exection feature. The status of this feature is as follows, as of the last

Re: [HACKERS] Asynchronous execution on FDW

On Thu, Jul 2, 2015 at 3:07 PM, Kyotaro HORIGUCHI horiguchi.kyot...@lab.ntt.co.jp wrote: Ouch! I mistakenly made two CF entries for this patch. Could someone remove this entry for me? https://commitfest.postgresql.org/5/290/ The correct entry is /5/291/ Done. -- Michael -- Sent via

Re: [HACKERS] Asynchronous execution on FDW

Thank you. At Thu, 2 Jul 2015 16:02:27 +0900, Michael Paquier michael.paqu...@gmail.com wrote in CAB7nPqTs0YCwXedt1P=JjxFJeoj9UzLzkLuiX8=jdtpyutn...@mail.gmail.com On Thu, Jul 2, 2015 at 3:07 PM, Kyotaro HORIGUCHI horiguchi.kyot...@lab.ntt.co.jp wrote: Ouch! I mistakenly made two CF entries