Re: [HACKERS] Asynchronous execution on FDW

2015-08-10 Thread Robert Haas
On Mon, Aug 10, 2015 at 3:23 AM, Heikki Linnakangas hlinn...@iki.fi wrote: I've marked this as rejected in the commitfest, because others are working on a more general solution with parallel workers. That's still work-in-progress, and it's not certain if it's going to make it into 9.6, but if

Re: [HACKERS] Asynchronous execution on FDW

2015-08-10 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: On Mon, Aug 10, 2015 at 3:23 AM, Heikki Linnakangas hlinn...@iki.fi wrote: I've marked this as rejected in the commitfest, because others are working on a more general solution with parallel workers. That's still work-in-progress, and it's not

Re: [HACKERS] Asynchronous execution on FDW

2015-08-10 Thread Heikki Linnakangas
I've marked this as rejected in the commitfest, because others are working on a more general solution with parallel workers. That's still work-in-progress, and it's not certain if it's going to make it into 9.6, but if it does it will largely render this obsolete. We can revisit this patch later

Re: [HACKERS] Asynchronous execution on FDW

2015-07-24 Thread Kouhei Kaigai
Hello Horiguchi-san, As for ForeignScan, it is merely an API for FDW and does nothing substantial so it would have nothing special to do. As for postgres_fdw, current patch restricts one execution per one foreign server at once by itself. We would have to provide another execution

Re: [HACKERS] Asynchronous execution on FDW

2015-07-24 Thread Kyotaro HORIGUCHI
Hello, At Thu, 23 Jul 2015 09:38:39 +, Kouhei Kaigai kai...@ak.jp.nec.com wrote in 9a28c8860f777e439aa12e8aea7694f80111b...@bpxm15gp.gisp.nec.co.jp I expected workloads like single shot scan on a partitioned large fact table on DWH system. Yep, if workload is expected to rescan so

Re: [HACKERS] Asynchronous execution on FDW

2015-07-23 Thread Kouhei Kaigai
If we have ParallelAppend node that kicks a background worker process for each underlying child node in parallel, does ForeignScan need to do something special? Although I don't see the point of the background worker in your story but at least for ParalleMergeAppend, it would

Re: [HACKERS] Asynchronous execution on FDW

2015-07-22 Thread Kouhei Kaigai
] Asynchronous execution on FDW Hello, thank you for the comment. At Fri, 17 Jul 2015 14:34:53 -0400, Robert Haas robertmh...@gmail.com wrote in ca+tgmoaijk1svzw_gkfu+zssxcijkfelqu2aomvuphpsfw4...@mail.gmail.com On Fri, Jul 3, 2015 at 4:41 PM, Heikki Linnakangas hlinn...@iki.fi wrote: At a quick

Re: [HACKERS] Asynchronous execution on FDW

2015-07-22 Thread Kyotaro HORIGUCHI
Hello, thank you for the comment. At Fri, 17 Jul 2015 14:34:53 -0400, Robert Haas robertmh...@gmail.com wrote in ca+tgmoaijk1svzw_gkfu+zssxcijkfelqu2aomvuphpsfw4...@mail.gmail.com On Fri, Jul 3, 2015 at 4:41 PM, Heikki Linnakangas hlinn...@iki.fi wrote: At a quick glance, I think this has all

Re: [HACKERS] Asynchronous execution on FDW

2015-07-22 Thread Kyotaro HORIGUCHI
Hello, Let me ask an elemental question. If we have ParallelAppend node that kicks a background worker process for each underlying child node in parallel, does ForeignScan need to do something special? Although I don't see the point of the background worker in your story but at least for

Re: [HACKERS] Asynchronous execution on FDW

2015-07-17 Thread Robert Haas
On Fri, Jul 3, 2015 at 4:41 PM, Heikki Linnakangas hlinn...@iki.fi wrote: At a quick glance, I think this has all the same problems as starting the execution at ExecInit phase. The correct way to do this is to kick off the queries in the first IterateForeignScan() call. You said that ExecProc

Re: [HACKERS] Asynchronous execution on FDW

2015-07-10 Thread Kyotaro HORIGUCHI
Hi, Currently there's no means to observe what it is doing from outside, so the additional sixth patch is to output debug messages about asynchronous execution. The sixth patch did not contain one message shown in the example. Attached is the revised version. Other patches are not changed.

Re: [HACKERS] Asynchronous execution on FDW

2015-07-06 Thread Kyotaro HORIGUCHI
Hello, thank you for looking this. If it is acceptable to reconstruct the executor nodes to have additional return state PREP_RUN or such (which means it needs one more call for the first tuple) , I'll modify the whole executor to handle the state in the next patch to do so. I haven't take the

Re: [HACKERS] Asynchronous execution on FDW

2015-07-03 Thread Heikki Linnakangas
On 07/02/2015 08:48 AM, Kyotaro HORIGUCHI wrote: - It was a problem when to give the first kick for async exec. It is not in ExecInit phase, and ExecProc phase does not fit, too. An extra phase ExecPreProc or something is too invasive. So I tried pre-exec callback. Any init-node can

Re: [HACKERS] Asynchronous execution on FDW

2015-07-02 Thread Kyotaro HORIGUCHI
Ouch! I mistakenly made two CF entries for this patch. Could someone remove this entry for me? https://commitfest.postgresql.org/5/290/ The correct entry is /5/291/ == Hello. This is the new version of FDW async exection feature. The status of this feature is as follows, as of the last

Re: [HACKERS] Asynchronous execution on FDW

2015-07-02 Thread Michael Paquier
On Thu, Jul 2, 2015 at 3:07 PM, Kyotaro HORIGUCHI horiguchi.kyot...@lab.ntt.co.jp wrote: Ouch! I mistakenly made two CF entries for this patch. Could someone remove this entry for me? https://commitfest.postgresql.org/5/290/ The correct entry is /5/291/ Done. -- Michael -- Sent via

Re: [HACKERS] Asynchronous execution on FDW

2015-07-02 Thread Kyotaro HORIGUCHI
Thank you. At Thu, 2 Jul 2015 16:02:27 +0900, Michael Paquier michael.paqu...@gmail.com wrote in CAB7nPqTs0YCwXedt1P=JjxFJeoj9UzLzkLuiX8=jdtpyutn...@mail.gmail.com On Thu, Jul 2, 2015 at 3:07 PM, Kyotaro HORIGUCHI horiguchi.kyot...@lab.ntt.co.jp wrote: Ouch! I mistakenly made two CF entries

[HACKERS] Asynchronous execution on FDW

2015-07-01 Thread Kyotaro HORIGUCHI
Hello. This is the new version of FDW async exection feature. The status of this feature is as follows, as of the last commitfest. - Async execution is valuable to have. - But do the first kick in ExecInit phase is wrong. So the design outline of this version is as following, - The patch set