Hello.
Fully-asynchronous executor needs that every node is stateful and
suspendable at the time of requesting for the next tuples to
underneath nodes. I tried pure push-base executor but failed.
After the miserable patch upthread, I finally managed to make
executor nodes suspendable using comput
At Thu, 31 Aug 2017 21:52:36 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI
wrote in
<20170831.215236.135328985.horiguchi.kyot...@lab.ntt.co.jp>
> At Thu, 03 Aug 2017 09:30:57 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI
> wrote in
> <20170803.093057.261590619.horiguchi.kyot...@lab.ntt.co.j
At Thu, 03 Aug 2017 09:30:57 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI
wrote in
<20170803.093057.261590619.horiguchi.kyot...@lab.ntt.co.jp>
> > Unfortunately, that's probably another gigantic patch (that
> > should probably be written by Andres).
>
> Yeah, but async executor on the current
Thank you for the comment.
At Tue, 1 Aug 2017 16:27:41 -0400, Robert Haas wrote in
> On Mon, Jul 31, 2017 at 5:42 AM, Kyotaro HORIGUCHI
> wrote:
> > Another is getting rid of recursive call to run an execution
> > tree.
>
> That happens to be exactly what Andres did for expression evaluation
On Mon, Jul 31, 2017 at 5:42 AM, Kyotaro HORIGUCHI
wrote:
> Another is getting rid of recursive call to run an execution
> tree.
That happens to be exactly what Andres did for expression evaluation
in commit b8d7f053c5c2bf2a7e8734fe3327f6a8bc711755, and I think
generalizing that to include the pl
At Fri, 28 Jul 2017 17:31:05 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI
wrote in
<20170728.173105.238045591.horiguchi.kyot...@lab.ntt.co.jp>
> Thank you for the comment.
>
> At Wed, 26 Jul 2017 17:16:43 -0400, Robert Haas wrote
> in
> > regression all the same. Every type of intermediat
Thank you for the comment.
At Wed, 26 Jul 2017 17:16:43 -0400, Robert Haas wrote
in
> But if we do, then I fear we'll just be reintroducing the same
> performance regression that we introduced by switching to this
> framework from the previous one - or maybe a different one, but a
> regression
On Wed, Jul 26, 2017 at 5:43 PM, Tom Lane wrote:
> I have not been paying any attention to this thread whatsoever,
> but I wonder if you can address your problem by building on top of
> the ExecProcNode replacement that Andres is working on,
> https://www.postgresql.org/message-id/20170726012641.b
Robert Haas writes:
> Ostensibly, the advantage of this framework over my previous proposal
> is that it avoids inserting anything into ExecProcNode(), which is
> probably a good thing to avoid given how frequently ExecProcNode() is
> called. Unless the parent and the child both know about asynch
On Tue, Jul 25, 2017 at 5:11 AM, Kyotaro HORIGUCHI
wrote:
> [ new patches ]
I spent some time today refreshing my memory of what's going with this
thread today.
Ostensibly, the advantage of this framework over my previous proposal
is that it avoids inserting anything into ExecProcNode(), which
Hello,
8bf58c0d9bd33686 badly conflicts with this patch, so I'll rebase
this and added a patch to refactor the function that Anotonin
pointed. This would be merged into 0002 patch.
At Tue, 18 Jul 2017 16:24:52 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI
wrote in
<20170718.162452.221576658.h
Hello,
At Tue, 11 Jul 2017 10:28:51 +0200, Antonin Houska wrote in
<6448.1499761731@localhost>
> Kyotaro HORIGUCHI wrote:
> > Effectively it is a waiting-queue followed by a
> > completed-list. The point of the compaction is keeping the order
> > of waiting or not-yet-completed requests, which
Kyotaro HORIGUCHI wrote:
> > Just one idea that I had while reading the code.
> >
> > In ExecAsyncEventLoop you iterate estate->es_pending_async, then move the
> > complete requests to the end and finaly adjust estate->es_num_pending_async
> > so
> > that the array no longer contains the comple
Thank you for the thought.
This is at PoC level so I'd be grateful for this kind of
fundamental comments.
At Wed, 28 Jun 2017 20:22:24 +0200, Antonin Houska wrote in
<392.1498674144@localhost>
> Kyotaro HORIGUCHI wrote:
>
> > The patch got conflicted. This is a new version just rebased to
> >
Hi, I've returned.
At Thu, 29 Jun 2017 14:08:27 +0900, Amit Langote
wrote in
<63a5a01c-2967-83e0-8bbf-c981404f5...@lab.ntt.co.jp>
> Hi,
>
> On 2017/06/29 13:45, Kyotaro HORIGUCHI wrote:
> > Thank you for looking this.
> >
> > At Wed, 28 Jun 2017 10:23:54 +0200, Antonin Houska wrote:
> >> Can
Hi,
On 2017/06/29 13:45, Kyotaro HORIGUCHI wrote:
> Thank you for looking this.
>
> At Wed, 28 Jun 2017 10:23:54 +0200, Antonin Houska wrote:
>> Kyotaro HORIGUCHI wrote:
>>
>>> The patch got conflicted. This is a new version just rebased to
>>> the current master. Furtuer amendment will be taken
Thank you for looking this.
At Wed, 28 Jun 2017 10:23:54 +0200, Antonin Houska wrote in
<4579.1498638234@localhost>
> Kyotaro HORIGUCHI wrote:
>
> > The patch got conflicted. This is a new version just rebased to
> > the current master. Furtuer amendment will be taken later.
>
> Can you pleas
Kyotaro HORIGUCHI wrote:
> The patch got conflicted. This is a new version just rebased to
> the current master. Furtuer amendment will be taken later.
Just one idea that I had while reading the code.
In ExecAsyncEventLoop you iterate estate->es_pending_async, then move the
complete requests to
Kyotaro HORIGUCHI wrote:
> The patch got conflicted. This is a new version just rebased to
> the current master. Furtuer amendment will be taken later.
Can you please explain this part of make_append() ?
/* Currently async on partitioned tables is not available */
Assert(nasyncplans == 0 || par
The patch got conflicted. This is a new version just rebased to
the current master. Furtuer amendment will be taken later.
> The attached patch is rebased on the current master, but no
> substantial changes other than disallowing partitioned tables on
> async by assertion.
regards,
--
Kyotaro H
At Mon, 22 May 2017 13:12:14 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI
wrote in
<20170522.131214.20936668.horiguchi.kyot...@lab.ntt.co.jp>
> > The attached patch is rebased on the current master, but no
> > substantial changes other than disallowing partitioned tables on
> > async by assert
Hello.
At Tue, 04 Apr 2017 19:25:39 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI
wrote in
<20170404.192539.29699823.horiguchi.kyot...@lab.ntt.co.jp>
> The attached patch is rebased on the current master, but no
> substantial changes other than disallowing partitioned tables on
> async by asse
Hello,
At Sun, 2 Apr 2017 12:21:14 -0400, Corey Huinker
wrote in
> >
> >
> > I'll continue working on this from this point aiming to the next
> > commit fest.
> >
> >
> This probably will not surprise you given the many commits in the past 2
> weeks, but the patches no longer apply to master:
>
>
> I'll continue working on this from this point aiming to the next
> commit fest.
>
>
This probably will not surprise you given the many commits in the past 2
weeks, but the patches no longer apply to master:
$ git apply
~/async/0001-Allow-wait-event-set-to-be-registered-to-resource-ow.patch
Hello. This is the final report in this CF period.
At Fri, 17 Mar 2017 17:35:05 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI
wrote in
<20170317.173505.152063931.horiguchi.kyot...@lab.ntt.co.jp>
> Async-capable plan is generated in planner. An Append contains at
> least one async-capable child
At Thu, 16 Mar 2017 17:16:32 -0400, Corey Huinker
wrote in
> On Thu, Mar 16, 2017 at 4:17 PM, Tom Lane wrote:
>
> > Corey Huinker writes:
> > > I reworked the test such that all of the foreign tables inherit from the
> > > same parent table, and if you query that you do get async execution. B
On Thu, Mar 16, 2017 at 4:17 PM, Tom Lane wrote:
> Corey Huinker writes:
> > I reworked the test such that all of the foreign tables inherit from the
> > same parent table, and if you query that you do get async execution. But
> It
> > doesn't work when just stringing together those foreign tabl
Corey Huinker writes:
> I reworked the test such that all of the foreign tables inherit from the
> same parent table, and if you query that you do get async execution. But It
> doesn't work when just stringing together those foreign tables with UNION
> ALLs.
> I don't know how to proceed with thi
On Mon, Mar 13, 2017 at 9:28 PM, Amit Langote wrote:
> On 2017/03/14 10:08, Corey Huinker wrote:
> >> I don't think the plan itself will change as a result of applying this
> >> patch. You might however be able to observe some performance
> improvement.
> >
> > I could see no performance improvem
On 2017/03/14 10:08, Corey Huinker wrote:
>> I don't think the plan itself will change as a result of applying this
>> patch. You might however be able to observe some performance improvement.
>
> I could see no performance improvement, even with 16 separate queries
> combined with UNION ALL. Quer
>
> I don't think the plan itself will change as a result of applying this
> patch. You might however be able to observe some performance improvement.
>
> Thanks,
> Amit
>
I could see no performance improvement, even with 16 separate queries
combined with UNION ALL. Query performance was always wi
On 2017/03/14 6:31, Corey Huinker wrote:
> On Mon, Mar 13, 2017 at 1:06 AM, Corey Huinker
> wrote:
>
>>
>>> I think it will, because Append itself has been made async-capable by one
>>> of the patches and UNION ALL uses Append. But as mentioned above, only
>>> the postgres_fdw foreign tables wil
On Mon, Mar 13, 2017 at 1:06 AM, Corey Huinker
wrote:
>
>> I think it will, because Append itself has been made async-capable by one
>> of the patches and UNION ALL uses Append. But as mentioned above, only
>> the postgres_fdw foreign tables will be able to utilize this for now.
>>
>>
> Ok, I'll
>
>
> I think it will, because Append itself has been made async-capable by one
> of the patches and UNION ALL uses Append. But as mentioned above, only
> the postgres_fdw foreign tables will be able to utilize this for now.
>
>
Ok, I'll re-run my test from a few weeks back and see if anything has
On 2017/03/11 8:19, Corey Huinker wrote:
>
> On Thu, Feb 23, 2017 at 6:59 AM, Kyotaro HORIGUCHI
> mailto:horiguchi.kyot...@lab.ntt.co.jp>>
> wrote:
>
> 9e43e87
>
>
> Patch fails on current master, but correctly applies to 9e43e87. Thanks
> for including the commit id.
>
> Regression tests
On Thu, Feb 23, 2017 at 6:59 AM, Kyotaro HORIGUCHI <
horiguchi.kyot...@lab.ntt.co.jp> wrote:
> 9e43e87
Patch fails on current master, but correctly applies to 9e43e87. Thanks for
including the commit id.
Regression tests pass.
As with my last attempt at reviewing this patch, I'm confused about
Hello, I totally reorganized the patch set to four pathces on the
current master (9e43e87).
At Wed, 22 Feb 2017 17:39:45 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI
wrote in
<20170222.173945.262776579.horiguchi.kyot...@lab.ntt.co.jp>
> Finally, I couldn't see the crash for the (maybe) same c
Thank you very much for testing this!
At Tue, 7 Feb 2017 13:28:42 +0900, Amit Langote
wrote in <9058d70b-a6b0-8b3c-091a-fe77ed0df...@lab.ntt.co.jp>
> Horiguchi-san,
>
> On 2017/01/31 12:45, Kyotaro HORIGUCHI wrote:
> > I noticed that this patch is conflicting with 665d1fa (Logical
> > replicati
Horiguchi-san,
On 2017/01/31 12:45, Kyotaro HORIGUCHI wrote:
> I noticed that this patch is conflicting with 665d1fa (Logical
> replication) so I rebased this. Only executor/Makefile
> conflicted.
With the latest set of patches, I observe a crash due to an Assert failure:
#0 0x003969632625
On Fri, Feb 3, 2017 at 5:04 AM, Antonin Houska wrote:
> Kyotaro HORIGUCHI wrote:
>
> > I noticed that this patch is conflicting with 665d1fa (Logical
> > replication) so I rebased this. Only executor/Makefile
> > conflicted.
>
> I was lucky enough to see an infinite loop when using this patch, w
Kyotaro HORIGUCHI wrote:
> I noticed that this patch is conflicting with 665d1fa (Logical
> replication) so I rebased this. Only executor/Makefile
> conflicted.
I was lucky enough to see an infinite loop when using this patch, which I
fixed by this change:
diff --git a/src/backend/executor/exec
Thank you.
At Wed, 1 Feb 2017 14:11:58 +0900, Michael Paquier
wrote in
> On Tue, Jan 31, 2017 at 12:45 PM, Kyotaro HORIGUCHI
> wrote:
> > I noticed that this patch is conflicting with 665d1fa (Logical
> > replication) so I rebased this. Only executor/Makefile
> > conflicted.
>
> The patches s
On Tue, Jan 31, 2017 at 12:45 PM, Kyotaro HORIGUCHI
wrote:
> I noticed that this patch is conflicting with 665d1fa (Logical
> replication) so I rebased this. Only executor/Makefile
> conflicted.
The patches still apply, moved to CF 2017-03. Be aware of that:
$ git diff HEAD~6 --check
contrib/post
Hello,
I cannot respond until next Monday, so I move this to the next CF
by myself.
At Tue, 15 Nov 2016 20:25:13 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI
wrote in
<20161115.202513.268072050.horiguchi.kyot...@lab.ntt.co.jp>
> Hello, this is a maintenance post of reased patches.
> I added
Hi, this is the 7th patch to make instrumentation work.
Explain analyze shows the following result by the previous patch set .
| Aggregate (cost=820.25..820.26 rows=1 width=8) (actual
time=4324.676..4324.676
| rows=1 loops=1)
| -> Append (cost=0.00..791.00 rows=11701 width=4) (actual
time=
This is the rebased version on the current master(-0004), and
added resowner stuff (0005) and unlikely(0006).
At Tue, 18 Oct 2016 10:30:51 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI
wrote in
<20161018.103051.30820907.horiguchi.kyot...@lab.ntt.co.jp>
> > > - Errors in the executor can leak t
Hello, this works but ExecAppend gets a bit degradation.
At Mon, 03 Oct 2016 19:46:32 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI
wrote in
<20161003.194632.204401048.horiguchi.kyot...@lab.ntt.co.jp>
> > Some notes:
> >
> > - EvalPlanQual rechecks are broken.
This is fixed by adding (restor
On Tue, Oct 4, 2016 at 7:53 AM, Amit Khandekar wrote:
> Also, parent pointers are not required in the new design. Thinking of
> parent pointers, now it seems the event won't get bubbled up the tree
> with the new design. But still, , I think it's possible to switch over
> to the other asynchronous
On 4 October 2016 at 02:30, Robert Haas wrote:
> On Wed, Sep 28, 2016 at 12:30 AM, Amit Khandekar
> wrote:
>> On 24 September 2016 at 06:39, Robert Haas wrote:
>>> Since Kyotaro Horiguchi found that my previous design had a
>>> system-wide performance impact due to the ExecProcNode changes, I
>
On Wed, Sep 28, 2016 at 12:30 AM, Amit Khandekar wrote:
> On 24 September 2016 at 06:39, Robert Haas wrote:
>> Since Kyotaro Horiguchi found that my previous design had a
>> system-wide performance impact due to the ExecProcNode changes, I
>> decided to take a different approach here: I created a
Thank you for the thought.
At Fri, 23 Sep 2016 21:09:03 -0400, Robert Haas wrote
in
> [ Adjusting subject line to reflect the actual topic of discussion better. ]
>
> On Fri, Sep 23, 2016 at 9:29 AM, Robert Haas wrote:
> > On Fri, Sep 23, 2016 at 8:45 AM, Amit Khandekar
> > wrote:
> >> For
Hello, thank you for the comment.
At Wed, 28 Sep 2016 10:00:08 +0530, Amit Khandekar
wrote in
> On 24 September 2016 at 06:39, Robert Haas wrote:
> > Since Kyotaro Horiguchi found that my previous design had a
> > system-wide performance impact due to the ExecProcNode changes, I
> > decided to
Sorry for delayed response, I'll have enough time from now and
address this.
At Fri, 23 Sep 2016 21:09:03 -0400, Robert Haas wrote
in
> Well, I promised to post this, so here it is. It's not really working
> all that well at this point, and it's definitely not doing anything
> that interesting
On 24 September 2016 at 06:39, Robert Haas wrote:
> Since Kyotaro Horiguchi found that my previous design had a
> system-wide performance impact due to the ExecProcNode changes, I
> decided to take a different approach here: I created an async
> infrastructure where both the requestor and the requ
[ Adjusting subject line to reflect the actual topic of discussion better. ]
On Fri, Sep 23, 2016 at 9:29 AM, Robert Haas wrote:
> On Fri, Sep 23, 2016 at 8:45 AM, Amit Khandekar
> wrote:
>> For e.g., in the above plan which you specified, suppose :
>> 1. Hash Join has called ExecProcNode() for
* Robert Haas (robertmh...@gmail.com) wrote:
> On Mon, Aug 10, 2015 at 3:23 AM, Heikki Linnakangas wrote:
> > I've marked this as rejected in the commitfest, because others are
> > working on a more general solution with parallel workers. That's still
> > work-in-progress, and it's not certain if
On Mon, Aug 10, 2015 at 3:23 AM, Heikki Linnakangas wrote:
> I've marked this as rejected in the commitfest, because others are
> working on a more general solution with parallel workers. That's still
> work-in-progress, and it's not certain if it's going to make it into
> 9.6, but if it does it w
I've marked this as rejected in the commitfest, because others are
working on a more general solution with parallel workers. That's still
work-in-progress, and it's not certain if it's going to make it into
9.6, but if it does it will largely render this obsolete. We can revisit
this patch later in
Hello Horiguchi-san,
> > > As for ForeignScan, it is merely an API for FDW and does nothing
> > > substantial so it would have nothing special to do. As for
> > > postgres_fdw, current patch restricts one execution per one
> > > foreign server at once by itself. We would have to provide
> > > anot
Hello,
At Thu, 23 Jul 2015 09:38:39 +, Kouhei Kaigai wrote
in <9a28c8860f777e439aa12e8aea7694f80111b...@bpxm15gp.gisp.nec.co.jp>
> I expected workloads like single shot scan on a partitioned large
> fact table on DWH system. Yep, if workload is expected to rescan
> so frequently, its expecte
> > If we have ParallelAppend node that kicks a background worker process for
> > each underlying child node in parallel, does ForeignScan need to do
> > something
> > special?
>
> Although I don't see the point of the background worker in your
> story but at least for ParalleMergeAppend, it woul
Hello,
> Let me ask an elemental question.
>
> If we have ParallelAppend node that kicks a background worker process for
> each underlying child node in parallel, does ForeignScan need to do something
> special?
Although I don't see the point of the background worker in your
story but at least f
Subject: Re: [HACKERS] Asynchronous execution on FDW
>
> Hello, thank you for the comment.
>
> At Fri, 17 Jul 2015 14:34:53 -0400, Robert Haas wrote
> in
> > On Fri, Jul 3, 2015 at 4:41 PM, Heikki Linnakangas wrote:
> > > At a quick glance, I think this has all t
Hello, thank you for the comment.
At Fri, 17 Jul 2015 14:34:53 -0400, Robert Haas wrote
in
> On Fri, Jul 3, 2015 at 4:41 PM, Heikki Linnakangas wrote:
> > At a quick glance, I think this has all the same problems as starting the
> > execution at ExecInit phase. The correct way to do this is to
On Fri, Jul 3, 2015 at 4:41 PM, Heikki Linnakangas wrote:
> At a quick glance, I think this has all the same problems as starting the
> execution at ExecInit phase. The correct way to do this is to kick off the
> queries in the first IterateForeignScan() call. You said that "ExecProc
> phase does
Hi,
> Currently there's no means to observe what it is doing from
> outside, so the additional sixth patch is to output debug
> messages about asynchronous execution.
The sixth patch did not contain one message shown in the example.
Attached is the revised version.
Other patches are not changed.
Hello, thank you for looking this.
If it is acceptable to reconstruct the executor nodes to have
additional return state PREP_RUN or such (which means it needs
one more call for the first tuple) , I'll modify the whole
executor to handle the state in the next patch to do so.
I haven't take the ad
On 07/02/2015 08:48 AM, Kyotaro HORIGUCHI wrote:
- It was a problem when to give the first kick for async exec. It
is not in ExecInit phase, and ExecProc phase does not fit,
too. An extra phase ExecPreProc or something is too
invasive. So I tried "pre-exec callback".
Any init-node ca
Thank you.
At Thu, 2 Jul 2015 16:02:27 +0900, Michael Paquier
wrote in
> On Thu, Jul 2, 2015 at 3:07 PM, Kyotaro HORIGUCHI
> wrote:
> > Ouch! I mistakenly made two CF entries for this patch. Could
> > someone remove this entry for me?
> >
> > https://commitfest.postgresql.org/5/290/
> >
> > Th
On Thu, Jul 2, 2015 at 3:07 PM, Kyotaro HORIGUCHI
wrote:
> Ouch! I mistakenly made two CF entries for this patch. Could
> someone remove this entry for me?
>
> https://commitfest.postgresql.org/5/290/
>
> The correct entry is "/5/291/"
Done.
--
Michael
--
Sent via pgsql-hackers mailing list (
Ouch! I mistakenly made two CF entries for this patch. Could
someone remove this entry for me?
https://commitfest.postgresql.org/5/290/
The correct entry is "/5/291/"
==
Hello. This is the new version of FDW async exection feature.
The status of this feature is as follows, as of the last co
Hello. This is the new version of FDW async exection feature.
The status of this feature is as follows, as of the last commitfest.
- Async execution is valuable to have.
- But do the first kick in ExecInit phase is wrong.
So the design outline of this version is as following,
- The patch set co
72 matches
Mail list logo
