Re: [HACKERS] A GUC to prevent leader processes from running subplans?
On Sun, Nov 12, 2017 at 8:51 PM, Amit Kapila wrote: > On Sun, Nov 12, 2017 at 9:18 AM, Thomas Munro > wrote: >> How about parallel_leader_participation = on|off? The attached >> version has it that way, and adds regression tests to exercise on, off >> and off-but-couldn't-start-any-workers for both kinds of gather node. >> >> I'm not sure why node->need_to_rescan is initialised by both >> ExecGatherInit() and ExecGather(). Only the latter's value matters, >> right? >> > > I don't see anything like need_to_rescan in the GatherState node. Do > you intend to say need_to_scan_locally? If yes, then I think whatever > you said is right. Right, that's what I meant to write. Thanks. >> I've added this to the January Commitfest. >> > > +1 to this idea. Do you think such an option at table level can be > meaningful? We have a parallel_workers as a storage option for > tables, so users might want leader to participate in parallelism only > for some of the tables. I'm not sure. I think the reason for turning it off (other than developer testing) would be that the leader is getting tied up doing work that takes a long time (sorting, hashing, aggregating) and that's causing the workers to be blocked because their output queue is full. I think that type of behaviour comes from certain plan types, and it probably wouldn't make sense to associate this behaviour with the tables you're scanning. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A GUC to prevent leader processes from running subplans?
On Sun, Nov 12, 2017 at 9:18 AM, Thomas Munro wrote: > On Sat, Oct 21, 2017 at 8:09 AM, Robert Haas wrote: >> On Tue, Oct 17, 2017 at 7:27 AM, Thomas Munro >> wrote: >> >> I don't think overloading force_parallel_mode is a good idea, but >> having some other GUC for this seems OK to me. Not sure I like >> multiplex_gather, though. > > How about parallel_leader_participation = on|off? The attached > version has it that way, and adds regression tests to exercise on, off > and off-but-couldn't-start-any-workers for both kinds of gather node. > > I'm not sure why node->need_to_rescan is initialised by both > ExecGatherInit() and ExecGather(). Only the latter's value matters, > right? > I don't see anything like need_to_rescan in the GatherState node. Do you intend to say need_to_scan_locally? If yes, then I think whatever you said is right. > I've added this to the January Commitfest. > +1 to this idea. Do you think such an option at table level can be meaningful? We have a parallel_workers as a storage option for tables, so users might want leader to participate in parallelism only for some of the tables. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A GUC to prevent leader processes from running subplans?
On Sat, Oct 21, 2017 at 8:09 AM, Robert Haas wrote: > On Tue, Oct 17, 2017 at 7:27 AM, Thomas Munro > wrote: >> While testing parallelism work I've wanted to be able to prevent >> gather nodes from running the plan in the leader process, and I've >> heard others say the same. One way would be to add a GUC >> "multiplex_gather", like in the attached patch. If you set it to off, >> Gather and Gather Merge won't run the subplan unless they have to >> because no workers could be launched. I thought about adding a new >> value for force_parallel_mode instead, but someone mentioned they >> might want to do this on a production system too and >> force_parallel_mode is not really for end users. Better ideas? > > I don't think overloading force_parallel_mode is a good idea, but > having some other GUC for this seems OK to me. Not sure I like > multiplex_gather, though. How about parallel_leader_participation = on|off? The attached version has it that way, and adds regression tests to exercise on, off and off-but-couldn't-start-any-workers for both kinds of gather node. I'm not sure why node->need_to_rescan is initialised by both ExecGatherInit() and ExecGather(). Only the latter's value matters, right? I've added this to the January Commitfest. -- Thomas Munro http://www.enterprisedb.com parallel-leader-participation-v1.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A GUC to prevent leader processes from running subplans?
On Tue, Oct 17, 2017 at 7:27 AM, Thomas Munro wrote: > While testing parallelism work I've wanted to be able to prevent > gather nodes from running the plan in the leader process, and I've > heard others say the same. One way would be to add a GUC > "multiplex_gather", like in the attached patch. If you set it to off, > Gather and Gather Merge won't run the subplan unless they have to > because no workers could be launched. I thought about adding a new > value for force_parallel_mode instead, but someone mentioned they > might want to do this on a production system too and > force_parallel_mode is not really for end users. Better ideas? I don't think overloading force_parallel_mode is a good idea, but having some other GUC for this seems OK to me. Not sure I like multiplex_gather, though. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] A GUC to prevent leader processes from running subplans?
Hi hackers, While testing parallelism work I've wanted to be able to prevent gather nodes from running the plan in the leader process, and I've heard others say the same. One way would be to add a GUC "multiplex_gather", like in the attached patch. If you set it to off, Gather and Gather Merge won't run the subplan unless they have to because no workers could be launched. I thought about adding a new value for force_parallel_mode instead, but someone mentioned they might want to do this on a production system too and force_parallel_mode is not really for end users. Better ideas? -- Thomas Munro http://www.enterprisedb.com 0001-Add-a-GUC-to-control-whether-Gather-runs-subplans.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
