Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Robert Haas]

On Mon, Aug 15, 2016 at 9:10 PM, Tom Lane  wrote:
> Robert Haas  writes:
>> On Sat, Aug 13, 2016 at 11:56 AM, Alvaro Herrera
>>  wrote:
>>> I too prefer to keep it turned off in 9.6 and consider enabling it by
>>> default on a future release (10 is probably good).  Interested users can
>>> carefully test the feature without endangering other unsuspecting users.
>>> I agree with the idea of keeping it enabled in master, so that it'll get
>>> a modicum of testing there by hackers, too.
>
>> Sounds like that is the consensus.  Who's going to implement it?
>
> I believe we're talking about reverting 77cd477c4 (in 9.6 only not
> master), correct?  It's a little harder than just "git revert" because
> of the subsequent max_parallel_degree -> max_parallel_workers_per_gather
> name change, but still not exactly rocket science.
>
> Since 77cd477c4 was your commit, I'd sort of expect you to do the
> honors, but if you don't want to I can.

OK, I'll do it.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Tom Lane]

Robert Haas  writes:
> On Sat, Aug 13, 2016 at 11:56 AM, Alvaro Herrera
>  wrote:
>> I too prefer to keep it turned off in 9.6 and consider enabling it by
>> default on a future release (10 is probably good).  Interested users can
>> carefully test the feature without endangering other unsuspecting users.
>> I agree with the idea of keeping it enabled in master, so that it'll get
>> a modicum of testing there by hackers, too.

> Sounds like that is the consensus.  Who's going to implement it?

I believe we're talking about reverting 77cd477c4 (in 9.6 only not
master), correct?  It's a little harder than just "git revert" because
of the subsequent max_parallel_degree -> max_parallel_workers_per_gather
name change, but still not exactly rocket science.

Since 77cd477c4 was your commit, I'd sort of expect you to do the
honors, but if you don't want to I can.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Robert Haas]

On Sat, Aug 13, 2016 at 11:56 AM, Alvaro Herrera
 wrote:
> Michael Paquier wrote:
>
>> Being cautious pays more in the long term, so seeing the number of
>> bugs that showed up I'd rather vote for having it disabled by default
>> in 9.6 stable, and enabled on master to aim at enabling it in 10.0.
>
> I too prefer to keep it turned off in 9.6 and consider enabling it by
> default on a future release (10 is probably good).  Interested users can
> carefully test the feature without endangering other unsuspecting users.
>
> I agree with the idea of keeping it enabled in master, so that it'll get
> a modicum of testing there by hackers, too.

Sounds like that is the consensus.  Who's going to implement it?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Alvaro Herrera]

Michael Paquier wrote:

> Being cautious pays more in the long term, so seeing the number of
> bugs that showed up I'd rather vote for having it disabled by default
> in 9.6 stable, and enabled on master to aim at enabling it in 10.0.

I too prefer to keep it turned off in 9.6 and consider enabling it by
default on a future release (10 is probably good).  Interested users can
carefully test the feature without endangering other unsuspecting users.

I agree with the idea of keeping it enabled in master, so that it'll get
a modicum of testing there by hackers, too.

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Michael Paquier]

On Thu, Aug 4, 2016 at 10:52 PM, David G. Johnston
 wrote:
> My initial reaction was +1 but now I'm leaning toward enabled by default.
>
> Those who would upgrade to 9.6 within a year of its release are most likely,
> process and personality wise, to be those for whom the risks and rewards of
> new features is part of their everyday routine.
>
> If indeed we release 10.0 with it enabled by default we should be confident
> in its stability at that point in the 9.6.x branch.

Being cautious pays more in the long term, so seeing the number of
bugs that showed up I'd rather vote for having it disabled by default
in 9.6 stable, and enabled on master to aim at enabling it in 10.0.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[David G. Johnston]

On Thu, Aug 4, 2016 at 9:25 AM, Robert Haas  wrote:

> On Thu, Aug 4, 2016 at 12:56 AM, Tom Lane  wrote:
> > Noah Misch  writes:
> >> On Wed, Apr 20, 2016 at 02:28:15PM -0400, Tom Lane wrote:
> >>> +1, but let's put an entry on the 9.6 open-items page to remind us to
> >>> make that decision at the right time.
> >
> >> It's that time.  Do we restore the max_parallel_workers_per_gather=0
> default,
> >> or is enabling this by default the right thing after all?
> >
> > At this point I'd have to vote against enabling by default in 9.6.  The
> > fact that in the past week we've found bugs as bad as e1a93dd6a does not
> > give me a warm fuzzy feeling about the parallel-query code being ready
> > for prime time.
> >
> > Of course the question is how do we ever get to that point if we chicken
> > out with enabling it by default now.  Maybe we could keep it turned on
> > in HEAD.
>
> However, we have a long track
> record of being cautious about things like this, so I would be OK with
> the idea of disabling this in the 9.6 branch and leaving it turned on
> in master, with the hope of shipping it enabled-by-default in v10.


​My initial reaction was +1 but now I'm leaning toward enabled by default.

Those who would upgrade to 9.6 within a year of its release are most
likely, process and personality wise, to be those for whom the risks and
rewards of new features is part of their everyday routine.

If indeed we release 10.0 with it enabled by default we should be confident
in its stability at that point in the 9.6.x branch.

David J.​

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Robert Haas]

On Thu, Aug 4, 2016 at 12:56 AM, Tom Lane  wrote:
> Noah Misch  writes:
>> On Wed, Apr 20, 2016 at 02:28:15PM -0400, Tom Lane wrote:
>>> +1, but let's put an entry on the 9.6 open-items page to remind us to
>>> make that decision at the right time.
>
>> It's that time.  Do we restore the max_parallel_workers_per_gather=0 default,
>> or is enabling this by default the right thing after all?
>
> At this point I'd have to vote against enabling by default in 9.6.  The
> fact that in the past week we've found bugs as bad as e1a93dd6a does not
> give me a warm fuzzy feeling about the parallel-query code being ready
> for prime time.
>
> Of course the question is how do we ever get to that point if we chicken
> out with enabling it by default now.  Maybe we could keep it turned on
> in HEAD.

What do other people think about this topic?

Personally, I've found the process of fixing the parallel query bugs
rather exhausting, and if we leave it turned on by default in 9.6,
we'll probably get a lot more of those bug reports a lot sooner than
we will otherwise.  But I'd kind of rather get it out of the way than
put it off: those bugs need to be fixed at some point, and we won't
find them if nobody runs the code.  However, we have a long track
record of being cautious about things like this, so I would be OK with
the idea of disabling this in the 9.6 branch and leaving it turned on
in master, with the hope of shipping it enabled-by-default in v10.  I
think it would not be good to leave disabled it by default in master
-- the code won't get much testing then, we won't find the bugs, and
we'll build more stuff on top of what we've already got without
finding the cracks in the foundation.  I think users will be more
inclined to forgive us for parallel query bugs in 2016 than in 2026;
better to debug it before the honeymoon wears off.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Tom Lane]

Noah Misch  writes:
> On Wed, Apr 20, 2016 at 02:28:15PM -0400, Tom Lane wrote:
>> +1, but let's put an entry on the 9.6 open-items page to remind us to
>> make that decision at the right time.

> It's that time.  Do we restore the max_parallel_workers_per_gather=0 default,
> or is enabling this by default the right thing after all?

At this point I'd have to vote against enabling by default in 9.6.  The
fact that in the past week we've found bugs as bad as e1a93dd6a does not
give me a warm fuzzy feeling about the parallel-query code being ready
for prime time.

Of course the question is how do we ever get to that point if we chicken
out with enabling it by default now.  Maybe we could keep it turned on
in HEAD.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Noah Misch]

On Wed, Apr 20, 2016 at 02:28:15PM -0400, Tom Lane wrote:
> Andres Freund  writes:
> > max_parallel_degree currently defaults to 0.  I think we should enable
> > it by default for at least the beta period. Otherwise we're primarily
> > going to get reports back after the release.
> 
> > Then, at the end of beta, we can decide what the default should be.
> 
> +1, but let's put an entry on the 9.6 open-items page to remind us to
> make that decision at the right time.

It's that time.  Do we restore the max_parallel_workers_per_gather=0 default,
or is enabling this by default the right thing after all?


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Gavin Flower]

On 22/04/16 17:36, Amit Kapila wrote:
On Fri, Apr 22, 2016 at 1:31 AM, Gavin Flower 
> 
wrote:


On 22/04/16 06:07, Robert Haas wrote:

On Thu, Apr 21, 2016 at 1:48 PM, Tom Lane > wrote:

Robert Haas > writes:

On Wed, Apr 20, 2016 at 2:28 PM, Tom Lane
> wrote:

Andres Freund > writes:

max_parallel_degree currently defaults to 0. 
I think we should enable

it by default for at least the beta period.
Otherwise we're primarily
going to get reports back after the release.

So, I suggest that the only sensible non-zero values
here are probably
"1" or "2", given a default pool of 8 worker processes
system-wide.
Andres told me yesterday he'd vote for "2".  Any other
opinions?

It has to be at least 2 for beta purposes, else you are
not testing
situations with more than one worker process at all, which
would be
rather a large omission no?

That's what Andres, thought, too.  From my point of view, the big
thing is to be using workers at all.  It is of course possible
that
there could be some bugs where a single worker is not enough, but
there's a lot of types of bug where even one worker would probably
find the problem.  But I'm OK with changing the default to 2.

I'm curious.

Why not 4?


IIUC, the idea to change max_parallel_degree for beta is to catch any 
bugs in parallelism code, not to do any performance testing of same.  
So, I think either 1 or 2 should be sufficient to hit the bugs if 
there are any.  Do you have any reason to think that we might miss 
some category of bugs if we don't use higher max_parallel_degree?



With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com 
No.  Just felt that 4 would not be too great for the type of processor 
chips used on servers to handle.


For complications, such as race conditions and implied logical 
assumptions - I tend to think of 0, 1, 2, 3, many.


Essentially just a gut feeling that 4 might reveal more corner cases.


Cheers,
Gavin



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Gavin Flower]

On 23/04/16 00:56, Robert Haas wrote:

On Thu, Apr 21, 2016 at 7:20 PM, Tom Lane  wrote:

Robert Haas  writes:

On Thu, Apr 21, 2016 at 4:01 PM, Gavin Flower
 wrote:

Why not 4?  As most processors now have at least 4 physical cores, & surely
it be more likely to flush out race conditions.

Because if we did that, then it's extremely likely that people would
end up writing queries that are faster only if workers are present,
and then not get any workers.

Is that because max_worker_processes is only 8 by default?  Maybe we
need to raise that, at least for beta purposes?

I'm not really in favor of that.  I mean, almost all of our default
settings are optimized for running PostgreSQL on, for example, a
Raspberry Pi 2, so it would seem odd to suddenly swing the other
direction and assume that there are more than 8 unused CPU cores.  It
doesn't make sense to me to roll out settings in beta that we wouldn't
be willing to release with if they work out.  That's why, honestly, I
would prefer max_parallel_degree=1, which I think would be practical
for many real-world deployments.  max_parallel_degree=2 is OK.  Beyond
that, we're just setting people up to fail, I think.  Higher settings
should probably only be used on substantial hardware, and not
everybody has that.

If Java can find out how many processors there are available to it, 
since JDK1.4, then surely PostgreSQL can do the same?


So how about the default being half the available processors rounded up 
to the nearest integer?


Perhaps the GUC for workers should be a percentage of the available 
processors, with the minimum & maximum workers optionally specified - or 
something of that nature?



Cheers,
Gavin


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Robert Haas]

On Fri, Apr 22, 2016 at 10:07 AM, Joshua D. Drake  
wrote:
> This is the problem right here.
>
> We should be shipping for a reasonable production configuration. It is not
> reasonable to assume that someone is going to be running on a Rasberry Pi 2.
> Yes, we can effectively run on that platform that doesn't mean it should be
> our default target configuration. Consider that a 5.00/mo Digital Ocean VM
> is going to outperform a Rasberry Pi.

I don't disagree with that, and I think there is a considerable amount
of work that could be done to create a saner "out of the box"
configuration.  But I don't think that the two weeks before beta is
the right time to start building a consensus around what that might
look like.

> True, but isn't that also what context switching and (possibly)
> hyperthreading are for?

Sure.  What you should expect, though, is that overall system
throughput will be higher if the system is not oversubscribed.  You
can use parallel query selectively to speed up certain queries even if
that takes you above the number of CPUs you have; if those queries are
on a deadline, finishing them sooner may be worth whatever you lose in
overall throughput.

> I think your argument sounds more like a production solution, not a Beta
> solution. We should be pushing it a little bit in Beta.

Shipping with max_parallel_workers=2 *is* pushing it a little bit.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Joshua D. Drake]

On 04/22/2016 06:47 AM, Robert Haas wrote:

On Fri, Apr 22, 2016 at 9:33 AM, Tom Lane  wrote:

Robert Haas  writes:

On Thu, Apr 21, 2016 at 7:20 PM, Tom Lane  wrote:

Is that because max_worker_processes is only 8 by default?  Maybe we
need to raise that, at least for beta purposes?



I'm not really in favor of that.  I mean, almost all of our default
settings are optimized for running PostgreSQL on, for example, a
Raspberry Pi 2, so it would seem odd to suddenly swing the other
direction and assume that there are more than 8 unused CPU cores.


This is the problem right here.

We should be shipping for a reasonable production configuration. It is 
not reasonable to assume that someone is going to be running on a 
Rasberry Pi 2. Yes, we can effectively run on that platform that doesn't 
mean it should be our default target configuration. Consider that a 
5.00/mo Digital Ocean VM is going to outperform a Rasberry Pi.




It is much less likely to be true for parallel workers.  The reason
why those processes aren't contending for the CPU at the same time is
generally that most of the connections are in fact idle.  But a
parallel worker is never idle.  It is launched when it is needed to
run a query and exits immediately afterward.  If it's not contending
for the CPU, it will be contending for I/O bandwidth, or a lock.



True, but isn't that also what context switching and (possibly) 
hyperthreading are for?




So what I'm concerned about for beta purposes is that we have a setup that
can exercise cases like, say, varying orders in which different workers
return tuples, or potential deadlocks between sibling workers.  We'd get
no coverage of that behavioral space at max_parallel_degree=1.  I'm not
really convinced that we'll get adequate coverage at
max_parallel_degree=2.


The right solution to that is for people who have the right hardware
to raise the settings, not to unleash a ridiculous set of defaults on
everyone.  I really hope that some people do serious destruction
testing of parallel query and try to break it.  For example, you could
use the parallel_degree reloption to force 100 parallel workers to
scan the same relation.   That's likely to be dog slow, but it might
well turn up some bugs.


I think your argument sounds more like a production solution, not a Beta 
solution. We should be pushing it a little bit in Beta.


JD









--
Command Prompt, Inc.  http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Robert Haas]

On Fri, Apr 22, 2016 at 9:33 AM, Tom Lane  wrote:
> Robert Haas  writes:
>> On Thu, Apr 21, 2016 at 7:20 PM, Tom Lane  wrote:
>>> Is that because max_worker_processes is only 8 by default?  Maybe we
>>> need to raise that, at least for beta purposes?
>
>> I'm not really in favor of that.  I mean, almost all of our default
>> settings are optimized for running PostgreSQL on, for example, a
>> Raspberry Pi 2, so it would seem odd to suddenly swing the other
>> direction and assume that there are more than 8 unused CPU cores.
>
> I'm not following why you think that max_worker_processes cannot be
> set higher than the number of cores.  By that argument, it's insane
> that we ship with max_connections = 100.  In practice it's generally
> fine, and people can get away with oversubscribing their core count
> even more than that, because it's seldom that all those processes
> are actually contending for CPU at the same time.  There are enough
> inefficiencies in our parallel-query design that the same will most
> certainly be true for parallel workers.

It is much less likely to be true for parallel workers.  The reason
why those processes aren't contending for the CPU at the same time is
generally that most of the connections are in fact idle.  But a
parallel worker is never idle.  It is launched when it is needed to
run a query and exits immediately afterward.  If it's not contending
for the CPU, it will be contending for I/O bandwidth, or a lock.

> So what I'm concerned about for beta purposes is that we have a setup that
> can exercise cases like, say, varying orders in which different workers
> return tuples, or potential deadlocks between sibling workers.  We'd get
> no coverage of that behavioral space at max_parallel_degree=1.  I'm not
> really convinced that we'll get adequate coverage at
> max_parallel_degree=2.

The right solution to that is for people who have the right hardware
to raise the settings, not to unleash a ridiculous set of defaults on
everyone.  I really hope that some people do serious destruction
testing of parallel query and try to break it.  For example, you could
use the parallel_degree reloption to force 100 parallel workers to
scan the same relation.   That's likely to be dog slow, but it might
well turn up some bugs.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Tom Lane]

Robert Haas  writes:
> On Thu, Apr 21, 2016 at 7:20 PM, Tom Lane  wrote:
>> Is that because max_worker_processes is only 8 by default?  Maybe we
>> need to raise that, at least for beta purposes?

> I'm not really in favor of that.  I mean, almost all of our default
> settings are optimized for running PostgreSQL on, for example, a
> Raspberry Pi 2, so it would seem odd to suddenly swing the other
> direction and assume that there are more than 8 unused CPU cores.

I'm not following why you think that max_worker_processes cannot be
set higher than the number of cores.  By that argument, it's insane
that we ship with max_connections = 100.  In practice it's generally
fine, and people can get away with oversubscribing their core count
even more than that, because it's seldom that all those processes
are actually contending for CPU at the same time.  There are enough
inefficiencies in our parallel-query design that the same will most
certainly be true for parallel workers.

So what I'm concerned about for beta purposes is that we have a setup that
can exercise cases like, say, varying orders in which different workers
return tuples, or potential deadlocks between sibling workers.  We'd get
no coverage of that behavioral space at max_parallel_degree=1.  I'm not
really convinced that we'll get adequate coverage at
max_parallel_degree=2.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Andreas Joseph Krogh]

På fredag 22. april 2016 kl. 14:56:33, skrev Robert Haas >:
On Thu, Apr 21, 2016 at 7:20 PM, Tom Lane  wrote:
 > Robert Haas  writes:
 >> On Thu, Apr 21, 2016 at 4:01 PM, Gavin Flower
 >>  wrote:
 >>> Why not 4?  As most processors now have at least 4 physical cores, & 
surely
 >>> it be more likely to flush out race conditions.
 >
 >> Because if we did that, then it's extremely likely that people would
 >> end up writing queries that are faster only if workers are present,
 >> and then not get any workers.
 >
 > Is that because max_worker_processes is only 8 by default?  Maybe we
 > need to raise that, at least for beta purposes?

 I'm not really in favor of that.  I mean, almost all of our default
 settings are optimized for running PostgreSQL on, for example, a
 Raspberry Pi 2, so it would seem odd to suddenly swing the other
 direction and assume that there are more than 8 unused CPU cores.  It
 doesn't make sense to me to roll out settings in beta that we wouldn't
 be willing to release with if they work out.  That's why, honestly, I
 would prefer max_parallel_degree=1, which I think would be practical
 for many real-world deployments.  max_parallel_degree=2 is OK.  Beyond
 that, we're just setting people up to fail, I think.  Higher settings
 should probably only be used on substantial hardware, and not
 everybody has that.
 
Maybe it's time to ask the question if the settings should be optimized more 
for high-end HW and not som matchstick-box? I mean, most of the people I know 
who are responsible for databases run them on HW colser to high-end than 
low-end. I'm not sure why optimizing for low-end is such a great choice.
 
-- Andreas Joseph Krog


 

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Robert Haas]

On Thu, Apr 21, 2016 at 7:20 PM, Tom Lane  wrote:
> Robert Haas  writes:
>> On Thu, Apr 21, 2016 at 4:01 PM, Gavin Flower
>>  wrote:
>>> Why not 4?  As most processors now have at least 4 physical cores, & surely
>>> it be more likely to flush out race conditions.
>
>> Because if we did that, then it's extremely likely that people would
>> end up writing queries that are faster only if workers are present,
>> and then not get any workers.
>
> Is that because max_worker_processes is only 8 by default?  Maybe we
> need to raise that, at least for beta purposes?

I'm not really in favor of that.  I mean, almost all of our default
settings are optimized for running PostgreSQL on, for example, a
Raspberry Pi 2, so it would seem odd to suddenly swing the other
direction and assume that there are more than 8 unused CPU cores.  It
doesn't make sense to me to roll out settings in beta that we wouldn't
be willing to release with if they work out.  That's why, honestly, I
would prefer max_parallel_degree=1, which I think would be practical
for many real-world deployments.  max_parallel_degree=2 is OK.  Beyond
that, we're just setting people up to fail, I think.  Higher settings
should probably only be used on substantial hardware, and not
everybody has that.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Amit Kapila]

On Fri, Apr 22, 2016 at 1:31 AM, Gavin Flower  wrote:

> On 22/04/16 06:07, Robert Haas wrote:
>
>> On Thu, Apr 21, 2016 at 1:48 PM, Tom Lane  wrote:
>>
>>> Robert Haas  writes:
>>>
 On Wed, Apr 20, 2016 at 2:28 PM, Tom Lane  wrote:

> Andres Freund  writes:
>
>> max_parallel_degree currently defaults to 0.  I think we should enable
>> it by default for at least the beta period. Otherwise we're primarily
>> going to get reports back after the release.
>>
> So, I suggest that the only sensible non-zero values here are probably
 "1" or "2", given a default pool of 8 worker processes system-wide.
 Andres told me yesterday he'd vote for "2".  Any other opinions?

>>> It has to be at least 2 for beta purposes, else you are not testing
>>> situations with more than one worker process at all, which would be
>>> rather a large omission no?
>>>
>> That's what Andres, thought, too.  From my point of view, the big
>> thing is to be using workers at all.  It is of course possible that
>> there could be some bugs where a single worker is not enough, but
>> there's a lot of types of bug where even one worker would probably
>> find the problem.  But I'm OK with changing the default to 2.
>>
>> I'm curious.
>
> Why not 4?


IIUC, the idea to change max_parallel_degree for beta is to catch any bugs
in parallelism code, not to do any performance testing of same.  So, I
think either 1 or 2 should be sufficient to hit the bugs if there are any.
Do you have any reason to think that we might miss some category of bugs if
we don't use higher max_parallel_degree?


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Tom Lane]

Robert Haas  writes:
> On Thu, Apr 21, 2016 at 4:01 PM, Gavin Flower
>  wrote:
>> Why not 4?  As most processors now have at least 4 physical cores, & surely
>> it be more likely to flush out race conditions.

> Because if we did that, then it's extremely likely that people would
> end up writing queries that are faster only if workers are present,
> and then not get any workers.

Is that because max_worker_processes is only 8 by default?  Maybe we
need to raise that, at least for beta purposes?

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Robert Haas]

On Thu, Apr 21, 2016 at 4:01 PM, Gavin Flower
 wrote:
> On 22/04/16 06:07, Robert Haas wrote:
>>
>> On Thu, Apr 21, 2016 at 1:48 PM, Tom Lane  wrote:
>>>
>>> Robert Haas  writes:

 On Wed, Apr 20, 2016 at 2:28 PM, Tom Lane  wrote:
>
> Andres Freund  writes:
>>
>> max_parallel_degree currently defaults to 0.  I think we should enable
>> it by default for at least the beta period. Otherwise we're primarily
>> going to get reports back after the release.

 So, I suggest that the only sensible non-zero values here are probably
 "1" or "2", given a default pool of 8 worker processes system-wide.
 Andres told me yesterday he'd vote for "2".  Any other opinions?
>>>
>>> It has to be at least 2 for beta purposes, else you are not testing
>>> situations with more than one worker process at all, which would be
>>> rather a large omission no?
>>
>> That's what Andres, thought, too.  From my point of view, the big
>> thing is to be using workers at all.  It is of course possible that
>> there could be some bugs where a single worker is not enough, but
>> there's a lot of types of bug where even one worker would probably
>> find the problem.  But I'm OK with changing the default to 2.
>>
> I'm curious.
>
> Why not 4?  As most processors now have at least 4 physical cores, & surely
> it be more likely to flush out race conditions.

Because if we did that, then it's extremely likely that people would
end up writing queries that are faster only if workers are present,
and then not get any workers.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Gavin Flower]

On 22/04/16 06:07, Robert Haas wrote:

On Thu, Apr 21, 2016 at 1:48 PM, Tom Lane  wrote:

Robert Haas  writes:

On Wed, Apr 20, 2016 at 2:28 PM, Tom Lane  wrote:

Andres Freund  writes:

max_parallel_degree currently defaults to 0.  I think we should enable
it by default for at least the beta period. Otherwise we're primarily
going to get reports back after the release.

So, I suggest that the only sensible non-zero values here are probably
"1" or "2", given a default pool of 8 worker processes system-wide.
Andres told me yesterday he'd vote for "2".  Any other opinions?

It has to be at least 2 for beta purposes, else you are not testing
situations with more than one worker process at all, which would be
rather a large omission no?

That's what Andres, thought, too.  From my point of view, the big
thing is to be using workers at all.  It is of course possible that
there could be some bugs where a single worker is not enough, but
there's a lot of types of bug where even one worker would probably
find the problem.  But I'm OK with changing the default to 2.


I'm curious.

Why not 4?  As most processors now have at least 4 physical cores, & 
surely it be more likely to flush out race conditions.



Cheers,
Gavin



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Robert Haas]

On Thu, Apr 21, 2016 at 1:48 PM, Tom Lane  wrote:
> Robert Haas  writes:
>> On Wed, Apr 20, 2016 at 2:28 PM, Tom Lane  wrote:
>>> Andres Freund  writes:
 max_parallel_degree currently defaults to 0.  I think we should enable
 it by default for at least the beta period. Otherwise we're primarily
 going to get reports back after the release.
>
>> So, I suggest that the only sensible non-zero values here are probably
>> "1" or "2", given a default pool of 8 worker processes system-wide.
>> Andres told me yesterday he'd vote for "2".  Any other opinions?
>
> It has to be at least 2 for beta purposes, else you are not testing
> situations with more than one worker process at all, which would be
> rather a large omission no?

That's what Andres, thought, too.  From my point of view, the big
thing is to be using workers at all.  It is of course possible that
there could be some bugs where a single worker is not enough, but
there's a lot of types of bug where even one worker would probably
find the problem.  But I'm OK with changing the default to 2.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Tom Lane]

Robert Haas  writes:
> On Wed, Apr 20, 2016 at 2:28 PM, Tom Lane  wrote:
>> Andres Freund  writes:
>>> max_parallel_degree currently defaults to 0.  I think we should enable
>>> it by default for at least the beta period. Otherwise we're primarily
>>> going to get reports back after the release.

> So, I suggest that the only sensible non-zero values here are probably
> "1" or "2", given a default pool of 8 worker processes system-wide.
> Andres told me yesterday he'd vote for "2".  Any other opinions?

It has to be at least 2 for beta purposes, else you are not testing
situations with more than one worker process at all, which would be
rather a large omission no?

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Robert Haas]

On Wed, Apr 20, 2016 at 2:28 PM, Tom Lane  wrote:
> Andres Freund  writes:
>> max_parallel_degree currently defaults to 0.  I think we should enable
>> it by default for at least the beta period. Otherwise we're primarily
>> going to get reports back after the release.
>
>> Then, at the end of beta, we can decide what the default should be.
>
> +1, but let's put an entry on the 9.6 open-items page to remind us to
> make that decision at the right time.

So, I suggest that the only sensible non-zero values here are probably
"1" or "2", given a default pool of 8 worker processes system-wide.
Andres told me yesterday he'd vote for "2".  Any other opinions?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Tom Lane]

Andres Freund  writes:
> max_parallel_degree currently defaults to 0.  I think we should enable
> it by default for at least the beta period. Otherwise we're primarily
> going to get reports back after the release.

> Then, at the end of beta, we can decide what the default should be.

+1, but let's put an entry on the 9.6 open-items page to remind us to
make that decision at the right time.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] max_parallel_degree > 0 for 9.6 beta

[Andreas Joseph Krogh]

På onsdag 20. april 2016 kl. 19:46:31, skrev Andres Freund >:
Hi,

 max_parallel_degree currently defaults to 0.  I think we should enable
 it by default for at least the beta period. Otherwise we're primarily
 going to get reports back after the release.

 Then, at the end of beta, we can decide what the default should be.
 
+1
 
Not enabling it per default gives the signal "It's not safe".
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andr...@visena.com 
www.visena.com 
 


 

[HACKERS] max_parallel_degree > 0 for 9.6 beta

[Andres Freund]

Hi,

max_parallel_degree currently defaults to 0.  I think we should enable
it by default for at least the beta period. Otherwise we're primarily
going to get reports back after the release.

Then, at the end of beta, we can decide what the default should be.

- Andres


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers