Re: Parallel append plan instability/randomness

2018-01-08 Thread Amit Kapila
On Tue, Jan 9, 2018 at 12:48 AM, Tom Lane wrote: > Robert Haas writes: >> On Sun, Jan 7, 2018 at 11:40 PM, Amit Kapila wrote: >>> One theory that can explain above failure is that the costs of >>> scanning some of the sub-paths is very close due to which sometimes >>> the results can vary. If t

Re: Parallel append plan instability/randomness

2018-01-08 Thread Tom Lane
I wrote: > BTW, I had not looked closely at list_qsort before, but now that I have > it seems seriously bizarre. Why is it allocating a new List header > rather than reusing the old one? Because it stomps on the next-links > of the list cells, the old header is effectively corrupted and can't > p

Re: Parallel append plan instability/randomness

2018-01-08 Thread Tom Lane
I wrote: >> In the regression test case at hand, the startup costs are all zero so >> this change wouldn't improve the test case's stability. So I'm thinking >> that in addition, it would be a good idea for these functions to break >> exact compare_path_costs ties in some arbitrary but determinist

Re: Parallel append plan instability/randomness

2018-01-08 Thread Robert Haas
On Mon, Jan 8, 2018 at 1:26 PM, Jim Finnerty wrote: > I see. Maybe the EXPLAIN output can be made more stable when "EXPLAIN (costs > FALSE)", though? I think that would be fixing the problem from the wrong end. We want to actually get a stable choice of plan, not display something other than th

Re: Parallel append plan instability/randomness

2018-01-08 Thread Tom Lane
I wrote: > In the regression test case at hand, the startup costs are all zero so > this change wouldn't improve the test case's stability. So I'm thinking > that in addition, it would be a good idea for these functions to break > exact compare_path_costs ties in some arbitrary but deterministic w

Re: Parallel append plan instability/randomness

2018-01-08 Thread Tom Lane
Robert Haas writes: > On Sun, Jan 7, 2018 at 11:40 PM, Amit Kapila wrote: >> One theory that can explain above failure is that the costs of >> scanning some of the sub-paths is very close due to which sometimes >> the results can vary. If that is the case, then probably using >> fuzz_factor in c

Re: Parallel append plan instability/randomness

2018-01-08 Thread Tom Lane
Robert Haas writes: > [ example where current parallel-append dispatch algorithm loses ] I should clarify my thinking a bit more: I was imagining that we might switch to a system where, as each worker comes free, it makes a dynamic choice of which subplan to take next. That means that not only d

Re: Parallel append plan instability/randomness

2018-01-08 Thread Robert Haas
On Mon, Jan 8, 2018 at 11:57 AM, Tom Lane wrote: >> Ignoring some details around partial vs. non-partial plans, that's >> pretty much what we ARE doing, but to make it efficient, we sort the >> paths at plan time so that those choices are easy to make at runtime. >> If we didn't do that, we could

Re: Parallel append plan instability/randomness

2018-01-08 Thread Tom Lane
Robert Haas writes: > On Mon, Jan 8, 2018 at 11:42 AM, Tom Lane wrote: >> More generally, I wonder if >> it wouldn't be better to implement this behavior at runtime rather >> than plan time. > Ignoring some details around partial vs. non-partial plans, that's > pretty much what we ARE doing, but

Re: Parallel append plan instability/randomness

2018-01-08 Thread Robert Haas
On Mon, Jan 8, 2018 at 11:42 AM, Tom Lane wrote: > Jim Finnerty writes: >> Ordering the plan output elements by estimated cost will cause spurious plan >> changes to be reported after table cardinalities change. Can we choose an >> explain output order that is stable to changes in cardinality, p

Re: Parallel append plan instability/randomness

2018-01-08 Thread Tom Lane
Jim Finnerty writes: > Ordering the plan output elements by estimated cost will cause spurious plan > changes to be reported after table cardinalities change. Can we choose an > explain output order that is stable to changes in cardinality, please? I found the code that's doing this, in create_a

Re: Parallel append plan instability/randomness

2018-01-08 Thread Robert Haas
On Mon, Jan 8, 2018 at 11:28 AM, Jim Finnerty wrote: > Ordering the plan output elements by estimated cost will cause spurious plan > changes to be reported after table cardinalities change. Can we choose an > explain output order that is stable to changes in cardinality, please? No. The order

Re: Parallel append plan instability/randomness

2018-01-08 Thread Robert Haas
On Sun, Jan 7, 2018 at 11:40 PM, Amit Kapila wrote: > One theory that can explain above failure is that the costs of > scanning some of the sub-paths is very close due to which sometimes > the results can vary. If that is the case, then probably using > fuzz_factor in costs comparison (as is done

Re: Parallel append plan instability/randomness

2018-01-08 Thread Amit Kapila
On Mon, Jan 8, 2018 at 2:11 PM, Amit Khandekar wrote: > On 8 January 2018 at 13:35, Amit Kapila wrote: >> On Mon, Jan 8, 2018 at 11:26 AM, Tom Lane wrote: >>> Amit Khandekar writes: The fact that b_star gets moved from 5th position to the first position in the scans, indicates that i

Re: Parallel append plan instability/randomness

2018-01-08 Thread Amit Khandekar
On 8 January 2018 at 13:35, Amit Kapila wrote: > On Mon, Jan 8, 2018 at 11:26 AM, Tom Lane wrote: >> Amit Khandekar writes: >>> The fact that b_star gets moved from 5th position to the first >>> position in the scans, indicates that it's cost shoots up from 1.04 to >>> a value greater than 1.16

Re: Parallel append plan instability/randomness

2018-01-08 Thread Amit Kapila
On Mon, Jan 8, 2018 at 11:26 AM, Tom Lane wrote: > Amit Khandekar writes: >> The fact that b_star gets moved from 5th position to the first >> position in the scans, indicates that it's cost shoots up from 1.04 to >> a value greater than 1.16. It does not look like a case where two >> costs are

Re: Parallel append plan instability/randomness

2018-01-07 Thread Tom Lane
Amit Khandekar writes: > The fact that b_star gets moved from 5th position to the first > position in the scans, indicates that it's cost shoots up from 1.04 to > a value greater than 1.16. It does not look like a case where two > costs are almost same due to which their positions swap sometimes.

Re: Parallel append plan instability/randomness

2018-01-07 Thread Amit Khandekar
On 8 January 2018 at 10:10, Amit Kapila wrote: > regression=# explain select round(avg(aa)), sum(aa) from a_star; > QUERY PLAN > --- > Finalize Aggregate (cost=2.30..2.31 rows=1 width=40

Re: Parallel append plan instability/randomness

2018-01-07 Thread Amit Kapila
On Sun, Jan 7, 2018 at 5:44 AM, Tom Lane wrote: > According to buildfarm member silverfish, > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=silverfish&dt=2018-01-06%2008%3A53%3A38 > > it's possible to sometimes get this failure in the regression tests: > > *** > /mnt/buildfarm/buildro

Parallel append plan instability/randomness

2018-01-06 Thread Tom Lane
According to buildfarm member silverfish, https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=silverfish&dt=2018-01-06%2008%3A53%3A38 it's possible to sometimes get this failure in the regression tests: *** /mnt/buildfarm/buildroot/HEAD/pgsql.build/../pgsql/src/test/regress/expected/select_