pgsql: Rename IndexInfo.ii_KeyAttrNumbers array
Rename IndexInfo.ii_KeyAttrNumbers array Rename ii_KeyAttrNumbers to ii_IndexAttrNumbers to prevent confusion with ii_NumIndexAttrs/ii_NumIndexKeyAttrs. ii_IndexAttrNumbers contains all attributes including "including" columns, not only key attribute. Discussion: https://www.postgresql.org/message-id/13123421-1d52-d0e4-c95c-6d69011e0595%40sigaev.ru Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/c9c875a28fa6cbc38c227fb9e656dd7be948166f Modified Files -- src/backend/catalog/index.c| 24 src/backend/catalog/toasting.c | 4 ++-- src/backend/commands/analyze.c | 2 +- src/backend/commands/indexcmds.c | 10 +- src/backend/utils/cache/relcache.c | 2 +- src/backend/utils/sort/tuplesort.c | 14 +++--- src/include/nodes/execnodes.h | 2 +- 7 files changed, 29 insertions(+), 29 deletions(-)
pgsql: Revert MERGE patch
Revert MERGE patch This reverts commits d204ef63776b8a00ca220adec23979091564e465, 83454e3c2b28141c0db01c7d2027e01040df5249 and a few more commits thereafter (complete list at the end) related to MERGE feature. While the feature was fully functional, with sufficient test coverage and necessary documentation, it was felt that some parts of the executor and parse-analyzer can use a different design and it wasn't possible to do that in the available time. So it was decided to revert the patch for PG11 and retry again in the future. Thanks again to all reviewers and bug reporters. List of commits reverted, in reverse chronological order: f1464c5380 Improve parse representation for MERGE ddb4158579 MERGE syntax diagram correction 530e69e59b Allow cpluspluscheck to pass by renaming variable 01b88b4df5 MERGE minor errata 3af7b2b0d4 MERGE fix variable warning in non-assert builds a5d86181ec MERGE INSERT allows only one VALUES clause 4b2d44031f MERGE post-commit review 4923550c20 Tab completion for MERGE aa3faa3c7a WITH support in MERGE 83454e3c2b New files for MERGE d204ef6377 MERGE SQL Command following SQL:2016 Author: Pavan Deolasee Reviewed-by: Michael Paquier Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/08ea7a2291db21a618d19d612c8060cda68f1892 Modified Files -- contrib/test_decoding/expected/ddl.out | 46 - contrib/test_decoding/sql/ddl.sql | 16 - doc/src/sgml/libpq.sgml|8 +- doc/src/sgml/mvcc.sgml | 28 +- doc/src/sgml/plpgsql.sgml |3 +- doc/src/sgml/ref/allfiles.sgml |1 - doc/src/sgml/ref/create_policy.sgml|7 - doc/src/sgml/ref/insert.sgml | 11 +- doc/src/sgml/ref/merge.sgml| 617 doc/src/sgml/reference.sgml|1 - doc/src/sgml/trigger.sgml | 20 - src/backend/access/heap/heapam.c | 28 +- src/backend/catalog/sql_features.txt |6 +- src/backend/commands/explain.c | 33 - src/backend/commands/prepare.c |1 - src/backend/commands/trigger.c | 270 +--- src/backend/executor/Makefile |2 +- src/backend/executor/README| 11 - src/backend/executor/execMain.c| 17 - src/backend/executor/execMerge.c | 682 src/backend/executor/execPartition.c | 121 -- src/backend/executor/execReplication.c |4 +- src/backend/executor/nodeModifyTable.c | 280 +--- src/backend/executor/spi.c |3 - src/backend/nodes/copyfuncs.c | 58 - src/backend/nodes/equalfuncs.c | 49 - src/backend/nodes/nodeFuncs.c | 64 +- src/backend/nodes/outfuncs.c | 40 - src/backend/nodes/readfuncs.c | 45 - src/backend/optimizer/plan/createplan.c| 22 +- src/backend/optimizer/plan/planner.c | 29 +- src/backend/optimizer/plan/setrefs.c | 54 - src/backend/optimizer/prep/preptlist.c | 41 - src/backend/optimizer/util/pathnode.c | 11 +- src/backend/optimizer/util/plancat.c |4 - src/backend/parser/Makefile|2 +- src/backend/parser/analyze.c | 18 +- src/backend/parser/gram.y | 151 +- src/backend/parser/parse_agg.c | 10 - src/backend/parser/parse_clause.c | 57 +- src/backend/parser/parse_collate.c |1 - src/backend/parser/parse_expr.c|3 - src/backend/parser/parse_func.c|3 - src/backend/parser/parse_merge.c | 654 src/backend/parser/parse_relation.c| 10 - src/backend/rewrite/rewriteHandler.c | 115 +- src/backend/rewrite/rowsecurity.c | 97 -- src/backend/tcop/pquery.c |5 - src/backend/tcop/utility.c | 16 - src/bin/psql/tab-complete.c| 87 +- src/include/access/heapam.h| 13 +- src/include/commands/trigger.h |6 +- src/include/executor/execMerge.h | 31 - src/include/executor/execPartition.h |1 - src/include/executor/instrument.h |7 +- src/include/executor/nodeModifyTable.h | 23 - src/include/executor/spi.h |1 - src/include/nodes/execnodes.h | 65 +- src/include/nodes/nodes.h |7 +- src/include/nodes/parsenodes.h | 53 +- src
Re: pgsql: Support partition pruning at execution time
On 11 April 2018 at 18:58, David Rowley wrote:
> On 10 April 2018 at 08:55, Tom Lane wrote:
>> Alvaro Herrera writes:
>>> David Rowley wrote:
Okay, I've written and attached a fix for this. I'm not 100% certain
that this is the cause of the problem on pademelon, but the code does
look wrong, so needs to be fixed. Hopefully, it'll make pademelon
happy, if not I'll think a bit harder about what might be causing that
instability.
>>
>>> Pushed it just now. Let's see what happens with pademelon now.
>>
>> I've had pademelon's host running a "make installcheck" loop all day
>> trying to reproduce the problem. I haven't gotten a bite yet (although
>> at 15+ minutes per cycle, this isn't a huge number of tests). I think
>> we were remarkably (un)lucky to see the problem so quickly after the
>> initial commit, and I'm afraid pademelon isn't going to help us prove
>> much about whether this was the same issue.
>>
>> This does remind me quite a bit though of the ongoing saga with the
>> postgres_fdw test instability. Given the frequency with which that's
>> failing in the buildfarm, you would not think it's impossible to
>> reproduce outside the buildfarm, and yet I'm here to tell you that
>> it's pretty damn hard. I haven't succeeded yet, and that's not for
>> lack of trying. Could there be something about the buildfarm
>> environment that makes these sorts of things more likely?
>
> coypu just demonstrated that this was not the cause of the problem [1]
>
> I'll study the code a bit more and see if I can think why this might
> be happening.
>
> [1]
> https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=coypu&dt=2018-04-11%2004%3A17%3A38&stg=install-check-C
I've spent a bit of time tonight trying to dig into this problem to
see if I can figure out what's going on.
I ended up running the following script on both a Linux x86_64 machine
and also a power8 machine.
#!/bin/bash
for x in {1..1000}
do
echo "$x";
for i in {1..1000}
do
psql -d postgres -f test.sql -o test.out
diff -u test.out test.expect
done
done
I was unable to recreate this problem after about 700k loops on the
Linux machine and 130k loops on the power8.
I've emailed the owner of coypu to ask if it would be possible to get
access to the machine, or have him run the script to see if it does
actually fail. Currently waiting to hear back.
I've made another pass over the nodeAppend.c code and I'm unable to
see what might cause this, although I did discover a bug where
first_partial_plan is not set taking into account that some subplans
may have been pruned away during executor init. The only thing I think
this would cause is for parallel workers to not properly help out with
some partial plans if some earlier subplans were pruned. I can see no
reason for this to have caused this particular issue since the
first_partial_plan would be 0 with and without the attached fix.
Tom, would there be any chance you could run the above script for a
while on pademelon to see if it can in fact reproduce the problem?
coypu did show this problem in the install check, so I don't think it
will need the other concurrent tests to fail. If you can recreate,
after adjusting the expected output, does the problem still exist in
5c0675215?
I also checked with other tests perform an EXPLAIN ANALYZE of a plan
with a Parallel Append and I see there's none. So I've not ruled out
that this is an existing bug. git grep "explain.*analyze" also does
not show much outside of the partition_prune tests either.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
set enable_indexonlyscan = off;
set parallel_setup_cost = 0;
set parallel_tuple_cost = 0;
set min_parallel_table_scan_size = 0;
set max_parallel_workers_per_gather = 2;
prepare ab_q5 (int, int, int) as
select avg(a) from ab where a in($1,$2,$3) and b < 4;
execute ab_q5 (1, 2, 3);
execute ab_q5 (1, 2, 3);
execute ab_q5 (1, 2, 3);
execute ab_q5 (1, 2, 3);
execute ab_q5 (1, 2, 3);
explain (analyze, costs off, summary off, timing off) execute ab_q5 (2, 3, 3);
test.expect
Description: Binary data
first_partial_plan_fix.patch
Description: Binary data
create table ab (a int not null, b int not null) partition by list (a);
create table ab_a2 partition of ab for values in(2) partition by list (b);
create table ab_a2_b1 partition of ab_a2 for values in (1);
create table ab_a2_b2 partition of ab_a2 for values in (2);
create table ab_a2_b3 partition of ab_a2 for values in (3);
create table ab_a1 partition of ab for values in(1) partition by list (b);
create table ab_a1_b1 partition of ab_a1 for values in (1);
create table ab_a1_b2 partition of ab_a1 for values in (2);
create table ab_a1_b3 partition of ab_a1 for values in (3);
create table ab_a3 partition of ab for values in(3) partition by list (b);
create table ab_a3_b1 partition of ab_a3 for values in (1);
create table ab_a3_b2 partiti
pgsql: Cleanup covering infrastructure
Cleanup covering infrastructure - Explicitly forbids opclass, collation and indoptions (like DESC/ASC etc) for including columns. Throw an error if user points that. - Truncated storage arrays for such attributes to store only key atrributes, added assertion checks. - Do not check opfamily and collation for including columns in CompareIndexInfo() Discussion: https://www.postgresql.org/message-id/5ee72852-3c4e-ee35-e2ed-c1d053d45...@sigaev.ru Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/c266ed31a8a3beed3533e6a78faeca78234cbd43 Modified Files -- src/backend/catalog/index.c | 22 ++ src/backend/commands/indexcmds.c | 44 +-- src/backend/optimizer/path/indxpath.c | 40 +++ src/backend/optimizer/util/plancat.c | 4 ++-- src/backend/parser/parse_utilcmd.c| 3 --- src/backend/utils/adt/ruleutils.c | 6 ++--- src/backend/utils/adt/selfuncs.c | 2 ++ src/backend/utils/cache/relcache.c| 8 +++ 8 files changed, 90 insertions(+), 39 deletions(-)
pgsql: Fix interference between covering indexes and partitioned tables
Fix interference between covering indexes and partitioned tables The bug is caused due to the original IndexStmt that DefineIndex receives being overwritten when processing the INCLUDE columns. Use separate list of index params to propagate to child tables. Add tests covering this case. Amit Langote and Alexander Korotkov. Re-commit 5c6110c6a960ad6fe1b0d0fec6ae36ef4eb913f5 because it discovered a bug fixed in c266ed31a8a3beed3533e6a78faeca78234cbd43 Discussion: https://www.postgresql.org/message-id/CAJGNTeO%3DBguEyG8wxMpU_Vgvg3nGGzy71zUQ0RpzEn_mb0bSWA%40mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/524054598fd300c75007f53aebd67f9ced33b7db Modified Files -- src/backend/commands/indexcmds.c | 27 ++- src/test/regress/expected/indexing.out | 24 src/test/regress/sql/indexing.sql | 19 +++ 3 files changed, 57 insertions(+), 13 deletions(-)
pgsql: Fix YA parallel-make hazard, this one in "make check" in plpytho
Fix YA parallel-make hazard, this one in "make check" in plpython. We have to ensure that submake-generated-headers is finished before the topmost make run launches any child makes. Discussion: https://postgr.es/m/20180411235843.gg32...@paquier.xyz Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/3e110a373b8102221af5436434441cd20eeb68fa Modified Files -- src/pl/plpython/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
pgsql: Use the right memory context for partkey's FmgrInfo
Use the right memory context for partkey's FmgrInfo
We were using CurrentMemoryContext to put the partsupfunc fmgr_info
into, which isn't right, because we want the PartitionKey as a whole to
be in the isolated Relation->rd_partkeycxt context. This can cause a
crash with user-defined support functions in the operator classes used
by partitioning keys. (Maybe this can cause problems with core-supplied
opclasses too, not sure.)
This is demonstrably broken in Postgres 10, too, but the initial
proposed fix runs afoul of a problem discussed back when 8a0596cb656e
("Get rid of copy_partition_key") reorganized that code: namely that it
is possible to jump out of RelationBuildPartitionKey because of some
error and leave a dangling memory context child of CacheMemoryContext.
Also, while reviewing this I noticed that the removed-in-pg11
copy_partition_key was doing something wrong, unfixed in pg10, namely
doing memcpy() on the FmgrInfo, which is bogus (should be doing
fmgr_info_copy). Therefore, in branch pg10, the sane fix seems to be to
backpatch both the aforementioned 8a0596cb656e and its followup
be2343221fb7 ("Protect against hypothetical memory leaks in
RelationGetPartitionKey"), so do that, then apply the fmgr_info memcxt
bugfix on top.
Add a test case exercising btree-based custom operator classes, which
causes a crash prior to this fix. This is not a security problem,
because in order to create an operator class you need superuser
privileges anyway.
Authors: Álvaro Herrera and Amit Langote
Reported and diagnosed by: Amit Langote
Discussion:
https://postgr.es/m/3041e853-b1dd-a0c6-ff21-7cc5633bf...@lab.ntt.co.jp
Branch
--
REL_10_STABLE
Details
---
https://git.postgresql.org/pg/commitdiff/5f11c6ec61a579d60347a5d13af7e42b17fadc56
Modified Files
--
src/backend/utils/cache/relcache.c | 109 -
src/include/access/hash.h | 2 +-
src/test/regress/expected/create_table.out | 14
src/test/regress/sql/create_table.sql | 15
4 files changed, 60 insertions(+), 80 deletions(-)
pgsql: Use the right memory context for partkey's FmgrInfo
Use the right memory context for partkey's FmgrInfo
We were using CurrentMemoryContext to put the partsupfunc fmgr_info
into, which isn't right, because we want the PartitionKey as a whole to
be in the isolated Relation->rd_partkeycxt context. This can cause a
crash with user-defined support functions in the operator classes used
by partitioning keys. (Maybe this can cause problems with core-supplied
opclasses too, not sure.)
This is demonstrably broken in Postgres 10, too, but the initial
proposed fix runs afoul of a problem discussed back when 8a0596cb656e
("Get rid of copy_partition_key") reorganized that code: namely that it
is possible to jump out of RelationBuildPartitionKey because of some
error and leave a dangling memory context child of CacheMemoryContext.
Also, while reviewing this I noticed that the removed-in-pg11
copy_partition_key was doing something wrong, unfixed in pg10, namely
doing memcpy() on the FmgrInfo, which is bogus (should be doing
fmgr_info_copy). Therefore, in branch pg10, the sane fix seems to be to
backpatch both the aforementioned 8a0596cb656e and its followup
be2343221fb7 ("Protect against hypothetical memory leaks in
RelationGetPartitionKey"), so do that, then apply the fmgr_info memcxt
bugfix on top.
Add a test case exercising btree-based custom operator classes, which
causes a crash prior to this fix. This is not a security problem,
because in order to create an operator class you need superuser
privileges anyway.
Authors: Álvaro Herrera and Amit Langote
Reported and diagnosed by: Amit Langote
Discussion:
https://postgr.es/m/3041e853-b1dd-a0c6-ff21-7cc5633bf...@lab.ntt.co.jp
Branch
--
master
Details
---
https://git.postgresql.org/pg/commitdiff/a4d56f583e7cff052c2699e62d867ae1c8fda4f3
Modified Files
--
src/backend/utils/cache/relcache.c | 2 +-
src/test/regress/expected/create_table.out | 16 ++--
src/test/regress/sql/create_table.sql | 17 +++--
3 files changed, 30 insertions(+), 5 deletions(-)
pgsql: YA attempt to stabilize the results of the postgres_fdw regressi
YA attempt to stabilize the results of the postgres_fdw regression test. We've made multiple attempts to stabilize the plans shown by commit 1bc0100d2, with little success so far. The reason for the remaining instability seems to be that if a transaction (such as auto-analyze) is running concurrently with the test, then get_actual_variable_range may return a maximum value for "T 1"."C 1" that's far away from the actual max, as a result of our having transiently inserted such a value earlier in the test. Because we use a non-MVCC snapshot to fetch the value (for performance reasons), the presence of other transactions can cause that function to return entries that are actually dead. To fix, use a less extreme value in the earlier transient insertion, so that whether it is visible or not won't affect the selectivity estimate. The use of there seems to have been picked with the aid of a dartboard anyway, rather than having a specific reason. Discussion: https://postgr.es/m/16962.1523551...@sss.pgh.pa.us Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/2fe977712c7375ccb1b6ddf7dfb234d0db903f16 Modified Files -- contrib/postgres_fdw/expected/postgres_fdw.out | 18 +- contrib/postgres_fdw/sql/postgres_fdw.sql | 12 ++-- 2 files changed, 15 insertions(+), 15 deletions(-)
pgsql: Add comment about default partition in check_new_partition_bound
Add comment about default partition in check_new_partition_bound The intention of the test is not immediately obvious, so we need this much. Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/181ccbb5e49cdc628e0c8334a9ed57dbc736efe8 Modified Files -- src/backend/catalog/partition.c | 6 ++ 1 file changed, 6 insertions(+)
pgsql: Revert lowering of lock level for ATTACH PARTITION
Revert lowering of lock level for ATTACH PARTITION I lowered the lock level for partitions being scanned from AccessExclusive to ShareLock in the course of 72cf7f310c07, but that was bogus, as pointed out by Robert Haas. Revert that bit. Doing this is possible, but requires more work. Discussion: https://postgr.es/m/ca+tgmobv7nfmqv+tzxcdssb9bjc-ol-anv6bnmcbfjvzlyp...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/b8ca984b2c739e096c08f260f477792495f4dfe4 Modified Files -- src/backend/commands/tablecmds.c | 23 +++ 1 file changed, 11 insertions(+), 12 deletions(-)
pgsql: Fix bogus affix-merging code.
Fix bogus affix-merging code. NISortAffixes() compared successive compound affixes incorrectly, thus possibly failing to merge identical affixes, or (less likely) merging ones that shouldn't be merged. The user-visible effects of this are unclear, to me anyway. Per bug #15150 from Alexander Lakhin. It's been broken for a long time, so back-patch to all supported branches. Arthur Zakirov Discussion: https://postgr.es/m/152353327780.31225.13445405496721177...@wrigleys.postgresql.org Branch -- REL9_6_STABLE Details --- https://git.postgresql.org/pg/commitdiff/0f439c8dd28935bf9600722c1a25534add9390f0 Modified Files -- src/backend/tsearch/spell.c | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-)
pgsql: Fix bogus affix-merging code.
Fix bogus affix-merging code. NISortAffixes() compared successive compound affixes incorrectly, thus possibly failing to merge identical affixes, or (less likely) merging ones that shouldn't be merged. The user-visible effects of this are unclear, to me anyway. Per bug #15150 from Alexander Lakhin. It's been broken for a long time, so back-patch to all supported branches. Arthur Zakirov Discussion: https://postgr.es/m/152353327780.31225.13445405496721177...@wrigleys.postgresql.org Branch -- REL_10_STABLE Details --- https://git.postgresql.org/pg/commitdiff/40132187ed6a48ab9d0a8f926bb722b864665954 Modified Files -- src/backend/tsearch/spell.c | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-)
pgsql: Fix bogus affix-merging code.
Fix bogus affix-merging code. NISortAffixes() compared successive compound affixes incorrectly, thus possibly failing to merge identical affixes, or (less likely) merging ones that shouldn't be merged. The user-visible effects of this are unclear, to me anyway. Per bug #15150 from Alexander Lakhin. It's been broken for a long time, so back-patch to all supported branches. Arthur Zakirov Discussion: https://postgr.es/m/152353327780.31225.13445405496721177...@wrigleys.postgresql.org Branch -- REL9_3_STABLE Details --- https://git.postgresql.org/pg/commitdiff/ac8ea0f27f3f1c3f537b8a4d136fa7f76bc520dc Modified Files -- src/backend/tsearch/spell.c | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-)
pgsql: Fix bogus affix-merging code.
Fix bogus affix-merging code. NISortAffixes() compared successive compound affixes incorrectly, thus possibly failing to merge identical affixes, or (less likely) merging ones that shouldn't be merged. The user-visible effects of this are unclear, to me anyway. Per bug #15150 from Alexander Lakhin. It's been broken for a long time, so back-patch to all supported branches. Arthur Zakirov Discussion: https://postgr.es/m/152353327780.31225.13445405496721177...@wrigleys.postgresql.org Branch -- REL9_4_STABLE Details --- https://git.postgresql.org/pg/commitdiff/f71d803c8de8daa7219ea52366ad3f4ff9345b3f Modified Files -- src/backend/tsearch/spell.c | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-)
pgsql: Fix bogus affix-merging code.
Fix bogus affix-merging code. NISortAffixes() compared successive compound affixes incorrectly, thus possibly failing to merge identical affixes, or (less likely) merging ones that shouldn't be merged. The user-visible effects of this are unclear, to me anyway. Per bug #15150 from Alexander Lakhin. It's been broken for a long time, so back-patch to all supported branches. Arthur Zakirov Discussion: https://postgr.es/m/152353327780.31225.13445405496721177...@wrigleys.postgresql.org Branch -- REL9_5_STABLE Details --- https://git.postgresql.org/pg/commitdiff/906e44d4dc83d551d26e52902435f18d9fa5cfc7 Modified Files -- src/backend/tsearch/spell.c | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-)
pgsql: Fix bogus affix-merging code.
Fix bogus affix-merging code. NISortAffixes() compared successive compound affixes incorrectly, thus possibly failing to merge identical affixes, or (less likely) merging ones that shouldn't be merged. The user-visible effects of this are unclear, to me anyway. Per bug #15150 from Alexander Lakhin. It's been broken for a long time, so back-patch to all supported branches. Arthur Zakirov Discussion: https://postgr.es/m/152353327780.31225.13445405496721177...@wrigleys.postgresql.org Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/65a69dfa08e212556d11e44a5a8a1861fd826ccd Modified Files -- src/backend/tsearch/spell.c | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-)
