pgsql: Refactor and cleanup runtime partition prune code a little
Refactor and cleanup runtime partition prune code a little * Move the execution pruning initialization steps that are common between both ExecInitAppend() and ExecInitMergeAppend() into a new function ExecInitPartitionPruning() defined in execPartition.c. Those steps include creation of a PartitionPruneState to be used for all instances of pruning and determining the minimal set of child subplans that need to be initialized by performing initial pruning if needed, and finally adjusting the subplan_map arrays in the PartitionPruneState to reflect the new set of subplans remaining after initial pruning if it was indeed performed. ExecCreatePartitionPruneState() is no longer exported out of execPartition.c and has been renamed to CreatePartitionPruneState() as a local sub-routine of ExecInitPartitionPruning(). * Likewise, ExecFindInitialMatchingSubPlans() that was in charge of performing initial pruning no longer needs to be exported. In fact, since it would now have the same body as the more generally named ExecFindMatchingSubPlans(), except differing in the value of initial_prune passed to the common subroutine find_matching_subplans_recurse(), it seems better to remove it and add an initial_prune argument to ExecFindMatchingSubPlans(). * Add an ExprContext field to PartitionPruneContext to remove the implicit assumption in the runtime pruning code that the ExprContext to use to compute pruning expressions that need one can always rely on the PlanState providing it. A future patch will allow runtime pruning (at least the initial pruning steps) to be performed without the corresponding PlanState yet having been created, so this will help. Author: Amit Langote Discussion: https://postgr.es/m/ca+hiwqeycpeqh2lmdop9mt+4-qove8hgfmkbjntemctzlpc...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/297daa9d43539fbf5fbb3c3a2cca190d0e3da471 Modified Files -- src/backend/executor/execPartition.c | 459 + src/backend/executor/nodeAppend.c | 41 +-- src/backend/executor/nodeMergeAppend.c | 34 +-- src/backend/partitioning/partprune.c | 24 +- src/include/executor/execPartition.h | 31 ++- src/include/partitioning/partprune.h | 2 + 6 files changed, 298 insertions(+), 293 deletions(-)
pgsql: Extend TAP tests of pg_dump to test for compression with gzip
Extend TAP tests of pg_dump to test for compression with gzip The test logic is extended with two new concepts: - Addition of a compression command called compress_cmd, executed between restore_cmd and dump_cmd to control the contents of the dumps. In the case of this commit, this is used to compress or decompress elements of a dump to test new code paths. - Addition of a new flag called compile_option, to check if a set of tests can be executed depending on the ./configure options used in a given build. The tests introduced here are for gzip, but they are designed so as they can easily be extended for new compression methods. Author: Georgios Kokolatos, Rachel Heaton Discussion: https://postgr.es/m/faUNEOpts9vunEaLnmxmG-DldLSg_ql137OC3JYDmgrOMHm1RvvWY2IdBkv_CRxm5spCCb_OmKNk2T03TMm0fBEWveFF9wA1WizPuAgB7Ss=@protonmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/98fe74218d97becb2a53581304c96091409fd929 Modified Files -- src/bin/pg_dump/Makefile | 2 + src/bin/pg_dump/t/002_pg_dump.pl | 93 +++- 2 files changed, 93 insertions(+), 2 deletions(-)
Re: [COMMITTERS] pgsql: Allow time delayed standbys and recovery
On Wed, 26 Feb 2014 at 13:55, Thom Brown wrote: > > On 26 February 2014 13:32, Simon Riggs wrote: >> >> On 26 February 2014 13:07, Thom Brown wrote: >> > On 19 December 2013 19:33, Simon Riggs wrote: >> >> >> >> On 19 December 2013 19:19, Thom Brown wrote: >> >> > On 12 December 2013 10:56, Simon Riggs wrote: >> >> >> Allow time delayed standbys and recovery >> >> >> >> >> >> Set min_recovery_apply_delay to force a delay in recovery apply for >> >> >> commit and >> >> >> restore point WAL records. Other records are replayed immediately. >> >> >> Delay is >> >> >> measured between WAL record time and local standby time. >> >> >> >> >> >> Robert Haas, Fabrízio de Royes Mello and Simon Riggs >> >> >> Detailed review by Mitsumasa Kondo >> >> > >> >> > Could a note be added about the additional disk space requirements for >> >> > WAL on the standby? If one is delaying replay, WAL files created over >> >> > the delayed period will build up in pg_xlog on the standby until >> >> > they're consumed upon replay, so considerations about storing those >> >> > additional files will need to be made. It may otherwise trip up >> >> > anyone who, say, delays replay by 24 hours in a busy production >> >> > environment, and finds they have run out of disk space and no longer >> >> > have a valid standby. >> >> >> >> Thanks Thom, good point. I will add a note to the docs. >> > >> > >> > Just a reminder that this note hasn't been added to the docs yet. >> >> Please cut a doc patch and I'll apply. > > > Attached. I know it's been 8 years, but I still think it would be a useful note to add to the docs. -- Thom
pgsql: pg_rewind: Fetch small files according to new size.
pg_rewind: Fetch small files according to new size. There's a race condition if a file changes in the source system after we have collected the file list. If the file becomes larger, we only fetched up to its original size. That can easily result in a truncated file. That's not a problem for relation files, files in pg_xact, etc. because any actions on them will be replayed from the WAL. However, configuration files are affected. This commit mitigates the race condition by fetching small files in whole, even if they have grown. A test is added in which an extra file copied is concurrently grown with the output of pg_rewind thus guaranteeing it to have changed in size during the operation. This is not a full fix: we still believe the original file size for files larger than 1 MB. That should be enough for configuration files, and doing more than that would require big changes to the chunking logic in libpq_source.c. This mitigates the race condition if the file is modified between the original scan of files and copying the file, but there's still a race condition if a file is changed while it's being copied. That's a much smaller window, though, and pg_basebackup has the same issue. This race can be seen with pg_auto_failover, which frequently uses ALTER SYSTEM, which updates postgresql.auto.conf. Often, pg_rewind will fail, because the postgresql.auto.conf file changed concurrently and a partial version of it was copied to the target. The partial file would fail to parse, preventing the server from starting up. Author: Heikki Linnakangas Reviewed-by: Cary Huang Discussion: https://postgr.es/m/f67feb24-5833-88cb-1020-19a4a2b83ac7%40iki.fi Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/16915126746e2d8597a92197a346fea0756f8e3e Modified Files -- src/bin/pg_rewind/libpq_source.c | 32 ++ src/bin/pg_rewind/local_source.c | 76 +++- src/bin/pg_rewind/pg_rewind.c| 5 +-- src/bin/pg_rewind/rewind_source.h| 13 ++ src/bin/pg_rewind/t/009_growing_files.pl | 76 5 files changed, 188 insertions(+), 14 deletions(-)
Re: pgsql: pg_rewind: Fetch small files according to new size.
> On 5 Apr 2022, at 15:02, Daniel Gustafsson wrote: > > pg_rewind: Fetch small files according to new size. The buildfarm is less impressed than CI was, I’m collecting more feedback and will then fix. / daniel
pgsql: Have VACUUM warn on relfrozenxid "in the future".
Have VACUUM warn on relfrozenxid "in the future". Commits 74cf7d46 and a61daa14 fixed pg_upgrade bugs involving oversights in how relfrozenxid or relminmxid are carried forward or initialized. Corruption caused by bugs of this nature was ameliorated by commit 78db307bb2, which taught VACUUM to always overwrite existing invalid relfrozenxid or relminmxid values that are apparently "in the future". Extend that work now by showing a warning in the event of overwriting either relfrozenxid or relminmxid due to an existing value that is "in the future". There is probably a decent chance that the sanity checks added by commit 699bf7d05c will raise an error before VACUUM reaches this point, but we shouldn't rely on that. Author: Peter Geoghegan Reviewed-By: Andres Freund Discussion: https://postgr.es/m/cah2-wzmrzezegvlv8ydw0abfmsvjjtziorqjvurf74ml4gl...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/e83ebfe6d767dafcefe00bc5f11392a3d6976c1b Modified Files -- src/backend/commands/vacuum.c | 70 --- 1 file changed, 52 insertions(+), 18 deletions(-)
pgsql: PLAN clauses for JSON_TABLE
PLAN clauses for JSON_TABLE These clauses allow the user to specify how data from nested paths are joined, allowing considerable freedom in shaping the tabular output of JSON_TABLE. PLAN DEFAULT allows the user to specify the global strategies when dealing with sibling or child nested paths. The is often sufficient to achieve the necessary goal, and is considerably simpler than the full PLAN clause, which allows the user to specify the strategy to be used for each named nested path. Nikita Glukhov Reviewers have included (in no particular order) Andres Freund, Alexander Korotkov, Pavel Stehule, Andrew Alsup, Erik Rijkers, Zhihong Yu, Himanshu Upadhyaya, Daniel Gustafsson, Justin Pryzby. Discussion: https://postgr.es/m/[email protected] Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/fadb48b00e02ccfd152baa80942de30205ab3c4f Modified Files -- src/backend/nodes/copyfuncs.c | 26 ++ src/backend/nodes/equalfuncs.c | 3 + src/backend/nodes/makefuncs.c | 19 + src/backend/nodes/outfuncs.c| 3 + src/backend/nodes/readfuncs.c | 3 + src/backend/parser/gram.y | 130 +- src/backend/parser/parse_jsontable.c| 323 +-- src/backend/utils/adt/jsonpath_exec.c | 118 -- src/backend/utils/adt/ruleutils.c | 50 +++ src/include/nodes/makefuncs.h | 2 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 42 ++ src/include/nodes/primnodes.h | 3 + src/include/parser/kwlist.h | 1 + src/test/regress/expected/jsonb_sqljson.out | 600 ++-- src/test/regress/sql/jsonb_sqljson.sql | 396 +- src/tools/pgindent/typedefs.list| 3 + 17 files changed, 1614 insertions(+), 109 deletions(-)
Re: pgsql: JSON_TABLE
On Tue, Apr 5, 2022 at 1:17 AM Erik Rijkers wrote: > > Op 04-04-2022 om 22:23 schreef Andrew Dunstan: > > JSON_TABLE > > Great that this is now committed! > > I notice one changed item: the NESTED-PATH-phrase does not accept an > alias anymore. The JSON_PATH v59 patches still had: > > | NESTED PATH json_path_specification [ AS path_name ] >COLUMNS ( json_table_column [, ...] ) > This is true. > My complaint is only half-hearted because I don't really understand what > the use of such nested-path aliases are. But it's a change from the https://github.com/obartunov/sqljsondoc/blob/master/README.jsonpath.md "Every path may be followed by a path name using an AS clause. Path names are identifiers and must be unique and don't coincide with the column names." SELECT jt.* FROM house, JSON_TABLE(js, '$.floor[*]' AS lvl COLUMNS ( level int, NESTED PATH '$.apt[*] ? (@.area > 1000)' AS big COLUMNS ( no int ) ) PLAN (lvl OUTER big) ) jt; > earlier patch, and the nested-path aliases are used too in the > 2017-03 'Technical report ISO/IEC TR 19075-6', which is as near I have > to a SQL Standard description. > > FWIW, I attach example sql+data from that .pdf from ISO (which is not > online anymore). > > > Thanks, > > Erik Rijkers > > > > > > > > > > > -- Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Re: pgsql: JSON_TABLE
On Tue, Apr 5, 2022 at 1:31 AM Andrew Dunstan wrote: > > > On 4/4/22 18:16, Erik Rijkers wrote: > > Op 04-04-2022 om 22:23 schreef Andrew Dunstan: > >> JSON_TABLE > > > > Great that this is now committed! > > > > I notice one changed item: the NESTED-PATH-phrase does not accept an > > alias anymore. The JSON_PATH v59 patches still had: > > > > | NESTED PATH json_path_specification [ AS path_name ] > > COLUMNS ( json_table_column [, ...] ) > > > > My complaint is only half-hearted because I don't really understand > > what the use of such nested-path aliases are. But it's a change from > > the earlier patch, and the nested-path aliases are used too in the > > 2017-03 'Technical report ISO/IEC TR 19075-6', which is as near I have > > to a SQL Standard description. > > > > FWIW, I attach example sql+data from that .pdf from ISO (which is not > > online anymore). > > > > > > These commits are being staggered. The last code patches will be > committed tomorrow. as for PostgreSQL 15devel-master/fadb48b00e aliases AS works SELECT jt.* FROM house, JSON_TABLE(js, '$.floor[*]' AS lvl COLUMNS ( level int, NESTED PATH '$.apt[*] ? (@.area > 1000)' AS big COLUMNS ( no int ) ) PLAN (lvl OUTER big) ) jt; level | no ---+ 1 | (null) 2 | (null) (2 rows) > > > cheers > > > andrew > > -- > Andrew Dunstan > EDB: https://www.enterprisedb.com > > > -- Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Re: pgsql: JSON_TABLE
On 4/5/22 15:05, Oleg Bartunov wrote: > On Tue, Apr 5, 2022 at 1:31 AM Andrew Dunstan wrote: >> >> On 4/4/22 18:16, Erik Rijkers wrote: >>> Op 04-04-2022 om 22:23 schreef Andrew Dunstan: JSON_TABLE >>> Great that this is now committed! >>> >>> I notice one changed item: the NESTED-PATH-phrase does not accept an >>> alias anymore. The JSON_PATH v59 patches still had: >>> >>> | NESTED PATH json_path_specification [ AS path_name ] >>> COLUMNS ( json_table_column [, ...] ) >>> >>> My complaint is only half-hearted because I don't really understand >>> what the use of such nested-path aliases are. But it's a change from >>> the earlier patch, and the nested-path aliases are used too in the >>> 2017-03 'Technical report ISO/IEC TR 19075-6', which is as near I have >>> to a SQL Standard description. >>> >>> FWIW, I attach example sql+data from that .pdf from ISO (which is not >>> online anymore). >>> >>> >> These commits are being staggered. The last code patches will be >> committed tomorrow. > as for PostgreSQL 15devel-master/fadb48b00e aliases AS works > > SELECT > jt.* > FROM > house, > JSON_TABLE(js, '$.floor[*]' AS lvl COLUMNS ( > level int, > NESTED PATH '$.apt[*] ? (@.area > 1000)' AS big COLUMNS ( > no int > ) > ) PLAN (lvl OUTER big) ) jt; > level | no > ---+ > 1 | (null) > 2 | (null) > (2 rows) Yeah, and I think that's the answer to Erik's question about why we need it, it's so you have a name you can refer to in the PLAN clause, as in you example. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
Re: pgsql: pg_rewind: Fetch small files according to new size.
> On 5 Apr 2022, at 15:36, Daniel Gustafsson wrote:
>
>> On 5 Apr 2022, at 15:02, Daniel Gustafsson
>> wrote:
>>
>> pg_rewind: Fetch small files according to new size.
>
> The buildfarm is less impressed than CI was, I’m collecting more feedback and
> will then fix.
Sorry for being slow, life took over and children with fever took priority.
The error in question was:
local_source.c:118:15: error: format specifies type 'unsigned long long' but
the argument has type 'size_t' (aka 'unsigned int') [-Werror,-Wformat]
srcpath, len, written_len);
~^
I'm running a fixup with casting to int and printing with %d (like how
pg_rewind.c:digestControlFile already does it for printing a size_t) through CI
just to be sure and will push once it's had a green run:
- pg_fatal("size of source file \"%s\" changed concurrently: "
UINT64_FORMAT " bytes expected, " UINT64_FORMAT " copied",
-srcpath, len, written_len);
+ pg_fatal("size of source file \"%s\" changed concurrently: %d
bytes expected, %d copied",
+srcpath, (int) len, (int) written_len);
--
Daniel Gustafsson https://vmware.com/
pgsql: Fix compilerwarning in logging size_t
Fix compilerwarning in logging size_t The pg_fatal log which included filesizes were using UINT64_FORMAT for the size_t variables, which failed on 32 bit buildfarm animals. Change to using plain int instead, which is in line with how digestControlFile is doing it already. Per buildfarm animals florican and lapwing. Discussion: https://postgr.es/m/[email protected] Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/75edb919613ee835e7680e40137e494c7856bcf9 Modified Files -- src/bin/pg_rewind/local_source.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
pgsql: Remove race condition in 022_crash_temp_files.pl test.
Remove race condition in 022_crash_temp_files.pl test. It's possible for the query that "waits for restart" to complete a successful iteration before the postmaster has noticed its SIGKILL'd child and begun the restart cycle. (This is a bit hard to believe perhaps, but it's been seen at least twice in the buildfarm, mainly on ancient platforms that likely have quirky schedulers.) To provide a more secure interlock, wait for the other session we're using to report that it's been forcibly shut down. Patch by me, based on a suggestion from Andres Freund. Back-patch to v14 where this test case came in. Discussion: https://postgr.es/m/[email protected] Branch -- REL_14_STABLE Details --- https://git.postgresql.org/pg/commitdiff/9a7229948c70945ca6ef0b36adfe61b74f4fdaf5 Modified Files -- src/test/recovery/t/022_crash_temp_files.pl | 34 - 1 file changed, 29 insertions(+), 5 deletions(-)
pgsql: Remove race condition in 022_crash_temp_files.pl test.
Remove race condition in 022_crash_temp_files.pl test. It's possible for the query that "waits for restart" to complete a successful iteration before the postmaster has noticed its SIGKILL'd child and begun the restart cycle. (This is a bit hard to believe perhaps, but it's been seen at least twice in the buildfarm, mainly on ancient platforms that likely have quirky schedulers.) To provide a more secure interlock, wait for the other session we're using to report that it's been forcibly shut down. Patch by me, based on a suggestion from Andres Freund. Back-patch to v14 where this test case came in. Discussion: https://postgr.es/m/[email protected] Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/e37ad5fa4df2319e26a7e779607130feae1a5029 Modified Files -- src/test/recovery/t/022_crash_temp_files.pl | 34 + 1 file changed, 30 insertions(+), 4 deletions(-)
pgsql: Change aggregated log format of pgbench.
Change aggregated log format of pgbench. Commit 4a39f87acd changed the aggregated log format. Problem is, now the explanatory paragraph for the log line in the document is too long. Also the log format included more optional columns, and it's harder to parse the log lines. This commit tries to solve the problems. - There's no optional log columns anymore. If a column is not meaningful with provided pgbench option, it will be presented as 0. - Reorder the log columns so that it's easier to parse them. - Adjust explanatory paragraph for the log line in the doc. Discussion: https://postgr.es/m/flat/202203280757.3tu4ovs3petm%40alvherre.pgsql Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/17a856d08bedeaec77be3f15572e01f553e9613f Modified Files -- doc/src/sgml/ref/pgbench.sgml | 64 +++ src/bin/pgbench/pgbench.c | 59 +++ 2 files changed, 82 insertions(+), 41 deletions(-)
pgsql: Improve comments for row filtering and toast interaction in logi
Improve comments for row filtering and toast interaction in logical replication. Reported-by: Antonin Houska Author: Amit Kapila Reviewed-by: Antonin Houska, Ajin Cherian Discussion: https://postgr.es/m/84638.1649152255@antos Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/2d09e44d309f64d3571f90f7620c9d924aecd010 Modified Files -- src/backend/replication/pgoutput/pgoutput.c | 9 + 1 file changed, 5 insertions(+), 4 deletions(-)
pgsql: Update Unicode data to CLDR 41
Update Unicode data to CLDR 41 No actual changes result. Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/376dc437de40bd17e99a37f72f88627a16d7f200 Modified Files -- src/Makefile.global.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
pgsql: Allow asynchronous execution in more cases.
Allow asynchronous execution in more cases. In commit 27e1f1456, create_append_plan() only allowed the subplan created from a given subpath to be executed asynchronously when it was an async-capable ForeignPath. To extend coverage, this patch handles cases when the given subpath includes some other Path types as well that can be omitted in the plan processing, such as a ProjectionPath directly atop an async-capable ForeignPath, allowing asynchronous execution in partitioned-scan/partitioned-join queries with non-Var tlist expressions and more UNION queries. Andrey Lepikhov and Etsuro Fujita, reviewed by Alexander Pyhalov and Zhihong Yu. Discussion: https://postgr.es/m/659c37a8-3e71-0ff2-394c-f04428c76f08%40postgrespro.ru Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/c2bb02bc2e858ba345b8b33f1f3a54628f719d93 Modified Files -- contrib/postgres_fdw/expected/postgres_fdw.out | 170 + contrib/postgres_fdw/sql/postgres_fdw.sql | 41 ++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c| 57 +++-- src/backend/optimizer/plan/setrefs.c | 18 ++- src/include/nodes/plannodes.h | 12 ++ src/include/optimizer/planmain.h | 1 + 9 files changed, 287 insertions(+), 15 deletions(-)
