pgsql: Add infrastructure to track WAL usage.

2020-04-03 Thread Amit Kapila
Add infrastructure to track WAL usage. This allows gathering the WAL generation statistics for each statement execution. The three statistics that we collect are the number of WAL records, the number of full page writes and the amount of WAL bytes generated. This helps the users who have write-i

pgsql: Include chunk overhead in hash table entry size estimate.

2020-04-03 Thread Jeff Davis
Include chunk overhead in hash table entry size estimate. Don't try to be precise about it, just use a constant 16 bytes of chunk overhead. Being smarter would require knowing the memory context where the chunk will be allocated, which is not known by all callers. Discussion: https://postgr.es/m

pgsql: Fix resource management bug with replication=database.

2020-04-03 Thread Robert Haas
Fix resource management bug with replication=database. Commit 0d8c9c1210c44b36ec2efcb223a1dfbe897a3661 allowed BASE_BACKUP to acquire a ResourceOwner without a transaction so that the backup manifest functionality could use a BufFile, but it overlooked the fact that when a walsender is used with r

pgsql: Be more careful about time_t vs. pg_time_t in basebackup.c.

2020-04-03 Thread Robert Haas
Be more careful about time_t vs. pg_time_t in basebackup.c. lapwing is complaining that about a call to pg_gmtime, saying that it "expected 'const pg_time_t *' but argument is of type 'time_t *'". I at first thought that the problem had someting to do with const, but Thomas Munro suggested that it

pgsql: pg_validatebackup: Fix 'make clean' to remove tmp_check.

2020-04-03 Thread Robert Haas
pg_validatebackup: Fix 'make clean' to remove tmp_check. Report by Tom Lane. Discussion: http://postgr.es/m/[email protected] Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/9f8f881caa0fabdf7ff46cc55a991ffeab39bd92 Modified Files -- src/bi

Re: pgsql: pg_validatebackup: Adjust TAP tests to placate perlcritic.

2020-04-03 Thread Robert Haas
On Fri, Apr 3, 2020 at 5:40 PM Andrew Dunstan wrote: > On 4/3/20 3:32 PM, Robert Haas wrote: > > pg_validatebackup: Adjust TAP tests to placate perlcritic. > > > > It seems that we have a policy that every Perl subroutine should > > end with an explicit "return", so add explicit "return" > > state

pgsql: pg_validatebackup: Adjust TAP tests to undo permissions change.

2020-04-03 Thread Robert Haas
pg_validatebackup: Adjust TAP tests to undo permissions change. It may be necessary to go further and remove this test altogether, but I'm going to try this fix first. It's not clear, at least to me, exactly how this is breaking buildfarm members, but it appears to be doing so. Branch -- mast

Re: pgsql: pg_validatebackup: Adjust TAP tests to placate perlcritic.

2020-04-03 Thread Andrew Dunstan
On 4/3/20 3:32 PM, Robert Haas wrote: > pg_validatebackup: Adjust TAP tests to placate perlcritic. > > It seems that we have a policy that every Perl subroutine should > end with an explicit "return", so add explicit "return" > statements to all the new subroutines added by my prior > commit 0d8c

pgsql: pg_validatebackup: Also use perl2host in TAP tests.

2020-04-03 Thread Robert Haas
pg_validatebackup: Also use perl2host in TAP tests. Second try at getting the buildfarm to be happy with 003_corrution.pl as added by commit 0d8c9c1210c44b36ec2efcb223a1dfbe897a3661. Per suggestion from Álvaro Herrera. Discussion: http://postgr.es/m/[email protected] Branch -

pgsql: Cosmetic improvements for code related to partitionwise join.

2020-04-03 Thread Tom Lane
Cosmetic improvements for code related to partitionwise join. Move have_partkey_equi_join and match_expr_to_partition_keys to relnode.c, since they're used only there. Refactor build_joinrel_partition_info to split out the code that fills the joinrel's partition key lists; this doesn't have any n

pgsql: pg_validatebackup: Use tempdir_short in TAP tests.

2020-04-03 Thread Robert Haas
pg_validatebackup: Use tempdir_short in TAP tests. The buildfarm is very unhappy right now because TAP test 003_corruption.pl uses TestLib::tempdir to generate the name of a temporary directory that is used as a tablespace name, and this results in a 'symbolic link target too long' error message o

pgsql: pg_validatebackup: Adjust TAP tests to placate perlcritic.

2020-04-03 Thread Robert Haas
pg_validatebackup: Adjust TAP tests to placate perlcritic. It seems that we have a policy that every Perl subroutine should end with an explicit "return", so add explicit "return" statements to all the new subroutines added by my prior commit 0d8c9c1210c44b36ec2efcb223a1dfbe897a3661. Per buildfar

pgsql: Generate backup manifests for base backups, and validate them.

2020-04-03 Thread Robert Haas
Generate backup manifests for base backups, and validate them. A manifest is a JSON document which includes (1) the file name, size, last modification time, and an optional checksum for each file backed up, (2) timelines and LSNs for whatever WAL will need to be replayed to make the backup consist

Re: pgsql: Include information on buffer usage during planning phase, in EX

2020-04-03 Thread Julien Rouhaud
On Fri, Apr 3, 2020 at 8:16 PM Fujii Masao wrote: > > On 2020/04/04 2:46, Tom Lane wrote: > > Fujii Masao writes: > >> Many thanks! Pushed. > > > > Looks like you didn't un-revert the main patch? > > Right now, pushed! I was waiting for my test to finish > without errors before commit. Anyway tha

Re: pgsql: Include information on buffer usage during planning phase, in EX

2020-04-03 Thread Fujii Masao
On 2020/04/04 2:46, Tom Lane wrote: Fujii Masao writes: Many thanks! Pushed. Looks like you didn't un-revert the main patch? Right now, pushed! I was waiting for my test to finish without errors before commit. Anyway thanks! Regards, -- Fujii Masao Advanced Computing Technology Center

pgsql: Include information on buffer usage during planning phase, in EX

2020-04-03 Thread Fujii Masao
Include information on buffer usage during planning phase, in EXPLAIN output, take two. When BUFFERS option is enabled, EXPLAIN command includes the information on buffer usage during each plan node, in its output. In addition to that, this commit makes EXPLAIN command include also the informatio

Re: pgsql: Improve handling of parameter differences in physical replicatio

2020-04-03 Thread Robert Haas
On Mon, Mar 30, 2020 at 2:10 PM Andres Freund wrote: > One important issue seems to me to be the size of the array that > TransactionIdIsInProgress() allocates: > /* > * If first time through, get workspace to remember main XIDs in. We > * malloc it permanently to avoid r

Re: pgsql: Include information on buffer usage during planning phase, in EX

2020-04-03 Thread Tom Lane
Fujii Masao writes: > Many thanks! Pushed. Looks like you didn't un-revert the main patch? regards, tom lane

pgsql: Fix bugs in gin_fuzzy_search_limit processing.

2020-04-03 Thread Tom Lane
Fix bugs in gin_fuzzy_search_limit processing. entryGetItem()'s three code paths each contained bugs associated with filtering the entries for gin_fuzzy_search_limit. The posting-tree path failed to advance "advancePast" after having decided to filter an item. If we ran out of items on the curre

pgsql: Fix bugs in gin_fuzzy_search_limit processing.

2020-04-03 Thread Tom Lane
Fix bugs in gin_fuzzy_search_limit processing. entryGetItem()'s three code paths each contained bugs associated with filtering the entries for gin_fuzzy_search_limit. The posting-tree path failed to advance "advancePast" after having decided to filter an item. If we ran out of items on the curre

pgsql: Fix bugs in gin_fuzzy_search_limit processing.

2020-04-03 Thread Tom Lane
Fix bugs in gin_fuzzy_search_limit processing. entryGetItem()'s three code paths each contained bugs associated with filtering the entries for gin_fuzzy_search_limit. The posting-tree path failed to advance "advancePast" after having decided to filter an item. If we ran out of items on the curre

pgsql: Fix bugs in gin_fuzzy_search_limit processing.

2020-04-03 Thread Tom Lane
Fix bugs in gin_fuzzy_search_limit processing. entryGetItem()'s three code paths each contained bugs associated with filtering the entries for gin_fuzzy_search_limit. The posting-tree path failed to advance "advancePast" after having decided to filter an item. If we ran out of items on the curre

pgsql: Fix bugs in gin_fuzzy_search_limit processing.

2020-04-03 Thread Tom Lane
Fix bugs in gin_fuzzy_search_limit processing. entryGetItem()'s three code paths each contained bugs associated with filtering the entries for gin_fuzzy_search_limit. The posting-tree path failed to advance "advancePast" after having decided to filter an item. If we ran out of items on the curre

pgsql: Fix bugs in gin_fuzzy_search_limit processing.

2020-04-03 Thread Tom Lane
Fix bugs in gin_fuzzy_search_limit processing. entryGetItem()'s three code paths each contained bugs associated with filtering the entries for gin_fuzzy_search_limit. The posting-tree path failed to advance "advancePast" after having decided to filter an item. If we ran out of items on the curre

Re: pgsql: Document color support

2020-04-03 Thread Robert Haas
On Sun, Mar 29, 2020 at 5:21 AM Peter Eisentraut wrote: > Document color support > > Add a documentation appendix that explains the PG_COLOR and PG_COLORS > environment variables. > > Discussion: > https://www.postgresql.org/message-id/flat/[email protected] Th

Re: pgsql: Include information on buffer usage during planning phase, in EX

2020-04-03 Thread Fujii Masao
On 2020/04/04 0:34, Tom Lane wrote: Fujii Masao writes: Attached is the patch that changes explain.c as follows (i.e., removes Buffers lines at all) to make it more stable. ln := regexp_replace(ln, '\m\d+\M', 'N', 'g'); -- In sort output, the above won't match units-

pgsql: Improve stability of explain regression test.

2020-04-03 Thread Fujii Masao
Improve stability of explain regression test. The explain regression test runs EXPLAIN commands via the function that filters unstable outputs. To produce more stable test output, this commit improves the function so that it also filters out text-mode Buffers lines. This is necessary because text-

pgsql: Add a glossary to the documentation

2020-04-03 Thread Alvaro Herrera
Add a glossary to the documentation More work is still needed, but this is a good start. Co-authored-by: Corey Huinker Co-authored-by: Jürgen Purtz Co-authored-by: Roger Harkavy Co-authored-by: Álvaro Herrera Reviewed-by: Justin Pryzby Discussion: https://postgr.es/m/CADkLM=eP6HOeqDjn0FdXuG

pgsql: pg_waldump: Don't call XLogDumpDisplayStats() if -q is specified

2020-04-03 Thread Robert Haas
pg_waldump: Don't call XLogDumpDisplayStats() if -q is specified. Commit ac44367efbef198c57a18b96dbc6a39191720994 introduced this problem. Report and fix by Fujii Masao. Discussion: http://postgr.es/m/[email protected] Branch -- master Details --- ht

pgsql: Add checksum helper functions.

2020-04-03 Thread Robert Haas
Add checksum helper functions. These functions make it easier to write code that wants to compute a checksum for some data while allowing the user to configure the type of checksum that gets used. This is another piece of infrastructure for the upcoming patch to add backup manifests. Patch writt

Re: pgsql: pg_waldump: Add a --quiet option.

2020-04-03 Thread Robert Haas
On Thu, Apr 2, 2020 at 9:41 PM Fujii Masao wrote: > On 2020/04/03 10:11, Robert Haas wrote: > > pg_waldump: Add a --quiet option. > > > > The primary motivation for this change is that it will be used by the > > upcoming patch to add backup manifests, but it also seems to have some > > potential m

Re: pgsql: Include information on buffer usage during planning phase, in EX

2020-04-03 Thread Tom Lane
Fujii Masao writes: > Attached is the patch that changes explain.c as follows (i.e., removes > Buffers lines at all) to make it more stable. > ln := regexp_replace(ln, '\m\d+\M', 'N', 'g'); > -- In sort output, the above won't match units-suffixed numbers > ln := reg

pgsql: Fix bogus CALLED_AS_TRIGGER() defenses.

2020-04-03 Thread Tom Lane
Fix bogus CALLED_AS_TRIGGER() defenses. contrib/lo's lo_manage() thought it could use trigdata->tg_trigger->tgname in its error message about not being called as a trigger. That naturally led to a core dump. unique_key_recheck() figured it could Assert that fcinfo->context is a TriggerData node

pgsql: Fix bogus CALLED_AS_TRIGGER() defenses.

2020-04-03 Thread Tom Lane
Fix bogus CALLED_AS_TRIGGER() defenses. contrib/lo's lo_manage() thought it could use trigdata->tg_trigger->tgname in its error message about not being called as a trigger. That naturally led to a core dump. unique_key_recheck() figured it could Assert that fcinfo->context is a TriggerData node

pgsql: Fix bogus CALLED_AS_TRIGGER() defenses.

2020-04-03 Thread Tom Lane
Fix bogus CALLED_AS_TRIGGER() defenses. contrib/lo's lo_manage() thought it could use trigdata->tg_trigger->tgname in its error message about not being called as a trigger. That naturally led to a core dump. unique_key_recheck() figured it could Assert that fcinfo->context is a TriggerData node

pgsql: Fix bogus CALLED_AS_TRIGGER() defenses.

2020-04-03 Thread Tom Lane
Fix bogus CALLED_AS_TRIGGER() defenses. contrib/lo's lo_manage() thought it could use trigdata->tg_trigger->tgname in its error message about not being called as a trigger. That naturally led to a core dump. unique_key_recheck() figured it could Assert that fcinfo->context is a TriggerData node

pgsql: Fix bogus CALLED_AS_TRIGGER() defenses.

2020-04-03 Thread Tom Lane
Fix bogus CALLED_AS_TRIGGER() defenses. contrib/lo's lo_manage() thought it could use trigdata->tg_trigger->tgname in its error message about not being called as a trigger. That naturally led to a core dump. unique_key_recheck() figured it could Assert that fcinfo->context is a TriggerData node

pgsql: Fix bogus CALLED_AS_TRIGGER() defenses.

2020-04-03 Thread Tom Lane
Fix bogus CALLED_AS_TRIGGER() defenses. contrib/lo's lo_manage() thought it could use trigdata->tg_trigger->tgname in its error message about not being called as a trigger. That naturally led to a core dump. unique_key_recheck() figured it could Assert that fcinfo->context is a TriggerData node

Re: pgsql: Include information on buffer usage during planning phase, in EX

2020-04-03 Thread Fujii Masao
On 2020/04/03 23:55, Tom Lane wrote: Fujii Masao writes: On 2020/04/03 22:35, Tom Lane wrote: I think this is a bad idea. It's overcomplicated, and more to the point: now that we've seen the problem we should realize that we're eventually going to have failures for *any* Buffers line in tex

Re: pgsql: Include information on buffer usage during planning phase, in EX

2020-04-03 Thread Tom Lane
Fujii Masao writes: > On 2020/04/03 22:35, Tom Lane wrote: >> I think this is a bad idea. It's overcomplicated, and more to the >> point: now that we've seen the problem we should realize that we're >> eventually going to have failures for *any* Buffers line in text-mode >> output. We're already

Re: pgsql: Include information on buffer usage during planning phase, in EX

2020-04-03 Thread Fujii Masao
On 2020/04/03 22:35, Tom Lane wrote: Fujii Masao writes: On 2020/04/03 12:30, Tom Lane wrote: so probably the thing to do is adapt the filter functions in explain.sql so that they suppress Buffers lines altogether in text output. Kind of annoying, but ... I'm thinking to suppress only B

Re: pgsql: Include information on buffer usage during planning phase, in EX

2020-04-03 Thread Tom Lane
Fujii Masao writes: > On 2020/04/03 12:30, Tom Lane wrote: >> so probably the thing to do is adapt >> the filter functions in explain.sql so that they suppress Buffers lines >> altogether in text output. Kind of annoying, but ... > I'm thinking to suppress only Buffers line just after Planning T

Re: pgsql: Include information on buffer usage during planning phase, in EX

2020-04-03 Thread Julien Rouhaud
On Fri, Apr 03, 2020 at 09:27:38PM +0900, Fujii Masao wrote: > > > On 2020/04/03 16:43, Julien Rouhaud wrote: > > On Fri, Apr 03, 2020 at 03:24:41PM +0900, Fujii Masao wrote: > > > > > > > > > On 2020/04/03 12:30, Tom Lane wrote: > > > > Fujii Masao writes: > > > > > On 2020/04/03 11:58, Micha

Re: pgsql: Include information on buffer usage during planning phase, in EX

2020-04-03 Thread Fujii Masao
On 2020/04/03 16:43, Julien Rouhaud wrote: On Fri, Apr 03, 2020 at 03:24:41PM +0900, Fujii Masao wrote: On 2020/04/03 12:30, Tom Lane wrote: Fujii Masao writes: On 2020/04/03 11:58, Michael Paquier wrote: prion did not like that: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=pr

Re: pgsql: Include information on buffer usage during planning phase, in EX

2020-04-03 Thread Julien Rouhaud
On Fri, Apr 03, 2020 at 03:24:41PM +0900, Fujii Masao wrote: > > > On 2020/04/03 12:30, Tom Lane wrote: > > Fujii Masao writes: > > > On 2020/04/03 11:58, Michael Paquier wrote: > > > > prion did not like that: > > > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2020-04-03%2