pgsql: Move Perl test modules to a better namespace

2021-10-24 Thread Andrew Dunstan
Move Perl test modules to a better namespace

The five modules in our TAP test framework all had names in the top
level namespace. This is unwise because, even though we're not
exporting them to CPAN, the names can leak, for example if they are
exported by the RPM build process. We therefore move the modules to the
PostgreSQL::Test namespace. In the process PostgresNode is renamed to
Cluster, and TestLib is renamed to Utils. PostgresVersion becomes simply
PostgreSQL::Version, to avoid possible confusion about what it's the
version of.

Discussion: 
https://postgr.es/m/[email protected]

Reviewed by Erik Rijkers and Michael Paquier

Branch
--
master

Details
---
https://git.postgresql.org/pg/commitdiff/b3b4d8e68ae83f432f43f035c7eb481ef93e1583

Modified Files
--
contrib/amcheck/t/001_verify_heapam.pl |   6 +-
contrib/amcheck/t/002_cic.pl   |   6 +-
contrib/amcheck/t/003_cic_2pc.pl   |   6 +-
contrib/auto_explain/t/001_auto_explain.pl |   6 +-
contrib/bloom/t/001_wal.pl |   8 +-
contrib/oid2name/t/001_basic.pl|   2 +-
contrib/test_decoding/t/001_repl_stats.pl  |   6 +-
contrib/vacuumlo/t/001_basic.pl|   2 +-
src/bin/initdb/t/001_initdb.pl |   6 +-
src/bin/pg_amcheck/t/001_basic.pl  |   2 +-
src/bin/pg_amcheck/t/002_nonesuch.pl   |   6 +-
src/bin/pg_amcheck/t/003_check.pl  |   8 +-
src/bin/pg_amcheck/t/004_verify_heapam.pl  |   6 +-
src/bin/pg_amcheck/t/005_opclass_damage.pl |   6 +-
.../pg_archivecleanup/t/010_pg_archivecleanup.pl   |   4 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl   |  18 +--
src/bin/pg_basebackup/t/020_pg_receivewal.pl   |   6 +-
src/bin/pg_basebackup/t/030_pg_recvlogical.pl  |   6 +-
src/bin/pg_checksums/t/001_basic.pl|   2 +-
src/bin/pg_checksums/t/002_actions.pl  |   8 +-
src/bin/pg_config/t/001_pg_config.pl   |   2 +-
src/bin/pg_controldata/t/001_pg_controldata.pl |   6 +-
src/bin/pg_ctl/t/001_start_stop.pl |  14 +-
src/bin/pg_ctl/t/002_status.pl |   8 +-
src/bin/pg_ctl/t/003_promote.pl|  12 +-
src/bin/pg_ctl/t/004_logrotate.pl  |   6 +-
src/bin/pg_dump/t/001_basic.pl |   6 +-
src/bin/pg_dump/t/002_pg_dump.pl   |  10 +-
src/bin/pg_dump/t/003_pg_dump_with_server.pl   |   8 +-
src/bin/pg_dump/t/010_dump_connstr.pl  |  16 +--
src/bin/pg_resetwal/t/001_basic.pl |   6 +-
src/bin/pg_resetwal/t/002_corrupted.pl |   6 +-
src/bin/pg_rewind/t/001_basic.pl   |   2 +-
src/bin/pg_rewind/t/002_databases.pl   |   2 +-
src/bin/pg_rewind/t/003_extrafiles.pl  |   2 +-
src/bin/pg_rewind/t/004_pg_xlog_symlink.pl |   4 +-
src/bin/pg_rewind/t/005_same_timeline.pl   |   2 +-
src/bin/pg_rewind/t/006_options.pl |   6 +-
src/bin/pg_rewind/t/007_standby_source.pl  |  10 +-
src/bin/pg_rewind/t/008_min_recovery_point.pl  |  12 +-
src/bin/pg_rewind/t/RewindTest.pm  |  14 +-
src/bin/pg_test_fsync/t/001_basic.pl   |   2 +-
src/bin/pg_test_timing/t/001_basic.pl  |   2 +-
src/bin/pg_verifybackup/t/001_basic.pl |   4 +-
src/bin/pg_verifybackup/t/002_algorithm.pl |   6 +-
src/bin/pg_verifybackup/t/003_corruption.pl|  10 +-
src/bin/pg_verifybackup/t/004_options.pl   |   6 +-
src/bin/pg_verifybackup/t/005_bad_manifest.pl  |   6 +-
src/bin/pg_verifybackup/t/006_encoding.pl  |   6 +-
src/bin/pg_verifybackup/t/007_wal.pl   |   6 +-
src/bin/pg_waldump/t/001_basic.pl  |   2 +-
src/bin/pgbench/t/001_pgbench_with_server.pl   |  14 +-
src/bin/pgbench/t/002_pgbench_no_server.pl |   4 +-
src/bin/psql/t/001_basic.pl|   6 +-
src/bin/psql/t/010_tab_completion.pl   |   8 +-
src/bin/psql/t/020_cancel.pl   |  10 +-
src/bin/scripts/t/010_clusterdb.pl |   6 +-
src/bin/scripts/t/011_clusterdb_all.pl |   6 +-
src/bin/scripts/t/020_createdb.pl  |   6 +-
src/bin/scripts/t/040_createuser.pl|   6 +-
src/bin/scripts/t/050_dropdb.pl|   6 +-
src/bin/scripts/t/070_dropuser.pl  |   6 +-
src/bin/scripts/t/080_pg_isready.pl|   6 +-
src/bin/scripts/t/090_reindexdb.pl |   8 +-
src/bin/scripts/t/091_reindexdb_all.pl |   4 +-
src/bin/scripts/t/100_vacuumdb.pl  |   6 +-
src/bin/scripts/t/101_vacuumdb_all.pl  |   4 +-
src/bin/scripts/t/102_vacuumdb_stages.pl   |   4 +-
src/bin/scripts/t/200_connstr.pl   |   6 +-
src/test/authentication/t/001_pass

pgsql: Fix minor memory leaks in pg_dump.

2021-10-24 Thread Tom Lane
Fix minor memory leaks in pg_dump.

I found these by running pg_dump under "valgrind --leak-check=full".

The changes in flagInhIndexes() and getIndexes() replace allocation of
an array of which we use only some elements by individual allocations
of just the actually-needed objects.  The previous coding wasted some
memory, but more importantly it confused valgrind's leak tracking.

collectComments() and collectSecLabels() remain major blots on
the valgrind report, because they don't PQclear their query
results, in order to avoid a lot of strdup's.  That's a dubious
tradeoff, but I'll leave it alone here; an upcoming patch will
modify those functions enough to justify changing the tradeoff.

Branch
--
master

Details
---
https://git.postgresql.org/pg/commitdiff/70bef494000e4dbbeca0f0a40347ca1747aea701

Modified Files
--
src/bin/pg_dump/common.c | 42 
src/bin/pg_dump/pg_backup_archiver.c |  2 ++
src/bin/pg_dump/pg_dump.c| 63 +++-
3 files changed, 56 insertions(+), 51 deletions(-)



pgsql: Add replication command READ_REPLICATION_SLOT

2021-10-24 Thread Michael Paquier
Add replication command READ_REPLICATION_SLOT

The command is supported for physical slots for now, and returns the
type of slot, its restart_lsn and its restart_tli.

This will be useful for an upcoming patch related to pg_receivewal, to
allow the tool to be able to stream from the position of a slot, rather
than the last WAL position flushed by the backend (as reported by
IDENTIFY_SYSTEM) if the archive directory is found as empty, which would
be an advantage in the case of switching to a different archive
locations with the same slot used to avoid holes in WAL segment
archives.

Author: Ronan Dunklau
Reviewed-by: Kyotaro Horiguchi, Michael Paquier, Bharath Rupireddy
Discussion: https://postgr.es/m/18708360.4lzOvYHigE@aivenronan

Branch
--
master

Details
---
https://git.postgresql.org/pg/commitdiff/b4ada4e19fd7bedb433e46516ccd0ca4213d2719

Modified Files
--
doc/src/sgml/protocol.sgml  |  48 +
src/backend/replication/repl_gram.y |  16 -
src/backend/replication/repl_scanner.l  |   1 +
src/backend/replication/walsender.c | 106 
src/include/nodes/nodes.h   |   1 +
src/include/nodes/replnodes.h   |  11 +++
src/test/recovery/t/001_stream_rep.pl   |  32 -
src/test/recovery/t/006_logical_decoding.pl |  11 ++-
src/tools/pgindent/typedefs.list|   1 +
9 files changed, 224 insertions(+), 3 deletions(-)



pgsql: Clarify the logic in a few places in the new balanced merge code

2021-10-24 Thread Heikki Linnakangas
Clarify the logic in a few places in the new balanced merge code.

In selectnewtape(), use 'nOutputTapes' rather than 'nOutputRuns' in the
check for whether to start a new tape or to append a new run to an
existing tape. Until 'maxTapes' is reached, nOutputTapes is always equal
to nOutputRuns, so it doesn't change the logic, but it seems more logical
to compare # of tapes with # of tapes. Also, currently maxTapes is never
modified after the merging begins, but written this way, the code would
still work if it was. (Although the nOutputRuns == nOutputTapes assertion
would need to be removed and using nOutputRuns % nOutputTapes to
distribute the runs evenly across the tapes wouldn't do a good job
anymore).

Similarly in mergeruns(), change to USEMEM(state->tape_buffer_mem) to
account for the memory used for tape buffers. It's equal to availMem
currently, but tape_buffer_mem is more direct and future-proof. For
example, if we changed the logic to only allocate half of the remaining
memory to tape buffers, USEMEM(state->tape_buffer_mem) would still be
correct.

Coverity complained about these. Hopefully this patch helps it to
understand the logic better. Thanks to Tom Lane for initial analysis.

Branch
--
master

Details
---
https://git.postgresql.org/pg/commitdiff/166f94377c886516ca986ef8a623cd2e854fe911

Modified Files
--
src/backend/utils/sort/tuplesort.c | 12 +---
1 file changed, 9 insertions(+), 3 deletions(-)