pgsql: Add condition variable for recovery resume.

2021-03-11 Thread Thomas Munro
Add condition variable for recovery resume. Replace a sleep loop with a CV, to get a fast reaction time when recovery is resumed or the postmaster exits via standard infrastructure. Unfortunately we still need to wake up every second to perform extra polling during the recovery pause loop. Discus

pgsql: Add condition variable for walreceiver shutdown.

2021-03-11 Thread Thomas Munro
Add condition variable for walreceiver shutdown. Use this new CV to wait for walreceiver shutdown without a sleep/poll loop, while also benefiting from standard postmaster death handling. Discussion: https://postgr.es/m/CA%2BhUKGK1607VmtrDUHQXrsooU%3Dap4g4R2yaoByWOOA3m8xevUQ%40mail.gmail.com Br

pgsql: Poll postmaster less frequently in recovery.

2021-03-11 Thread Thomas Munro
Poll postmaster less frequently in recovery. Since commits 9f095299 and f98b8476 we don't poll the postmaster pipe at all during crash recovery on Linux and FreeBSD, but on other operating systems we were still doing it for every WAL record. Do it less frequently on operating systems where system

pgsql: Send statistics collected during shutdown checkpoint to the stat

2021-03-11 Thread Fujii Masao
Send statistics collected during shutdown checkpoint to the stats collector. When shutdown is requested, checkpointer performs checkpoint or restartpoint, and updates the statistics, before it exits. But previously checkpointer didn't send those statistics to the stats collector. Shutdown checkpo

pgsql: Force to send remaining WAL stats to the stats collector at walw

2021-03-11 Thread Fujii Masao
Force to send remaining WAL stats to the stats collector at walwriter exit. In walwriter's main loop, WAL stats message is only sent if enough time has passed since last one was sent to reach PGSTAT_STAT_INTERVAL msecs. This is necessary to avoid overloading to the stats collector. But this can ca

pgsql: Minor modernization for README.barrier.

2021-03-11 Thread Thomas Munro
Minor modernization for README.barrier. Itanium is very uncommon and being discontinued. ARM is everywhere. Prefer ARM as an example of an architecture with weak memory ordering. Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/43c66624964aa1d2f519ad6be0c5ea8f170cf3

pgsql: Save a few cycles during nbtree VACUUM.

2021-03-11 Thread Peter Geoghegan
Save a few cycles during nbtree VACUUM. Avoid calling RelationGetNumberOfBlocks() unnecessarily in the common case where there are no deleted but not yet recycled pages to recycle during a cleanup-only nbtree VACUUM operation. Follow-up to commit e5d8a999, which (among other things) taught the "s

Re: pgsql: Don't consider newly inserted tuples in nbtree VACUUM.

2021-03-11 Thread Alvaro Herrera
On 2021-Mar-11, Peter Geoghegan wrote: > On Thu, Mar 11, 2021 at 12:39 PM Alvaro Herrera > wrote: > > Given the short life of recheck_on_update I don't think we should > > consider it a precedent. > > Now we have a useful precedent. Apparently there is no way to truly > remove a storage paramet

Re: pgsql: Drop index behind pg_upgrade test issue.

2021-03-11 Thread Peter Geoghegan
On Thu, Mar 11, 2021 at 1:36 AM Laurenz Albe wrote: > I'd take that as an indication that Michael had a point with his > complaint[1]. Perhaps it would be better to implement one of his > ideas than to remove the test. The Postgres 11 and 12 "drop index" commits are now reverted. Thanks -- Pete

Re: pgsql: Don't consider newly inserted tuples in nbtree VACUUM.

2021-03-11 Thread Peter Geoghegan
On Thu, Mar 11, 2021 at 12:39 PM Alvaro Herrera wrote: > Given the short life of recheck_on_update I don't think we should > consider it a precedent. Now we have a useful precedent. Apparently there is no way to truly remove a storage parameter. That seems a little annoying, but not worth spendin

pgsql: Restore vacuum_cleanup_index_scale_factor coverage.

2021-03-11 Thread Peter Geoghegan
Restore vacuum_cleanup_index_scale_factor coverage. Revert two recent commits that had btree_index.sql drop regression test indexes rather than leave them behind for pg_dump testing. This is intended to restore pg_upgrade coverage of indexes with the vacuum_cleanup_index_scale_factor storage para

pgsql: Restore vacuum_cleanup_index_scale_factor coverage.

2021-03-11 Thread Peter Geoghegan
Restore vacuum_cleanup_index_scale_factor coverage. Revert two recent commits that had btree_index.sql drop regression test indexes rather than leave them behind for pg_dump testing. This is intended to restore pg_upgrade coverage of indexes with the vacuum_cleanup_index_scale_factor storage para

pgsql: Add back vacuum_cleanup_index_scale_factor parameter.

2021-03-11 Thread Peter Geoghegan
Add back vacuum_cleanup_index_scale_factor parameter. Commit 9f3665fb removed the vacuum_cleanup_index_scale_factor storage parameter. However, that creates dump/reload hazards when moving across major versions. Add back the vacuum_cleanup_index_scale_factor parameter (though not the GUC of the

Re: pgsql: Don't consider newly inserted tuples in nbtree VACUUM.

2021-03-11 Thread Alvaro Herrera
On 2021-Mar-11, Peter Geoghegan wrote: > On Thu, Mar 11, 2021 at 8:13 AM Alvaro Herrera > wrote: > > I disagree; that GUC was a feature in its own right, and it seems likely > > that people have set it in the hopes that it'd help them, even if it > > didn't actually achieve that. > > Michael wa

pgsql: Be clear about whether a recovery pause has taken effect.

2021-03-11 Thread Robert Haas
Be clear about whether a recovery pause has taken effect. Previously, the code and documentation seem to have essentially assumed than a call to pg_wal_replay_pause() would take place immediately, but that's not the case, because we only check for a pause in certain places. This means that a tool

pgsql: Re-simplify management of inStart in pqParseInput3's subroutines

2021-03-11 Thread Tom Lane
Re-simplify management of inStart in pqParseInput3's subroutines. Commit 92785dac2 copied some logic related to advancement of inStart from pqParseInput3 into getRowDescriptions and getAnotherTuple, because it wanted to allow user-defined row processor callbacks to potentially longjmp out of the l

pgsql: Re-simplify management of inStart in pqParseInput3's subroutines

2021-03-11 Thread Tom Lane
Re-simplify management of inStart in pqParseInput3's subroutines. Commit 92785dac2 copied some logic related to advancement of inStart from pqParseInput3 into getRowDescriptions and getAnotherTuple, because it wanted to allow user-defined row processor callbacks to potentially longjmp out of the l

pgsql: Re-simplify management of inStart in pqParseInput3's subroutines

2021-03-11 Thread Tom Lane
Re-simplify management of inStart in pqParseInput3's subroutines. Commit 92785dac2 copied some logic related to advancement of inStart from pqParseInput3 into getRowDescriptions and getAnotherTuple, because it wanted to allow user-defined row processor callbacks to potentially longjmp out of the l

pgsql: Re-simplify management of inStart in pqParseInput3's subroutines

2021-03-11 Thread Tom Lane
Re-simplify management of inStart in pqParseInput3's subroutines. Commit 92785dac2 copied some logic related to advancement of inStart from pqParseInput3 into getRowDescriptions and getAnotherTuple, because it wanted to allow user-defined row processor callbacks to potentially longjmp out of the l

pgsql: Re-simplify management of inStart in pqParseInput3's subroutines

2021-03-11 Thread Tom Lane
Re-simplify management of inStart in pqParseInput3's subroutines. Commit 92785dac2 copied some logic related to advancement of inStart from pqParseInput3 into getRowDescriptions and getAnotherTuple, because it wanted to allow user-defined row processor callbacks to potentially longjmp out of the l

pgsql: Re-simplify management of inStart in pqParseInput3's subroutines

2021-03-11 Thread Tom Lane
Re-simplify management of inStart in pqParseInput3's subroutines. Commit 92785dac2 copied some logic related to advancement of inStart from pqParseInput3 into getRowDescriptions and getAnotherTuple, because it wanted to allow user-defined row processor callbacks to potentially longjmp out of the l

Re: pgsql: Don't consider newly inserted tuples in nbtree VACUUM.

2021-03-11 Thread Peter Geoghegan
On Thu, Mar 11, 2021 at 11:25 AM Robert Haas wrote: > I can understand that those two settings might interact in some way > that is bad or unintended, but I feel like if I can't understand what > exactly the bad interaction is after reading the commit message, > that's probably a sign that the com

Re: pgsql: Don't consider newly inserted tuples in nbtree VACUUM.

2021-03-11 Thread Robert Haas
On Thu, Mar 11, 2021 at 2:22 PM Peter Geoghegan wrote: > It is convoluted. The issue that led to my backpatching to 13 was tied > to a 4% - 16% regression in an append-only workload by Mark Callaghan > (he is well known as the main person behind MyRocks at Facebook): > > https://smalldatum.blogspo

Re: pgsql: Don't consider newly inserted tuples in nbtree VACUUM.

2021-03-11 Thread Peter Geoghegan
On Thu, Mar 11, 2021 at 11:13 AM Robert Haas wrote: > On Thu, Mar 11, 2021 at 1:57 AM Michael Paquier wrote: > > I think that this commit has some issues that need more thoughts. > > My biggest issue with this commit is that I can't understand from the > commit message why it's a bug fix that des

Re: pgsql: Don't consider newly inserted tuples in nbtree VACUUM.

2021-03-11 Thread Robert Haas
On Thu, Mar 11, 2021 at 1:57 AM Michael Paquier wrote: > I think that this commit has some issues that need more thoughts. My biggest issue with this commit is that I can't understand from the commit message why it's a bug fix that deserves to be back-patched. To be honest, I can't even really un

pgsql: Refactor and generalize the ParallelSlot machinery.

2021-03-11 Thread Robert Haas
Refactor and generalize the ParallelSlot machinery. Create a wrapper object, ParallelSlotArray, to encapsulate the number of slots and the slot array itself, plus some other relevant bits of information. This reduces the number of parameters we have to pass around all over the place. Allow for a

Re: pgsql: Don't consider newly inserted tuples in nbtree VACUUM.

2021-03-11 Thread Peter Geoghegan
On Thu, Mar 11, 2021 at 9:03 AM Tom Lane wrote: > Agreed, just keep it as a no-op. Not sure whether there should be > a documentation entry for it, but I bet we get questions if there > isn't. I won't be adding back the psql completion stuff. Why can't adding back the reloption have the desired

Re: pgsql: Don't consider newly inserted tuples in nbtree VACUUM.

2021-03-11 Thread Tom Lane
Peter Geoghegan writes: > I now accept that the dump/reload hazards when upgrading to 14 are not > acceptable. ISTM that adding back the reloption on master/Postgres 14 > (without doing anything with it) is the only way to fix everything. Agreed, just keep it as a no-op. Not sure whether there s

Re: pgsql: Don't consider newly inserted tuples in nbtree VACUUM.

2021-03-11 Thread Peter Geoghegan
On Thu, Mar 11, 2021 at 8:13 AM Alvaro Herrera wrote: > I disagree; that GUC was a feature in its own right, and it seems likely > that people have set it in the hopes that it'd help them, even if it > didn't actually achieve that. Michael was talking about the reloption, not the GUC. Surely the

Re: pgsql: Drop index behind pg_upgrade test issue.

2021-03-11 Thread Peter Geoghegan
On Thu, Mar 11, 2021 at 8:27 AM Tom Lane wrote: > Strongly agree. I hadn't realized that the patch actually *removed* > the index reloption, rather than just turning it into a no-op. I think > that's a complete nonstarter. It will break pg_upgrade scenarios where > users have indexes with that

Re: pgsql: Drop index behind pg_upgrade test issue.

2021-03-11 Thread Tom Lane
Laurenz Albe writes: > On Thu, 2021-03-11 at 03:02 +, Peter Geoghegan wrote: >> Fix the test failure by dropping the index in question. > I'd take that as an indication that Michael had a point with his > complaint[1]. Perhaps it would be better to implement one of his > ideas than to remove

Re: pgsql: Don't consider newly inserted tuples in nbtree VACUUM.

2021-03-11 Thread Alvaro Herrera
On 2021-Mar-10, Peter Geoghegan wrote: > On Wed, Mar 10, 2021 at 10:57 PM Michael Paquier wrote: > > I think that this commit has some issues that need more thoughts. > > > > The removal of vacuum_cleanup_index_scale_factor means that any > > existing deployment of Postgres that includes at least

Re: pgsql: Drop index behind pg_upgrade test issue.

2021-03-11 Thread Laurenz Albe
On Thu, 2021-03-11 at 03:02 +, Peter Geoghegan wrote: > The vacuum_cleanup_index_scale_factor storage parameter was set in a > btree index that was previously left behind in the regression test > database. As a result, the index gets tested within pg_dump and > pg_restore tests, as well as pg_

pgsql: Set libcrypto callbacks for all connection threads in libpq

2021-03-11 Thread Michael Paquier
Set libcrypto callbacks for all connection threads in libpq Based on an analysis of the OpenSSL code with Jacob, moving to EVP for the cryptohash computations makes necessary the setup of the libcrypto callbacks that were getting set only for SSL connections, but not for connections without SSL.