pgsql: Modify additional power 2 calculations to use new helper functio

2020-04-07 Thread David Rowley
Modify additional power 2 calculations to use new helper functions 2nd pass of modifying various places which obtain the next power of 2 of a number and make them use the new functions added in f0705bb62. In passing, also modify num_combinations(). This can be implemented using simple bitshifting

pgsql: Fix crash when using COLLATE in partition bound expressions

2020-04-07 Thread Michael Paquier
Fix crash when using COLLATE in partition bound expressions Attempting to use a COLLATE clause with a type that it not collatable in a partition bound expression could crash the server. This commit fixes the code by adding more checks similar to what is done when computing index or partition attr

pgsql: Fix crash when using COLLATE in partition bound expressions

2020-04-07 Thread Michael Paquier
Fix crash when using COLLATE in partition bound expressions Attempting to use a COLLATE clause with a type that it not collatable in a partition bound expression could crash the server. This commit fixes the code by adding more checks similar to what is done when computing index or partition attr

Re: pgsql: Remove testing for precise LSN/reserved bytes in new TAP test

2020-04-07 Thread Kyotaro Horiguchi
At Wed, 08 Apr 2020 03:33:04 +, Alvaro Herrera wrote in > Remove testing for precise LSN/reserved bytes in new TAP test > > Trying to ensure that a slot's restart_lsn or amount of reserved bytes > exactly match some specific values seems unnecessary, and fragile as > shown by failures in mu

pgsql: Modify various power 2 calculations to use new helper functions

2020-04-07 Thread David Rowley
Modify various power 2 calculations to use new helper functions First pass of modifying various places that obtain the next power of 2 of a number and make them use the new functions added in pg_bitutils.h instead. This also removes the _hash_log2() function. There are no longer any callers in co

pgsql: Create memory context for HashAgg with a reasonable maxBlockSize

2020-04-07 Thread Jeff Davis
Create memory context for HashAgg with a reasonable maxBlockSize. If the memory context's maxBlockSize is too big, a single block allocation can suddenly exceed work_mem. For Hash Aggregation, this can mean spilling to disk too early or reporting a confusing memory usage number for EXPLAN ANALYZE.

Re: pgsql: Allow partitionwise joins in more cases.

2020-04-07 Thread Jeff Davis
On Wed, 2020-04-08 at 01:27 +, Etsuro Fujita wrote: > Allow partitionwise joins in more cases. This is causing a compiler warning for me if building without Asserts: partbounds.c: In function ‘partition_bounds_merge’: partbounds.c:1024:21: warning: unused variable ‘inner_binfo’ [-Wunused- var

pgsql: Add functions to calculate the next power of 2

2020-04-07 Thread David Rowley
Add functions to calculate the next power of 2 There are many areas in the code where we need to determine the next highest power of 2 of a given number. We tend to always do that in an ad-hoc way each time, generally with some tight for loop which performs a bitshift left once per loop and goes

pgsql: Put back mistakenly removed #include.

2020-04-07 Thread Tom Lane
Put back mistakenly removed #include. In commit 4dbcb3f84 I removed some code from parse_coerce.c, and also removed some apparently-no-longer-needed #includes. But removing datum.h broke some not-compiled-by-default code. Discussion: https://postgr.es/m/20200407205436.pyjhddw5bn5upvsu@developmen

pgsql: Remove testing for precise LSN/reserved bytes in new TAP test

2020-04-07 Thread Alvaro Herrera
Remove testing for precise LSN/reserved bytes in new TAP test Trying to ensure that a slot's restart_lsn or amount of reserved bytes exactly match some specific values seems unnecessary, and fragile as shown by failures in multiple buildfarm members. Discussion: https://postgr.es/m/20200407232602

pgsql: Support PrefetchBuffer() in recovery.

2020-04-07 Thread Thomas Munro
Support PrefetchBuffer() in recovery. Provide PrefetchSharedBuffer(), a variant that takes SMgrRelation, for use in recovery. Rename LocalPrefetchBuffer() to PrefetchLocalBuffer() for consistency. Add a return value to all of these. In recovery, tolerate and report missing files, so we can hand

Re: pgsql: Implement waiting for given lsn at transaction start

2020-04-07 Thread Michael Paquier
On Wed, Apr 08, 2020 at 06:07:23AM +0530, Amit Kapila wrote: > I think we have two options now (a) Provide feedback on the thread for > syntax and see what best we can do in that regard (b) Revert and try > it for PG14. I think generally building consensus on syntax at this > stage is difficult bu

pgsql: Allow partitionwise join to handle nested FULL JOIN USING cases.

2020-04-07 Thread Tom Lane
Allow partitionwise join to handle nested FULL JOIN USING cases. This case didn't work because columns merged by FULL JOIN USING are represented in the parse tree by COALESCE expressions, and the logic for recognizing a partitionable join failed to match upper-level join clauses to such expression

pgsql: Allow partitionwise joins in more cases.

2020-04-07 Thread Etsuro Fujita
Allow partitionwise joins in more cases. Previously, the partitionwise join technique only allowed partitionwise join when input partitioned tables had exactly the same partition bounds. This commit extends the technique to some cases when the tables have different partition bounds, by using an a

pgsql: Fix circle_in to accept "(x,y),r" as it's advertised to do.

2020-04-07 Thread Tom Lane
Fix circle_in to accept "(x,y),r" as it's advertised to do. Our documentation describes four allowed input syntaxes for circles, but the regression tests tried only three ... with predictable consequences. Remarkably, this has been wrong since the circle datatype was added in 1997, but nobody not

pgsql: Fix circle_in to accept "(x,y),r" as it's advertised to do.

2020-04-07 Thread Tom Lane
Fix circle_in to accept "(x,y),r" as it's advertised to do. Our documentation describes four allowed input syntaxes for circles, but the regression tests tried only three ... with predictable consequences. Remarkably, this has been wrong since the circle datatype was added in 1997, but nobody not

pgsql: Fix circle_in to accept "(x,y),r" as it's advertised to do.

2020-04-07 Thread Tom Lane
Fix circle_in to accept "(x,y),r" as it's advertised to do. Our documentation describes four allowed input syntaxes for circles, but the regression tests tried only three ... with predictable consequences. Remarkably, this has been wrong since the circle datatype was added in 1997, but nobody not

pgsql: Fix circle_in to accept "(x,y),r" as it's advertised to do.

2020-04-07 Thread Tom Lane
Fix circle_in to accept "(x,y),r" as it's advertised to do. Our documentation describes four allowed input syntaxes for circles, but the regression tests tried only three ... with predictable consequences. Remarkably, this has been wrong since the circle datatype was added in 1997, but nobody not

pgsql: Fix circle_in to accept "(x,y),r" as it's advertised to do.

2020-04-07 Thread Tom Lane
Fix circle_in to accept "(x,y),r" as it's advertised to do. Our documentation describes four allowed input syntaxes for circles, but the regression tests tried only three ... with predictable consequences. Remarkably, this has been wrong since the circle datatype was added in 1997, but nobody not

pgsql: Fix circle_in to accept "(x,y),r" as it's advertised to do.

2020-04-07 Thread Tom Lane
Fix circle_in to accept "(x,y),r" as it's advertised to do. Our documentation describes four allowed input syntaxes for circles, but the regression tests tried only three ... with predictable consequences. Remarkably, this has been wrong since the circle datatype was added in 1997, but nobody not

pgsql: snapshot scalability: Move delayChkpt from PGXACT to PGPROC.

2020-04-07 Thread Andres Freund
snapshot scalability: Move delayChkpt from PGXACT to PGPROC. The goal of separating hotly accessed per-backend data from PGPROC into PGXACT is to make accesses fast (GetSnapshotData() in particular). But delayChkpt is not actually accessed frequently; only when starting a checkpoint. As it is freq

Re: pgsql: Implement waiting for given lsn at transaction start

2020-04-07 Thread Amit Kapila
On Wed, Apr 8, 2020 at 5:34 AM David Steele wrote: > > On 4/7/20 7:32 PM, Alexander Korotkov wrote: > > On Wed, Apr 8, 2020 at 1:34 AM Anna Akenteva > > wrote: > >> On 2020-04-08 00:45, David G. Johnston wrote: > >>> > >>> While there is lots of discussion it ended up with the thread at > >>> "r

pgsql: Track SLRU page hits in SimpleLruReadPage_ReadOnly

2020-04-07 Thread Tomas Vondra
Track SLRU page hits in SimpleLruReadPage_ReadOnly SLRU page hits were tracked only in SimpleLruReadPage, but that's not enough because we may hit the page in SimpleLruReadPage_ReadOnly in which case we don't call SimpleLruReadPage at all. Reported-by: Kuntal Ghosh Discussion: https://postgr.es/m

Re: pgsql: Appease perlcritic

2020-04-07 Thread Michael Paquier
On Tue, Apr 07, 2020 at 11:12:14PM +, Alvaro Herrera wrote: > Appease perlcritic > > Food for the gods must always be found somehow, even when the land starves. You have made my day on this one, and that's just morning here. -- Michael signature.asc Description: PGP signature

pgsql: Fix XLogReader FD leak that makes backends unusable after 2PC us

2020-04-07 Thread Andres Freund
Fix XLogReader FD leak that makes backends unusable after 2PC usage. Before the fix every 2PC commit/abort leaked a file descriptor. As the files are opened using BasicOpenFile(), that quickly leads to the backend running out of file descriptors. Once enough 2PC abort/commit have caused enough FD

Re: pgsql: Implement waiting for given lsn at transaction start

2020-04-07 Thread David Steele
On 4/7/20 7:32 PM, Alexander Korotkov wrote: On Wed, Apr 8, 2020 at 1:34 AM Anna Akenteva wrote: On 2020-04-08 00:45, David G. Johnston wrote: While there is lots of discussion it ended up with the thread at "returned with feedback" (a month ago) and now we have a commit. There seems to be

Re: pgsql: Implement waiting for given lsn at transaction start

2020-04-07 Thread Alexander Korotkov
On Wed, Apr 8, 2020 at 12:27 AM Tom Lane wrote: > Alexander Korotkov writes: > > Implement waiting for given lsn at transaction start > > This commit adds following optional clause to BEGIN and START TRANSACTION > > commands. > > WAIT FOR LSN lsn [ TIMEOUT timeout ] > > This seems like a really

Re: pgsql: Implement waiting for given lsn at transaction start

2020-04-07 Thread Alexander Korotkov
On Wed, Apr 8, 2020 at 1:34 AM Anna Akenteva wrote: > On 2020-04-08 00:45, David G. Johnston wrote: > > > > While there is lots of discussion it ended up with the thread at > > "returned with feedback" (a month ago) and now we have a commit. > > There seems to be other relevant discussion not link

pgsql: Appease perlcritic

2020-04-07 Thread Alvaro Herrera
Appease perlcritic Food for the gods must always be found somehow, even when the land starves. Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/7e2ffb3885007183af7b161e046e126be2cfba12 Modified Files -- src/test/recovery/t/019_replslot_limit.pl | 1 + 1 f

pgsql: Remove nbtree BTreeTupleSetAltHeapTID() function.

2020-04-07 Thread Peter Geoghegan
Remove nbtree BTreeTupleSetAltHeapTID() function. Since heap TID is supposed to be just another key attribute to the implementation, it doesn't make much sense to have separate BTreeTupleSetNAtts() and BTreeTupleSetAltHeapTID() functions. Merge the two functions together. This slightly simplifie

pgsql: Allow users to limit storage reserved by replication slots

2020-04-07 Thread Alvaro Herrera
Allow users to limit storage reserved by replication slots Replication slots are useful to retain data that may be needed by a replication system. But experience has shown that allowing them to retain excessive data can lead to the primary failing because of running out of space. This new featur

Re: pgsql: Implement waiting for given lsn at transaction start

2020-04-07 Thread Anna Akenteva
On 2020-04-08 00:45, David G. Johnston wrote: While there is lots of discussion it ended up with the thread at "returned with feedback" (a month ago) and now we have a commit. There seems to be other relevant discussion not linked to. David J. Hello! Sorry about the confusion. This feature s

pgsql: Allow psql's \g and \gx commands to transiently change \pset opt

2020-04-07 Thread Tom Lane
Allow psql's \g and \gx commands to transiently change \pset options. We invented \gx to allow the "\pset expanded" flag to be forced on for the duration of one command output, but that turns out to not be nearly enough to satisfy the demand for variant output formats. Hence, make it possible to c

Re: pgsql: Implement waiting for given lsn at transaction start

2020-04-07 Thread David G. Johnston
On Tue, Apr 7, 2020 at 2:27 PM Tom Lane wrote: > Alexander Korotkov writes: > > Implement waiting for given lsn at transaction start > > This commit adds following optional clause to BEGIN and START TRANSACTION > > commands. > > WAIT FOR LSN lsn [ TIMEOUT timeout ] > > This seems like a really

Re: pgsql: Implement waiting for given lsn at transaction start

2020-04-07 Thread Tom Lane
Alexander Korotkov writes: > Implement waiting for given lsn at transaction start > This commit adds following optional clause to BEGIN and START TRANSACTION > commands. > WAIT FOR LSN lsn [ TIMEOUT timeout ] This seems like a really carelessly chosen syntax --- *three* new keywords, when you p

pgsql: Implement waiting for given lsn at transaction start

2020-04-07 Thread Alexander Korotkov
Implement waiting for given lsn at transaction start This commit adds following optional clause to BEGIN and START TRANSACTION commands. WAIT FOR LSN lsn [ TIMEOUT timeout ] New clause pospones transaction start till given lsn is applied on standby. This clause allows user be sure, that change

pgsql: Adjust bytea get_bit/set_bit to cope with bytea strings > 256MB.

2020-04-07 Thread Tom Lane
Adjust bytea get_bit/set_bit to cope with bytea strings > 256MB. Since the existing bit number argument can't exceed INT32_MAX, it's not possible for these functions to manipulate bits beyond the first 256MB of a bytea value. However, it'd be good if they could do at least that much, and not fall

pgsql: Adjust bytea get_bit/set_bit to cope with bytea strings > 256MB.

2020-04-07 Thread Tom Lane
Adjust bytea get_bit/set_bit to cope with bytea strings > 256MB. Since the existing bit number argument can't exceed INT32_MAX, it's not possible for these functions to manipulate bits beyond the first 256MB of a bytea value. However, it'd be good if they could do at least that much, and not fall

pgsql: Adjust bytea get_bit/set_bit to cope with bytea strings > 256MB.

2020-04-07 Thread Tom Lane
Adjust bytea get_bit/set_bit to cope with bytea strings > 256MB. Since the existing bit number argument can't exceed INT32_MAX, it's not possible for these functions to manipulate bits beyond the first 256MB of a bytea value. However, it'd be good if they could do at least that much, and not fall

pgsql: Adjust bytea get_bit/set_bit to cope with bytea strings > 256MB.

2020-04-07 Thread Tom Lane
Adjust bytea get_bit/set_bit to cope with bytea strings > 256MB. Since the existing bit number argument can't exceed INT32_MAX, it's not possible for these functions to manipulate bits beyond the first 256MB of a bytea value. However, it'd be good if they could do at least that much, and not fall

pgsql: Adjust bytea get_bit/set_bit to cope with bytea strings > 256MB.

2020-04-07 Thread Tom Lane
Adjust bytea get_bit/set_bit to cope with bytea strings > 256MB. Since the existing bit number argument can't exceed INT32_MAX, it's not possible for these functions to manipulate bits beyond the first 256MB of a bytea value. However, it'd be good if they could do at least that much, and not fall

pgsql: Support FETCH FIRST WITH TIES

2020-04-07 Thread Alvaro Herrera
Support FETCH FIRST WITH TIES WITH TIES is an option to the FETCH FIRST N ROWS clause (the SQL standard's spelling of LIMIT), where you additionally get rows that compare equal to the last of those N rows by the columns in the mandatory ORDER BY clause. There was a proposal by Andrew Gierth to im

pgsql: Adjust bytea get_bit/set_bit to use int8 not int4 for bit number

2020-04-07 Thread Tom Lane
Adjust bytea get_bit/set_bit to use int8 not int4 for bit numbering. Since the existing bit number argument can't exceed INT32_MAX, it's not possible for these functions to manipulate bits beyond the first 256MB of a bytea value. Lift that restriction by redeclaring the bit number arguments as in

pgsql: Remove debugging elog from pgstat_recv_resetslrucounter

2020-04-07 Thread Tomas Vondra
Remove debugging elog from pgstat_recv_resetslrucounter Reported-by: Thomas Munro Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/9c74ceb20b991f786f71666d4b4d557d2744a567 Modified Files -- src/backend/postmaster/pgstat.c | 2 -- 1 file changed, 2 deletio

Re: pgsql: Collect statistics about SLRU caches

2020-04-07 Thread Tomas Vondra
On Wed, Apr 08, 2020 at 04:54:40AM +1200, Thomas Munro wrote: Hi Tomas, In pgstat.c, + elog(LOG, "msg->m_index = %d", msg->m_index); Looks like unintentional debugging stuff? Yes, sorry about that. Will fix. regards -- Tomas Vondra http://www.2ndQuadrant.com Postgre

Re: pgsql: Collect statistics about SLRU caches

2020-04-07 Thread Thomas Munro
Hi Tomas, In pgstat.c, + elog(LOG, "msg->m_index = %d", msg->m_index); Looks like unintentional debugging stuff?

pgsql: Minor improvements in Incremental Sort explain

2020-04-07 Thread Tomas Vondra
Minor improvements in Incremental Sort explain Some places still used "Maximum" instead of "Peak" when displaying info about sort space, so fix that. Also, add a comment clarifying why it's correct to check the number of full/prefix sort groups. Author: James Coleman Discussion: https://postgr.e

Re: pgsql: Prevent archive recovery from scanning non-existent WAL files.

2020-04-07 Thread Fujii Masao
On Wed, Apr 8, 2020 at 1:36 AM Fujii Masao wrote: > > Prevent archive recovery from scanning non-existent WAL files. > > Previously when there were multiple timelines listed in the history file > of the recovery target timeline, archive recovery searched all of them, > starting from the newest tim

pgsql: Prevent archive recovery from scanning non-existent WAL files.

2020-04-07 Thread Fujii Masao
Prevent archive recovery from scanning non-existent WAL files. Previously when there were multiple timelines listed in the history file of the recovery target timeline, archive recovery searched all of them, starting from the newest timeline to the oldest one, to find the segment to read. That is,

pgsql: Consider Incremental Sort paths at additional places

2020-04-07 Thread Tomas Vondra
Consider Incremental Sort paths at additional places Commit d2d8a229bc introduced Incremental Sort, but it was considered only in create_ordered_paths() as an alternative to regular Sort. There are many other places that require sorted input and might benefit from considering Incremental Sort too.