Re: pgsql: Support partition pruning at execution time

2018-04-07 Thread Alvaro Herrera
Andrew Gierth wrote: > > "David" == David Rowley writes: > > David> I've attached my proposed fix for the unstable regression tests. > David> I removed the vacuums I'd added in the last version and > David> commented why we're doing set enable_indesonlyscan = off; > > Looks basically sane

pgsql: Attempt to stabilize partition_prune test output.

2018-04-07 Thread Andrew Gierth
Attempt to stabilize partition_prune test output. Disable index-only scan for tests that might report variable results for "Heap Fetches" statistic due to concurrent transactions affecting whether all-visible flags can be set. Author: David Rowley Discussion: https://postgr.es/m/cakjs1f_yjthdjnd

Re: pgsql: Support partition pruning at execution time

2018-04-07 Thread Andrew Gierth
> "David" == David Rowley writes: David> I've attached my proposed fix for the unstable regression tests. David> I removed the vacuums I'd added in the last version and David> commented why we're doing set enable_indesonlyscan = off; Looks basically sane - I'll try it out and commit it sh

pgsql: Support index INCLUDE in the AM properties interface.

2018-04-07 Thread Andrew Gierth
Support index INCLUDE in the AM properties interface. This rectifies an oversight in commit 8224de4f4, by adding a new property 'can_include' for pg_indexam_has_property, and adjusting the results of pg_index_column_has_property to give more appropriate results for INCLUDEd columns. Branch --

Re: pgsql: Support partition pruning at execution time

2018-04-07 Thread David Rowley
On 8 April 2018 at 15:34, Andrew Gierth wrote: > You can't ever assume that data you just inserted will become > all-visible just because you just vacuumed the table, unless you know > that there is NO concurrent activity that might have a snapshot (and no > other possible reason why OldestXmin mi

Re: pgsql: Support partition pruning at execution time

2018-04-07 Thread Andrew Gierth
> "David" == David Rowley writes: >> Setting autovacuum_naptime to 10 seconds makes it occur in 10 second >> intervals... David> Ok, I thought it might have been some concurrent vacuum on the David> table but the only tables I see being vacuumed are system David> tables. It's not vacuu

Re: pgsql: Support partition pruning at execution time

2018-04-07 Thread David Rowley
On 8 April 2018 at 15:21, Andrew Gierth wrote: > David> Setting autovacuum_naptime to 10 seconds makes it occur in 10 > David> second intervals... > > Analyze (including auto-analyze on a different table entirely) has a > snapshot, which can hold back OldestXmin, hence preventing the > all-visib

Re: pgsql: Support partition pruning at execution time

2018-04-07 Thread David Rowley
On 8 April 2018 at 15:02, David Rowley wrote: > On 8 April 2018 at 14:56, David Rowley wrote: >> It happens 12 or 13 times on my machine, then does not happen again >> for 60 seconds, then happens again. > > Setting autovacuum_naptime to 10 seconds makes it occur in 10 second > intervals... Ok,

Re: pgsql: Support partition pruning at execution time

2018-04-07 Thread Andrew Gierth
> "David" == David Rowley writes: >> It happens 12 or 13 times on my machine, then does not happen again >> for 60 seconds, then happens again. David> Setting autovacuum_naptime to 10 seconds makes it occur in 10 David> second intervals... Analyze (including auto-analyze on a different

Re: pgsql: Indexes with INCLUDE columns and their support in B-tree

2018-04-07 Thread Andrey Borodin
> 8 апр. 2018 г., в 1:51, Teodor Sigaev написал(а): > > Ooops, sorry, if it possible, I'd like to change list of reviewers to add > you, but I don't know how do it. > > Nevertheless, thank you very much for your work > > BTW, I miss Andrey Borodin in that list too... No problem :) Thanks for

Re: pgsql: Support partition pruning at execution time

2018-04-07 Thread David Rowley
On 8 April 2018 at 14:56, David Rowley wrote: > It happens 12 or 13 times on my machine, then does not happen again > for 60 seconds, then happens again. Setting autovacuum_naptime to 10 seconds makes it occur in 10 second intervals... -- David Rowley http://www.2ndQuadrant

Re: pgsql: Support partition pruning at execution time

2018-04-07 Thread David Rowley
On 8 April 2018 at 12:15, Alvaro Herrera wrote: > Yeah, I don't quite understand this problem, and I tend to agree that > it likely isn't this patch's fault. However, for the moment I'm going > to avoid pushing the patch you propose because maybe there's a bug > elsewhere and it'd be good to unde

pgsql: Remove overzeleous assertions in pg_atomic_flag code.

2018-04-07 Thread Andres Freund
Remove overzeleous assertions in pg_atomic_flag code. The atomics code asserts proper alignment in various places. That's mainly because the alignment of 64bit integers is not sufficient for atomic operations on all platforms. Some ABIs only have four byte alignment, but don't have atomic behavior

pgsql: Remove overzeleous assertions in pg_atomic_flag code.

2018-04-07 Thread Andres Freund
Remove overzeleous assertions in pg_atomic_flag code. The atomics code asserts proper alignment in various places. That's mainly because the alignment of 64bit integers is not sufficient for atomic operations on all platforms. Some ABIs only have four byte alignment, but don't have atomic behavior

pgsql: Remove overzeleous assertions in pg_atomic_flag code.

2018-04-07 Thread Andres Freund
Remove overzeleous assertions in pg_atomic_flag code. The atomics code asserts proper alignment in various places. That's mainly because the alignment of 64bit integers is not sufficient for atomic operations on all platforms. Some ABIs only have four byte alignment, but don't have atomic behavior

pgsql: Remove overzeleous assertions in pg_atomic_flag code.

2018-04-07 Thread Andres Freund
Remove overzeleous assertions in pg_atomic_flag code. The atomics code asserts proper alignment in various places. That's mainly because the alignment of 64bit integers is not sufficient for atomic operations on all platforms. Some ABIs only have four byte alignment, but don't have atomic behavior

Re: pgsql: Support partition pruning at execution time

2018-04-07 Thread Alvaro Herrera
Yeah, I don't quite understand this problem, and I tend to agree that it likely isn't this patch's fault. However, for the moment I'm going to avoid pushing the patch you propose because maybe there's a bug elsewhere and it'd be good to understand it. I'm looking at it now. If others would prefe

Re: pgsql: Support partition pruning at execution time

2018-04-07 Thread David Rowley
On 8 April 2018 at 11:26, David Rowley wrote: > On 8 April 2018 at 10:59, David Rowley wrote: >> Sometimes I see: >> >> relname | relallvisible >> -+--- >> tprt_1 | 0 >> tprt_2 | 1 >> >> Other times I see: >> >> relname | relallvisible >>

Re: pgsql: Support partition pruning at execution time

2018-04-07 Thread David Rowley
On 8 April 2018 at 10:59, David Rowley wrote: > Sometimes I see: > > relname | relallvisible > -+--- > tprt_1 | 0 > tprt_2 | 1 > > Other times I see: > > relname | relallvisible > -+--- > tprt_1 | 0 > tprt_2 |

pgsql: Fix EXEC BACKEND + Windows builds for group privs

2018-04-07 Thread Stephen Frost
Fix EXEC BACKEND + Windows builds for group privs Under EXEC BACKEND we also need to be going through the group privileges setup since we do support that on Unixy systems, so add that to SubPostmasterMain(). Under Windows, we need to simply return true from GetDataDirectoryCreatePerm(), but that

Re: pgsql: Support partition pruning at execution time

2018-04-07 Thread David Rowley
On 8 April 2018 at 09:57, Tom Lane wrote: > Alvaro Herrera writes: >> Support partition pruning at execution time > > Buildfarm member lapwing doesn't like this. I can reproduce the > failures here by setting force_parallel_mode = regress. Kind > of looks like instrumentation counts aren't gett

Re: pgsql: Allow group access on PGDATA

2018-04-07 Thread Stephen Frost
Peter, * Peter Geoghegan (p...@bowt.ie) wrote: > On Sat, Apr 7, 2018 at 2:46 PM, Stephen Frost wrote: > > Allow group access on PGDATA > > > > Allow the cluster to be optionally init'd with read access for the > > group. > > Looks like this broke prion and culicidae. On culicidae, clearly the >

Re: pgsql: Allow group access on PGDATA

2018-04-07 Thread Tom Lane
Peter Geoghegan writes: > On Sat, Apr 7, 2018 at 2:46 PM, Stephen Frost wrote: >> Allow group access on PGDATA > Looks like this broke prion and culicidae. prion's problem seems to be ye same old postgres_fdw plan instability. (We really need to buckle down and find the cause of that. I have a

Re: pgsql: Allow group access on PGDATA

2018-04-07 Thread Peter Geoghegan
On Sat, Apr 7, 2018 at 2:46 PM, Stephen Frost wrote: > Allow group access on PGDATA > > Allow the cluster to be optionally init'd with read access for the > group. Looks like this broke prion and culicidae. On culicidae, clearly the problem is here: """ Upgrade Complete Optimize

Re: pgsql: Support partition pruning at execution time

2018-04-07 Thread Tom Lane
Alvaro Herrera writes: > Support partition pruning at execution time Buildfarm member lapwing doesn't like this. I can reproduce the failures here by setting force_parallel_mode = regress. Kind of looks like instrumentation counts aren't getting propagated from workers back to the leader?

pgsql: Refactor dir/file permissions

2018-04-07 Thread Stephen Frost
Refactor dir/file permissions Consolidate directory and file create permissions for tools which work with the PG data directory by adding a new module (common/file_perm.c) that contains variables (pg_file_create_mode, pg_dir_create_mode) and constants to initialize them (0600 for files and 0700 fo

pgsql: Allow group access on PGDATA

2018-04-07 Thread Stephen Frost
Allow group access on PGDATA Allow the cluster to be optionally init'd with read access for the group. This means a relatively non-privileged user can perform a backup of the cluster without requiring write privileges, which enhances security. The mode of PGDATA is used to determine whether grou

Re: pgsql: Support partition pruning at execution time

2018-04-07 Thread David Rowley
On 8 April 2018 at 09:02, Alvaro Herrera wrote: > Support partition pruning at execution time I'm looking at buildfarm member lapwing's failure [1] now. Probably it can be fixed by adding a vacuum, but will need a few mins to test and produce a patch. [1] https://buildfarm.postgresql.org/cgi-b

Re: pgsql: Indexes with INCLUDE columns and their support in B-tree

2018-04-07 Thread Erik Rijkers
On 2018-04-07 22:51, Teodor Sigaev wrote: Ooops, sorry, if it possible, I'd like to change list of reviewers to add you, but I don't know how do it. No problem, really. Thanks, great feature! I'm glad it got in.

pgsql: Add bms_prev_member function

2018-04-07 Thread Alvaro Herrera
Add bms_prev_member function This works very much like the existing bms_last_member function, only it traverses through the Bitmapset in the opposite direction from the most significant bit down to the least significant bit. A special prevbit value of -1 may be used to have the function determine

pgsql: Support partition pruning at execution time

2018-04-07 Thread Alvaro Herrera
Support partition pruning at execution time Existing partition pruning is only able to work at plan time, for query quals that appear in the parsed query. This is good but limiting, as there can be parameters that appear later that can be usefully used to further prune partitions. This commit ad

Re: pgsql: Indexes with INCLUDE columns and their support in B-tree

2018-04-07 Thread Teodor Sigaev
Ooops, sorry, if it possible, I'd like to change list of reviewers to add you, but I don't know how do it. Nevertheless, thank you very much for your work BTW, I miss Andrey Borodin in that list too... Erik Rijkers wrote: On 2018-04-07 22:01, Teodor Sigaev wrote: Indexes with INCLUDE columns

Re: pgsql: Indexes with INCLUDE columns and their support in B-tree

2018-04-07 Thread Erik Rijkers
On 2018-04-07 22:01, Teodor Sigaev wrote: Indexes with INCLUDE columns and their support in B-tree Author: Anastasia Lubennikova with contribition by Alexander Korotkov and me Reviewed by: Peter Geoghegan, Tomas Vondra, Antonin Houska, Jeff Janes, David Rowley, Alexan

pgsql: Raise error when affecting tuple moved into different partition.

2018-04-07 Thread Andres Freund
Raise error when affecting tuple moved into different partition. When an update moves a row between partitions (supported since 2f178441044b), our normal logic for following update chains in READ COMMITTED mode doesn't work anymore. Cross partition updates are modeled as an delete from the old and

pgsql: Indexes with INCLUDE columns and their support in B-tree

2018-04-07 Thread Teodor Sigaev
Indexes with INCLUDE columns and their support in B-tree This patch introduces INCLUDE clause to index definition. This clause specifies a list of columns which will be included as a non-key part in the index. The INCLUDE columns exist solely to allow more queries to benefit from index-only scan

Re: pgsql: Add json(b)_to_tsvector function

2018-04-07 Thread Tom Lane
Andres Freund writes: > On 2018-04-07 21:53:01 +0300, Teodor Sigaev wrote: >> test select_parallel ... FAILED >> >> I don't understand how it's connected to json_to_tsquery. Can somebody point >> me what I'm missing? > Given that it "only" failed during upgrade check, not the earlie

Re: pgsql: Add json(b)_to_tsvector function

2018-04-07 Thread Teodor Sigaev
Given that it "only" failed during upgrade check, not the earlier parallel check, it is possible that it's entirely unrelated and just a low likelihood event? May be, may be. So, we have bug with low probability. This however does clearly seem related: https://buildfarm.postgresql.org/cgi-bin/s

Re: pgsql: Add json(b)_to_tsvector function

2018-04-07 Thread Andres Freund
On 2018-04-07 21:53:01 +0300, Teodor Sigaev wrote: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=desmoxytes&dt=2018-04-07 > 18%3A32%3A02 > > test select_parallel ... FAILED > > I don't understand how it's connected to json_to_tsquery. Can somebody point > me what I'm miss

Re: pgsql: Add json(b)_to_tsvector function

2018-04-07 Thread Teodor Sigaev
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=desmoxytes&dt=2018-04-07 18%3A32%3A02 test select_parallel ... FAILED I don't understand how it's connected to json_to_tsquery. Can somebody point me what I'm missing? Teodor Sigaev wrote: Add json(b)_to_tsvector function

pgsql: Make test of json(b)_to_tsvector language-independ

2018-04-07 Thread Teodor Sigaev
Make test of json(b)_to_tsvector language-independ Missed in 1c1791e00065f6986f9d44a78ce7c28b2d1322dd commit Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/01bb85169afadfe63e2f0e344ff671292080de7e Modified Files -- src/test/regress/expected/json.out |

Re: pgsql: Add json(b)_to_tsvector function

2018-04-07 Thread Teodor Sigaev
Forget to add to test 'english' FTS configuration, will fix https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=dunlin&dt=2018-04-07%2018%3A05%3A17 Teodor Sigaev wrote: Add json(b)_to_tsvector function Jsonb has a complex nature so there isn't best-for-everything way to convert it to tsvect

pgsql: Add json(b)_to_tsvector function

2018-04-07 Thread Teodor Sigaev
Add json(b)_to_tsvector function Jsonb has a complex nature so there isn't best-for-everything way to convert it to tsvector for full text search. Current to_tsvector(json(b)) suggests to convert only string values, but it's possible to index keys, numerics and even booleans value. To solve that j

Re: pgsql: Fix timing issue in new subscription truncate test

2018-04-07 Thread Andres Freund
Hi, On 2018-04-07 13:31:53 -0400, Tom Lane wrote: > Oh ... digging into that, it's because that test depends on hstore, > which I hadn't installed. How much is that dependency really > buying us? There's some separate logic for non-core datatypes, so it seems good to test that. Hstore seems like

Re: pgsql: Fix timing issue in new subscription truncate test

2018-04-07 Thread Tom Lane
I wrote: > Testing the fix here, I just noticed that there's a pre-existing > problem in subscription/t/002_types.pl. It passes make check OK, > but make installcheck not so much. Investigating ... Oh ... digging into that, it's because that test depends on hstore, which I hadn't installed. How

Re: pgsql: Fix timing issue in new subscription truncate test

2018-04-07 Thread Tom Lane
Andres Freund writes: > On April 7, 2018 9:59:34 AM PDT, Peter Eisentraut wrote: >> We need to wait for the initial sync of all subscriptions. On >> some (faster?) machines, this didn't make a difference, but >> the (slower?) buildfarm machines are upset. > Failed on my very fast laptop fwiw.

Re: pgsql: Fix timing issue in new subscription truncate test

2018-04-07 Thread Andres Freund
On April 7, 2018 9:59:34 AM PDT, Peter Eisentraut wrote: >Fix timing issue in new subscription truncate test > >We need to wait for the initial sync of all subscriptions. On >some (faster?) machines, this didn't make a difference, but >the (slower?) buildfarm machines are upset. Failed on my v

pgsql: Fix timing issue in new subscription truncate test

2018-04-07 Thread Peter Eisentraut
Fix timing issue in new subscription truncate test We need to wait for the initial sync of all subscriptions. On some (faster?) machines, this didn't make a difference, but the (slower?) buildfarm machines are upset. Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/

pgsql: Deactive flapping checksum isolation tests.

2018-04-07 Thread Andres Freund
Deactive flapping checksum isolation tests. They've been broken for days, and prevent other tests from being run. The plan is to revert their addition later. Discussion: https://postgr.es/m/20180407162252.wfo5aorjrjw2n...@alap3.anarazel.de Branch -- master Details --- https://git.postg

Re: pgsql: Logical replication support for TRUNCATE

2018-04-07 Thread Tom Lane
Andres Freund writes: > On 2018-04-07 15:43:53 +, Peter Eisentraut wrote: >> Logical replication support for TRUNCATE > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=serinus&dt=2018-04-07%2015%3A46%3A01 > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2018-0

Re: pgsql: Logical replication support for TRUNCATE

2018-04-07 Thread Andres Freund
Hi, On 2018-04-07 15:43:53 +, Peter Eisentraut wrote: > Logical replication support for TRUNCATE > > Update the built-in logical replication system to make use of the > previously added logical decoding for TRUNCATE support. Add the > required truncate callback to pgoutput and a new logical

pgsql: Logical replication support for TRUNCATE

2018-04-07 Thread Peter Eisentraut
Logical replication support for TRUNCATE Update the built-in logical replication system to make use of the previously added logical decoding for TRUNCATE support. Add the required truncate callback to pgoutput and a new logical replication protocol message. Publications get a new attribute to de

pgsql: Logical decoding of TRUNCATE

2018-04-07 Thread Peter Eisentraut
Logical decoding of TRUNCATE Add a new WAL record type for TRUNCATE, which is only used when wal_level >= logical. (For physical replication, TRUNCATE is already replicated via SMGR records.) Add new callback for logical decoding output plugins to receive TRUNCATE actions. Author: Simon Riggs

pgsql: Predicate locking in hash indexes.

2018-04-07 Thread Teodor Sigaev
Predicate locking in hash indexes. Hash index searches acquire predicate locks on the primary page of a bucket. It acquires a lock on both the old and new buckets for scans that happen concurrently with page splits. During a bucket split, a predicate lock is copied from the primary page of an old

pgsql: Document partprune.c a little better

2018-04-07 Thread Alvaro Herrera
Document partprune.c a little better Author: Amit Langote Reviewed-by: Álvaro Herrera, David Rowley Discussion: https://postgr.es/m/CA+HiwqGzq4D6z=8r0ap+xhbtfcq-4ct+t2ekqje9fpm84_j...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/971d7ddbe19ad95254