Re: postgres_fdw: oddity in costing presorted foreign scans with local stats

2019-06-14 Thread Etsuro Fujita
On Mon, Jun 10, 2019 at 5:37 PM Etsuro Fujita wrote: > On Thu, Jun 6, 2019 at 5:58 PM Etsuro Fujita wrote: > > I made stricter an assertion test I added on retrieved_rows. Also, I > > did some editorialization further and added the commit message. > > Attached is an updated version of the

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-06-14 Thread Bruce Momjian
On Thu, Jun 13, 2019 at 09:14:13PM -0400, Bruce Momjian wrote: > On Mon, Jun 3, 2019 at 12:04:54PM -0400, Robert Haas wrote: > > > > What I'm talking about here is that it also has to be reasonably > > possible to write an encryption key command that does something > > useful. I don't have a

Re: pg_dump multi VALUES INSERT

2019-06-14 Thread Peter Eisentraut
Shouldn't the --rows-per-insert option also be available via pg_dumpall? All the other options for switching between COPY and INSERT are settable in pg_dumpall. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Minimal logical decoding on standbys

2019-06-14 Thread Amit Khandekar
On Wed, 12 Jun 2019 at 00:06, Alvaro Herrera wrote: > > On 2019-May-23, Andres Freund wrote: > > > On 2019-05-23 09:37:50 -0400, Robert Haas wrote: > > > On Thu, May 23, 2019 at 9:30 AM Sergei Kornilov wrote: > > > > > wal_level is PGC_POSTMASTER. > > > > > > > > But primary can be restarted

Re: pg_dump multi VALUES INSERT

2019-06-14 Thread Alvaro Herrera
On 2019-Jun-14, Peter Eisentraut wrote: > Shouldn't the --rows-per-insert option also be available via pg_dumpall? > All the other options for switching between COPY and INSERT are > settable in pg_dumpall. Uh, yeah, absolutely. Surafel, are you in a position to provide a patch for that

Allow table AM's to cache stuff in relcache

2019-06-14 Thread Heikki Linnakangas
Index AM's can cache stuff in RelationData->rd_amcache. In the zedstore table AM we've been hacking on, I'd like to also use rd_amcache to cache some information, but that's not currently possible, because rd_amcache can only be used by index AMs, not table AMs. Attached patch allows

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-06-14 Thread Stephen Frost
Greetings, * Ian Barwick (ian.barw...@2ndquadrant.com) wrote: > Consider the following cascading standby setup with PostgreSQL 12: > > - there exists a running primary "A" > - standby "B" is cloned from primary "A" using "pg_basebackup > --write-recovery-conf" > - standby "C" is cloned from

Re: Allow table AM's to cache stuff in relcache

2019-06-14 Thread Tom Lane
Heikki Linnakangas writes: > Index AM's can cache stuff in RelationData->rd_amcache. In the zedstore > table AM we've been hacking on, I'd like to also use rd_amcache to cache > some information, but that's not currently possible, because rd_amcache > can only be used by index AMs, not table

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-14 Thread Tom Lane
Christoph Berg writes: > Re: Tom Lane 2019-06-11 <24452.1560285...@sss.pgh.pa.us> >> The only way I can get it to pick "Etc/UCT" is if that's what I put >> into /etc/localtime. (In which case I maintain that that's not a bug, >> or at least not our bug.) > Did you try a symlink or a plain file

Re: Converting NOT IN to anti-joins during planning

2019-06-14 Thread Li, Zheng
- The big "IF" here is if we can calculate the size of the subplan to know if it'll be hashed or not at the point in planning where this conversion is done. I personally can't quite see how that'll work reliably without actually planning the subquery, which I really doubt

Re: Index Skip Scan

2019-06-14 Thread Jesper Pedersen
Hi David, On 6/14/19 3:19 AM, David Rowley wrote: I read over this thread a few weeks ago while travelling back from PGCon. (I wish I'd read it on the outward trip instead since it would have been good to talk about it in person.) First off. I think this is a pretty great feature. It certainly

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-06-14 Thread Tomas Vondra
On Thu, Jun 13, 2019 at 11:38:12PM -0400, James Coleman wrote: On Wed, Jun 5, 2019 at 12:14 PM Rafia Sabih wrote: > > 2) Provide some fallback at execution time. For example, we might watch > > the size of the group, and if we run into an unexpectedly large one we > > might abandon the

Re: SQL/JSON path issues/questions

2019-06-14 Thread Alexander Korotkov
Hi! On Fri, Jun 14, 2019 at 10:16 AM Kyotaro Horiguchi wrote: > At Thu, 13 Jun 2019 14:59:51 +0100, Thom Brown wrote > in > > Hi, > > > > I've been reading through the documentation regarding jsonpath and > > jsonb_path_query etc., and I have found it lacking explanation for > > some

Re: CREATE STATISTICS documentation bug

2019-06-14 Thread Alvaro Herrera
On 2019-Jun-14, Tom Lane wrote: > Robert Haas writes: > > P.S. I think the fact that we print "psql: " before the ERROR here is > > useless clutter. We didn't do that in v11 and prior and I think we > > should kill it with fire. > > Agreed, particularly seeing that the error is not originating

Re: Binary support for pgoutput plugin

2019-06-14 Thread Dave Cramer
Dave Cramer On Fri, 14 Jun 2019 at 14:36, Tomas Vondra wrote: > On Wed, Jun 12, 2019 at 10:35:48AM -0400, Dave Cramer wrote: > >On Mon, 10 Jun 2019 at 07:49, Petr Jelinek > >wrote: > > > >> Hi, > >> > >> On 10/06/2019 13:27, Dave Cramer wrote: > >> > So back to binary output. > >> > > >> >

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-14 Thread Christopher Browne
On Fri, Jun 14, 2019, 3:12 PM Tom Lane wrote: > A possibly better idea is to push back on tzdb's choice to unify > these zones. Don't know if they'd listen, but we could try. The > UCT symlink hasn't been out there so long that it's got much inertia. One oddity; AIX had a preference for CUT

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-14 Thread Joe Conway
On 6/13/19 11:07 AM, Bruce Momjian wrote: > On Thu, Jun 13, 2019 at 04:26:47PM +0900, Masahiko Sawada wrote: >> Yeah, in principle since data key of 2 tier key architecture should >> not go outside database I think we should not tell data keys to >> utility commands. So the rearranging WAL format

Re: Binary support for pgoutput plugin

2019-06-14 Thread Tomas Vondra
On Wed, Jun 12, 2019 at 10:35:48AM -0400, Dave Cramer wrote: On Mon, 10 Jun 2019 at 07:49, Petr Jelinek wrote: Hi, On 10/06/2019 13:27, Dave Cramer wrote: > So back to binary output. > > From what I can tell the place to specify binary options would be in the > create publication and or in

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-14 Thread Tom Lane
Andrew Gierth writes: > This isn't good enough, because it still picks "UCT" on a system with no > /etc/localtime and no TZ variable. Testing on HEAD as of 3da73d683 (on > FreeBSD, but it'll be the same anywhere else): [ shrug... ] Too bad. I doubt that that's a common situation anyway. > We

Re: pg_dump multi VALUES INSERT

2019-06-14 Thread Fabien COELHO
Hello Alvaro, Shouldn't the --rows-per-insert option also be available via pg_dumpall? All the other options for switching between COPY and INSERT are settable in pg_dumpall. Uh, yeah, absolutely. Surafel, are you in a position to provide a patch for that quickly? End of the week, more

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-06-14 Thread Tomas Vondra
On Wed, Jun 05, 2019 at 06:14:14PM +0200, Rafia Sabih wrote: On Mon, 3 Jun 2019 at 15:39, James Coleman wrote: On Sun, Jun 2, 2019 at 5:18 PM Tomas Vondra wrote: > Currently, the costing logic (cost_incremental_sort) assumes all groups > have the same size, which is fine for uniform

CREATE STATISTICS documentation bug

2019-06-14 Thread Robert Haas
https://www.postgresql.org/docs/12/sql-createstatistics.html contains this example command: CREATE STATISTICS s2 (mcv) ON (a, b) FROM t2; But that produces: psql: ERROR: only simple column references are allowed in CREATE STATISTICS I think the parentheses around (a, b) just need to be

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-14 Thread Andrew Gierth
> "Tom" == Tom Lane writes: >>> Anyway, moving on to the question of what should we do about this, >>> I don't really have anything better to offer than back-patching >>> 23bd3cec6. >> The PG12 behavior seems sane, so +1. Tom> OK, I'll make that happen. This isn't good enough,

Re: CREATE STATISTICS documentation bug

2019-06-14 Thread Tom Lane
Robert Haas writes: > P.S. I think the fact that we print "psql: " before the ERROR here is > useless clutter. We didn't do that in v11 and prior and I think we > should kill it with fire. Agreed, particularly seeing that the error is not originating with psql; it's just passing it on.

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-14 Thread Andrew Gierth
> "Tom" == Tom Lane writes: >> This isn't good enough, because it still picks "UCT" on a system with no >> /etc/localtime and no TZ variable. Testing on HEAD as of 3da73d683 (on >> FreeBSD, but it'll be the same anywhere else): Tom> [ shrug... ] Too bad. I doubt that that's a common

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-14 Thread Andrew Gierth
> "Andrew" == Andrew Gierth writes: > "Tom" == Tom Lane writes: >>> This isn't good enough, because it still picks "UCT" on a system >>> with no /etc/localtime and no TZ variable. Testing on HEAD as of >>> 3da73d683 (on FreeBSD, but it'll be the same anywhere else): Tom> [ shrug...

Re: pg_dump multi VALUES INSERT

2019-06-14 Thread Alvaro Herrera
On 2019-Jun-14, Fabien COELHO wrote: > > Hello Alvaro, > > > > Shouldn't the --rows-per-insert option also be available via pg_dumpall? > > > All the other options for switching between COPY and INSERT are > > > settable in pg_dumpall. > > > > Uh, yeah, absolutely. > > > > Surafel, are you

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-14 Thread Joe Conway
On 6/14/19 6:09 PM, Bruce Momjian wrote: > On Fri, Jun 14, 2019 at 02:27:17PM -0400, Joe Conway wrote: >> On 6/13/19 11:07 AM, Bruce Momjian wrote: >> > In addition, while the 8k blocks would use a block cipher, the WAL would >> > likely use a stream cipher, and it will be very hard to use

Extracting only the columns needed for a query

2019-06-14 Thread Melanie Plageman
While hacking on zedstore, we needed to get a list of the columns to be projected--basically all of the columns needed to satisfy the query. The two use cases we have for this is 1) to pass this column list down to the AM layer for the AM to leverage it 2) for use during planning to improving

Re: CREATE STATISTICS documentation bug

2019-06-14 Thread Tom Lane
Alvaro Herrera writes: > On 2019-Jun-14, Tom Lane wrote: >> Robert Haas writes: >>> P.S. I think the fact that we print "psql: " before the ERROR here is >>> useless clutter. We didn't do that in v11 and prior and I think we >>> should kill it with fire. >> Agreed, particularly seeing that the

Re: fix psql \conninfo & \connect when using hostaddr

2019-06-14 Thread Alvaro Herrera
On 2019-May-27, Noah Misch wrote: > Other than that, the \connect behavior change makes sense to me. However, > nothing updated \connect documentation. (Even the commit log message said > nothing about \connect.) I added this blurb to the paragraph that documents the reusing behavior:

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-14 Thread Bruce Momjian
On Fri, Jun 14, 2019 at 02:27:17PM -0400, Joe Conway wrote: > On 6/13/19 11:07 AM, Bruce Momjian wrote: > > On Thu, Jun 13, 2019 at 04:26:47PM +0900, Masahiko Sawada wrote: > >> Yeah, in principle since data key of 2 tier key architecture should > >> not go outside database I think we should not

Re: Custom table AMs need to include heapam.h because of BulkInsertState

2019-06-14 Thread David Rowley
On Fri, 14 Jun 2019 at 07:53, Andres Freund wrote: > > On June 12, 2019 6:42:11 PM PDT, David Rowley > wrote: > >Do you see any issue with calling table_finish_bulk_insert() when the > >partition's CopyMultiInsertBuffer is evicted from the > >CopyMultiInsertInfo rather than at the end of the

Re: pgsql: Avoid spurious deadlocks when upgrading a tuple lock

2019-06-14 Thread Alvaro Herrera
On 2019-Jun-14, Tom Lane wrote: > I wrote: > >> Hm, I don't get that warning. Does this patch silence it, please? > > > Uh, no patch attached? But initializing the variable where it's > > declared would certainly silence it. > > BTW, after looking around a bit I wonder if this complaint isn't

Draft back-branch release notes are up for review

2019-06-14 Thread Tom Lane
I've committed first-draft release notes for next week's releases at https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=0995cefa74510ee0e38d1bf095b2eef2c1ea37c4 Please send any review comments by Sunday. regards, tom lane

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-14 Thread Andrew Gierth
> "Tom" == Tom Lane writes: >> This isn't good enough, because it still picks "UCT" on a system >> with no /etc/localtime and no TZ variable. Testing on HEAD as of >> 3da73d683 (on FreeBSD, but it'll be the same anywhere else): Tom> [ shrug... ] Too bad. I doubt that that's a common

Re: fix psql \conninfo & \connect when using hostaddr

2019-06-14 Thread Alvaro Herrera
On 2019-Jun-13, Fabien COELHO wrote: > Thanks for the pointer! I did not notice this one. At least the API looks > better than the one I was suggesting:-) > > ISTM that this function could be used to set other parameters, fixing some > other issues such as ignoring special parameters on

Re: Extracting only the columns needed for a query

2019-06-14 Thread David Rowley
On Sat, 15 Jun 2019 at 13:46, Melanie Plageman wrote: > > While hacking on zedstore, we needed to get a list of the columns to > be projected--basically all of the columns needed to satisfy the > query. The two use cases we have for this is > 1) to pass this column list down to the AM layer for

Re: SQL/JSON path issues/questions

2019-06-14 Thread Alexander Korotkov
On Thu, Jun 13, 2019 at 5:00 PM Thom Brown wrote: > I've been reading through the documentation regarding jsonpath and > jsonb_path_query etc., and I have found it lacking explanation for > some functionality, and I've also had some confusion when using the > feature. BTW, I've some general idea

Re: CREATE STATISTICS documentation bug

2019-06-14 Thread Alvaro Herrera
On 2019-Jun-14, Tom Lane wrote: > Alvaro Herrera writes: > > Proposal: each program declares at startup whether it wants the program > > name prefix or not. > > Well, to clarify: I think it's reasonable to include "psql: " if the > message is originating in psql. So I don't think your idea

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-06-14 Thread David Rowley
On Mon, 8 Apr 2019 at 04:09, Tom Lane wrote: > Also, I would not define "significantly bloated" as "the table has > grown at all". I think the threshold ought to be at least ~100 > buckets, if we're starting at 16. I've revised the patch to add a new constant named

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-06-14 Thread Amit Kapila
On Fri, Jun 14, 2019 at 9:38 PM Stephen Frost wrote: > * Ian Barwick (ian.barw...@2ndquadrant.com) wrote: > > > > Note this issue is not specific to pg_basebackup, primary_conninfo (or any > > other settings > > formerly in recovery.conf), it has just manifested itself as the built-in > >

Re: Converting NOT IN to anti-joins during planning

2019-06-14 Thread Simon Riggs
On Wed, 6 Mar 2019 at 04:11, David Rowley wrote: > Hi Jim, > > Thanks for replying here. > > On Wed, 6 Mar 2019 at 16:37, Jim Finnerty wrote: > > > > Actually, we're working hard to integrate the two approaches. I haven't > had > > time since I returned to review your patch, but I understand

Re: pg_upgrade: Improve invalid option handling

2019-06-14 Thread Daniel Gustafsson
> On 13 Jun 2019, at 10:19, Peter Eisentraut > wrote: > Currently, calling pg_upgrade with an invalid command-line option aborts > pg_upgrade but leaves a pg_upgrade_internal.log file lying around. This > patch reorder things a bit so that that file is not created until all > the options have

Re: POC: Cleaning up orphaned files using undo logs

2019-06-14 Thread Amit Kapila
On Fri, Jun 14, 2019 at 8:26 AM Thomas Munro wrote: > > * current versions of the record and worker code discussed upthread by > Amit and others > Thanks for posting the complete patchset. Last time, I mentioned the remaining work in undo-processing patchset, the status of which is as follows:

Check the number of potential synchronous standbys

2019-06-14 Thread ??????
Hi all, When the number of potential synchronous standbys is smaller than num_sync, such as 'FIRST 3 (1,2)', 'ANY 4 (1,2,3)' in the synchronous_standby_names, the processes will wait for synchronous replication forever. Obviously, it's not expected. I think return false and a error

Re: Minimal logical decoding on standbys

2019-06-14 Thread Amit Khandekar
On Wed, 22 May 2019 at 15:05, Amit Khandekar wrote: > > On Tue, 9 Apr 2019 at 22:23, Amit Khandekar wrote: > > > > On Sat, 6 Apr 2019 at 04:45, Andres Freund wrote: > > > > diff --git a/src/backend/replication/slot.c > > > > b/src/backend/replication/slot.c > > > > index 006446b..5785d2f

[PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-06-14 Thread Ian Barwick
Hi Consider the following cascading standby setup with PostgreSQL 12: - there exists a running primary "A" - standby "B" is cloned from primary "A" using "pg_basebackup --write-recovery-conf" - standby "C" is cloned from standby "B" using "pg_basebackup --write-recovery-conf" So far, so

Re: pg_upgrade: Improve invalid option handling

2019-06-14 Thread Peter Eisentraut
On 2019-06-13 14:30, Masahiko Sawada wrote: > Why do we need to change pg_fatal() to fprintf() & exit()? It seems to > me that we can still use pg_fatal() here since we write the message to > stderr. It just makes the output more consistent with other tools, e.g., old: pg_upgrade: unrecognized

Re: Update list of combining characters

2019-06-14 Thread Peter Eisentraut
On 2019-06-13 15:52, Alvaro Herrera wrote: > I think there's an off-by-one bug in your script. Indeed. Here is an updated script and patch. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services use strict; use

Re: Converting NOT IN to anti-joins during planning

2019-06-14 Thread David Rowley
On Fri, 14 Jun 2019 at 20:41, Simon Riggs wrote: > > On Wed, 6 Mar 2019 at 04:11, David Rowley > wrote: >> >> Hi Jim, >> >> Thanks for replying here. >> >> On Wed, 6 Mar 2019 at 16:37, Jim Finnerty wrote: >> > >> > Actually, we're working hard to integrate the two approaches. I haven't >> >

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-14 Thread Christoph Berg
Re: Tom Lane 2019-06-11 <24452.1560285...@sss.pgh.pa.us> > The only way I can get it to pick "Etc/UCT" is if that's what I put > into /etc/localtime. (In which case I maintain that that's not a bug, > or at least not our bug.) Did you try a symlink or a plain file for /etc/localtime? > So I'm

Re: Add CREATE DATABASE LOCALE option

2019-06-14 Thread Heikki Linnakangas
On 05/06/2019 23:17, Peter Eisentraut wrote: I propose this patch to add a LOCALE option to CREATE DATABASE. This sets both LC_COLLATE and LC_CTYPE with one option. Similar behavior is already supported in initdb, CREATE COLLATION, and createdb. With collation providers other than libc,

Re: SQL/JSON path issues/questions

2019-06-14 Thread Kyotaro Horiguchi
Hi, Thom. At Thu, 13 Jun 2019 14:59:51 +0100, Thom Brown wrote in > Hi, > > I've been reading through the documentation regarding jsonpath and > jsonb_path_query etc., and I have found it lacking explanation for > some functionality, and I've also had some confusion when using the > feature. >

Re: Index Skip Scan

2019-06-14 Thread David Rowley
On Fri, 14 Jun 2019 at 04:32, Jesper Pedersen wrote: > Here is a rebased version. Hi Jesper, I read over this thread a few weeks ago while travelling back from PGCon. (I wish I'd read it on the outward trip instead since it would have been good to talk about it in person.) First off. I think

Re: upgrades in row-level locks can deadlock

2019-06-14 Thread Oleksii Kliukin
Alvaro Herrera wrote: > On 2019-Jun-13, Alvaro Herrera wrote: > >> On 2019-Jun-13, Oleksii Kliukin wrote: >> >>> Makes sense. For the symmetry I have included those that perform lock >>> upgrades in one session and those that doesn’t, while the other sessions >>> acquire locks, do updates or