Re: Assertion failure while streaming toasted data

2021-05-25 Thread Amit Kapila
On Tue, May 25, 2021 at 6:43 PM Dilip Kumar wrote: > > On Tue, May 25, 2021 at 5:46 PM Dilip Kumar wrote: > > > > On Tue, May 25, 2021 at 4:50 PM Amit Kapila wrote: > > > > > > Your patch will fix the reported scenario but I don't like the way > > > multi_insert flag is used to detect

Re: Refactor "mutually exclusive options" error reporting code in parse_subscription_options

2021-05-25 Thread Bharath Rupireddy
On Tue, May 25, 2021 at 7:08 PM Alvaro Herrera wrote: > > I see that the commit a3dc926 and discussion at [1] say below respectively: > > "All the options of those commands are changed to use hex values > > rather than enums to reduce the risk of compatibility bugs when > > introducing new

Re: storing an explicit nonce

2021-05-25 Thread Antonin Houska
Bruce Momjian wrote: > On Tue, May 25, 2021 at 04:48:21PM -0700, Andres Freund wrote: > > Hi, > > > > On 2021-05-25 17:29:03 -0400, Bruce Momjian wrote: > > > So, let me ask --- I thought CTR basically took an encrypted stream of > > > bits and XOR'ed them with the data. If that is true, then

Re: Multi-Column List Partitioning

2021-05-25 Thread Amit Langote
On Sun, May 23, 2021 at 6:49 PM Nitin Jadhav wrote: > > IMO, it is not such a bad syntax from a user's PoV. It's not hard to > > understand from this syntax that the partition constraint is something > > like (a, b) = (1, 2) OR (a, b) = (1, 5) OR ..., where the = performs > > row-wise

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-05-25 Thread David Rowley
On Wed, 26 May 2021 at 14:19, Andy Fan wrote: > I just checked the latest code, looks like we didn't improve this situation > except > that we introduced a GUC to control it. Am I missing something? I don't > have a > suggestion though. Various extra caching was done to help speed it up.

Re: locking [user] catalog tables vs 2pc vs logical rep

2021-05-25 Thread Tom Lane
Amit Kapila writes: > Fair enough. But the way we were looking at them as they will also > block (lead to deadlock) for logical replication of prepared > transactions and also logical replication in synchonous mode without > prepared transactions. Now, if we want to deal with the 2PC issues >

Re: locking [user] catalog tables vs 2pc vs logical rep

2021-05-25 Thread Amit Kapila
On Tue, May 25, 2021 at 12:40 PM Michael Paquier wrote: > > On Mon, May 24, 2021 at 10:03:01AM +0530, Amit Kapila wrote: > > So, this appears to be an existing caveat of synchronous replication. > > If that is the case, I am not sure if it is a good idea to just block > > such ops for the

Re: locking [user] catalog tables vs 2pc vs logical rep

2021-05-25 Thread Amit Kapila
On Tue, May 25, 2021 at 1:43 PM osumi.takami...@fujitsu.com wrote: > > On Monday, May 24, 2021 1:33 PM Amit Kapila wrote: > > On Tue, Apr 20, 2021 at 9:57 AM vignesh C wrote: > > > > > > This similar problem exists in case of synchronous replication setup > > > having synchronous_standby_names

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 09:31:02PM -0400, Bruce Momjian wrote: > I don't understand your computation above. You decrypt the page into > shared buffers, you change a hint bit, and rewrite the page. You are > re-XOR'ing the buffer copy with the same key and nonce. Doesn't that > only change the

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 10:23:46PM -0400, Stephen Frost wrote: > If I’ve got it right, that does mean that the nonces on the replica might > differ from those on the primary though and I’m not completely sure how I feel > about that. We might wish to explicitly document that, due to such risk,

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, On Tue, May 25, 2021 at 22:11 Bruce Momjian wrote: > On Tue, May 25, 2021 at 09:58:22PM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > On Tue, May 25, 2021 at 09:42:48PM -0400, Stephen Frost wrote: > > > > The nonce needs to be a new one, if we

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-05-25 Thread Andy Fan
On Fri, Mar 12, 2021 at 8:31 AM David Rowley wrote: > Thanks for these suggestions. > > On Mon, 22 Feb 2021 at 14:21, Justin Pryzby wrote: > > > > On Tue, Feb 16, 2021 at 11:15:51PM +1300, David Rowley wrote: > > > To summarise here, the planner performance gets a fair bit worse with > > > the

Re: storing an explicit nonce

2021-05-25 Thread Andres Freund
On 2021-05-25 22:11:46 -0400, Bruce Momjian wrote: > This is not possible if we are using the LSN for the full page write LSN > for the hint bit nonce, though we could use a dummy WAL record to > generate an LSN for this, right? We cannot use a dummy WAL record, see my explanation about the

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 09:58:22PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Tue, May 25, 2021 at 09:42:48PM -0400, Stephen Frost wrote: > > > The nonce needs to be a new one, if we include the hint bits in the set > > > of data which is

Re: Incorrect GUC descriptions in docs and postgresql.conf.sample

2021-05-25 Thread Michael Paquier
On Tue, May 25, 2021 at 09:01:30PM -0500, Justin Pryzby wrote: > On Tue, May 25, 2021 at 08:43:14PM -0500, Justin Pryzby wrote: >> Your patch adds documentation about GUCs that can only be set at server >> start/config/commandline. > > Oh: I realized that I read too quickly and misinterpretted

Re: Incorrect GUC descriptions in docs and postgresql.conf.sample

2021-05-25 Thread Justin Pryzby
On Tue, May 25, 2021 at 08:43:14PM -0500, Justin Pryzby wrote: > Your patch adds documentation about GUCs that can only be set at server > start/config/commandline. Oh: I realized that I read too quickly and misinterpretted what "only be set in the config" means (I know I'm not the only one).

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, May 25, 2021 at 09:42:48PM -0400, Stephen Frost wrote: > > The nonce needs to be a new one, if we include the hint bits in the set > > of data which is encrypted. > > > > However, what I believe folks are getting at here is that we

Re: storing an explicit nonce

2021-05-25 Thread Andres Freund
On 2021-05-25 21:51:31 -0400, Bruce Momjian wrote: > How do we prevent torn pages if we are writing the page with a new > nonce, and no WAL-logged full page image? That should only arise if we are guaranteed to replay from a redo point that is followed by at least one FPI for the page we're about

Re: Move pg_attribute.attcompression to earlier in struct for reduced size?

2021-05-25 Thread Michael Paquier
On Tue, May 25, 2021 at 08:33:47PM -0500, Justin Pryzby wrote: > It reminds me of reltablespace, which is stored as 0 to mean the database's > default tablespace. > > Also, values are currently retoasted during vacuum full if their column's > current compression method doesn't match the value's

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 09:42:48PM -0400, Stephen Frost wrote: > The nonce needs to be a new one, if we include the hint bits in the set > of data which is encrypted. > > However, what I believe folks are getting at here is that we could keep > the LSN the same, but increase the nonce when the

Re: Incorrect GUC descriptions in docs and postgresql.conf.sample

2021-05-25 Thread Justin Pryzby
Your patch adds documentation about GUCs that can only be set at server start/config/commandline. But it's not true for any of these, which are all HUP/SUSET. Please double check your logic :) src/backend/utils/misc/guc.c: {"autovacuum_work_mem", PGC_SIGHUP, RESOURCES_MEM,

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, May 25, 2021 at 07:48:54PM -0400, Stephen Frost wrote: > > Not sure what you're referring to in the second half ... simply knowing > > that some of the data has a given plaintext (such as having a really > > good idea that the word

Re: Skip partition tuple routing with constant partition key

2021-05-25 Thread Amit Langote
Hou-san, On Wed, May 26, 2021 at 10:05 AM houzj.f...@fujitsu.com wrote: > From: Amit Langote > Sent: Tuesday, May 25, 2021 10:06 PM > > Though again, I think we can do this without changing the relcache > > interface, > > such as RelationGetPartitionQual(). > > > > PartitionTupleRouting has

Incorrect GUC descriptions in docs and postgresql.conf.sample

2021-05-25 Thread Michael Paquier
Hi all, I got curious with what Justin just told here with max_logical_replication_workers: https://www.postgresql.org/message-id/20210526001359.ge3...@telsasoft.com And while looking at the full set of GUCs, I noticed much more than one parameter that needed adjustments in the documentation

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, May 25, 2021 at 08:03:14PM -0400, Stephen Frost wrote: > > Indeed they are, but that's not relevant to the thrust of this specific > > debate. > > > > Bruce is arguing that because clog is unprotected that it's not useful > > to

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 07:48:54PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Tue, May 25, 2021 at 05:22:43PM -0400, Stephen Frost wrote: > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > OK, this is good to know. I know the never-reuse

Re: Move pg_attribute.attcompression to earlier in struct for reduced size?

2021-05-25 Thread Justin Pryzby
On Sun, May 23, 2021 at 12:25:10PM -0400, Tom Lane wrote: > However, the more I looked at that code the less I liked it. > I think the way that compression selection is handled for indexes, > ie consult default_toast_compression on-the-fly, is *far* saner > than what is currently implemented for

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 04:48:21PM -0700, Andres Freund wrote: > Hi, > > On 2021-05-25 17:29:03 -0400, Bruce Momjian wrote: > > So, let me ask --- I thought CTR basically took an encrypted stream of > > bits and XOR'ed them with the data. If that is true, then why are > > changing hint bits a

Re: fdatasync performance problem with large number of DB files

2021-05-25 Thread Michael Paquier
On Tue, May 25, 2021 at 07:13:59PM -0500, Justin Pryzby wrote: > This one isn't documented as requiring a restart: > max_logical_replication_workers. There is much more than meets the eye here, and this is unrelated to this thread, so let's discuss that on a separate thread. I'll start a new one

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 08:03:14PM -0400, Stephen Frost wrote: > Indeed they are, but that's not relevant to the thrust of this specific > debate. > > Bruce is arguing that because clog is unprotected that it's not useful > to protect relation data, with regard to data integrity validation as >

RE: Skip partition tuple routing with constant partition key

2021-05-25 Thread houzj.f...@fujitsu.com
Hi Amit-san From: Amit Langote Sent: Tuesday, May 25, 2021 10:06 PM > Hou-san, > > Thanks for the patch and It looks more compact than mine. > > > > After taking a quick look at the patch, I found a possible issue. > > Currently, the patch does not search the parent's partition key expression >

Re: automatic analyze: readahead - add "IO read time" log message

2021-05-25 Thread Stephen Frost
Greetings, * Justin Pryzby (pry...@telsasoft.com) wrote: > This patch adds hits/misses/dirtied, but explain says > hit/read/dirtied/written. > > Should it say "read" instead of "misses" ? > > src/backend/access/heap/vacuumlazy.c: >_("buffer

Re: automatic analyze: readahead - add "IO read time" log message

2021-05-25 Thread Justin Pryzby
This patch adds hits/misses/dirtied, but explain says hit/read/dirtied/written. Should it say "read" instead of "misses" ? src/backend/access/heap/vacuumlazy.c: _("buffer usage: %lld hits, %lld misses, %lld dirtied\n"),

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Matthias van de Meent writes: > > I like the idea of the ZSON type, but I'm somewhat disappointed by its > > current limitations: > > I've not read the code, so maybe this thought is completely off-point, > but I wonder if anything could be

Re: automatic analyze: readahead - add "IO read time" log message

2021-05-25 Thread Stephen Frost
Greetings, * Egor Rogov (e.ro...@postgrespro.ru) wrote: > On 11.02.2021 01:10, Stephen Frost wrote: > >* Heikki Linnakangas (hlinn...@iki.fi) wrote: > >>On 05/02/2021 23:22, Stephen Frost wrote: > >>>Unless there's anything else on this, I'll commit these sometime next > >>>week. > >>One more

Re: fdatasync performance problem with large number of DB files

2021-05-25 Thread Justin Pryzby
On Sat, Mar 20, 2021 at 12:16:27PM +1300, Thomas Munro wrote: > > > + { > > > + {"recovery_init_sync_method", PGC_POSTMASTER, > > > ERROR_HANDLING_OPTIONS, > > > + gettext_noop("Sets the method for synchronizing the > > > data directory before crash

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2021-05-25 17:15:55 -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > We already discussed that there are too many other ways to break system > > > integrity that are not encrypted/integrity-checked, e.g.,

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2021-05-25 17:22:43 -0400, Stephen Frost wrote: > > Err, to be clear, I was saying that we could exclude the hint bits > > *entirely* from what's being encrypted and I don't think that would be a > > huge issue. > > It's a *huge* issue.

Re: storing an explicit nonce

2021-05-25 Thread Andres Freund
On 2021-05-25 17:15:55 -0400, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > We already discussed that there are too many other ways to break system > > integrity that are not encrypted/integrity-checked, e.g., changes to > > clog. Do you disagree? > > We had agreed that

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2021-05-25 17:04:50 -0400, Stephen Frost wrote: > > I do think it's reasonable to consider having hint bits not included in > > the encrypted part of the page and therefore remove the need to produce > > a new nonce for each hint bit

Re: pg_rewind fails if there is a read only file.

2021-05-25 Thread Michael Paquier
On Tue, May 25, 2021 at 03:17:37PM -0400, Andrew Dunstan wrote: > If we do decide to do something the question arises what should it do? > If we're to allow for it I'm wondering if the best thing would be simply > to ignore such a file. Enforcing assumptions that any file could be ready-only is a

Re: storing an explicit nonce

2021-05-25 Thread Andres Freund
On 2021-05-25 19:48:54 -0400, Stephen Frost wrote: > That's how CTR works, yes. The issue that you run into is that once > you've got two pages which have different data but were encrypted with > the same key and nonce then you can use crib-dragging. > > A good example of how this works is here:

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2021-05-25 16:34:10 -0400, Stephen Frost wrote: > > The nonce does need to be absolutely unique for a given encryption key and > > therefore needs to be global in some form. > > You can achieve that without a global counter though, by

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, May 25, 2021 at 05:25:36PM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > On Tue, May 25, 2021 at 05:15:55PM -0400, Stephen Frost wrote: > > > > > We already discussed that there are too many other

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, May 25, 2021 at 05:22:43PM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > OK, this is good to know. I know the never-reuse rule, so it is good to > > > know it can be relaxed for certain data without

Re: storing an explicit nonce

2021-05-25 Thread Andres Freund
Hi, On 2021-05-25 17:29:03 -0400, Bruce Momjian wrote: > So, let me ask --- I thought CTR basically took an encrypted stream of > bits and XOR'ed them with the data. If that is true, then why are > changing hint bits a problem? We already can see some of the bit stream > by knowing some bytes

Re: storing an explicit nonce

2021-05-25 Thread Andres Freund
Hi, On 2021-05-25 17:22:43 -0400, Stephen Frost wrote: > Err, to be clear, I was saying that we could exclude the hint bits > *entirely* from what's being encrypted and I don't think that would be a > huge issue. It's a *huge* issue. For one, the computational effort of doing so would be a

Re: storing an explicit nonce

2021-05-25 Thread Andres Freund
Hi, On 2021-05-25 17:04:50 -0400, Stephen Frost wrote: > I do think it's reasonable to consider having hint bits not included in > the encrypted part of the page and therefore remove the need to produce > a new nonce for each hint bit change. Huh. How are you going to track that efficiently? Do

Re: storing an explicit nonce

2021-05-25 Thread Andres Freund
Hi, On 2021-05-25 16:34:10 -0400, Stephen Frost wrote: > The nonce does need to be absolutely unique for a given encryption key and > therefore needs to be global in some form. You can achieve that without a global counter though, by prepending a per-relation nonce with some local counter. I'm

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 05:25:36PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Tue, May 25, 2021 at 05:15:55PM -0400, Stephen Frost wrote: > > > > We already discussed that there are too many other ways to break system > > > > integrity that are

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 05:22:43PM -0400, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > OK, this is good to know. I know the never-reuse rule, so it is good to > > know it can be relaxed for certain data without causing problems in > > other places. Should I modify my

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, May 25, 2021 at 05:15:55PM -0400, Stephen Frost wrote: > > > We already discussed that there are too many other ways to break system > > > integrity that are not encrypted/integrity-checked, e.g., changes to > > > clog. Do you

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, May 25, 2021 at 05:14:24PM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > Yes, I can see that happening. I think occasional leakage of hint bit > > > changes to be acceptable. We might decide they

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, May 25, 2021 at 05:04:50PM -0400, Stephen Frost wrote: > > > Now, if we want to consult some security experts and have them tell us > > > the hint bit visibility is not a problem, we could get by without using a > > > new nonce for

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 05:15:55PM -0400, Stephen Frost wrote: > > We already discussed that there are too many other ways to break system > > integrity that are not encrypted/integrity-checked, e.g., changes to > > clog. Do you disagree? > > We had agreed that this wasn't something that was

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, May 25, 2021 at 04:29:08PM -0400, Stephen Frost wrote: > > On Tue, May 25, 2021 at 14:56 Bruce Momjian wrote: > > > > On Tue, May 25, 2021 at 02:25:21PM -0400, Robert Haas wrote: > > > One question here is whether we're

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 05:14:24PM -0400, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > Yes, I can see that happening. I think occasional leakage of hint bit > > changes to be acceptable. We might decide they are all acceptable. > > I don't think that I agree with the

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, May 25, 2021 at 01:54:21PM -0700, Andres Freund wrote: > > On 2021-05-25 15:34:04 -0400, Bruce Momjian wrote: > > > My point is that we have to full-page-write cases where we change the > > > nonce --- we get a new LSN/nonce for free

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 05:04:50PM -0400, Stephen Frost wrote: > > Now, if we want to consult some security experts and have them tell us > > the hint bit visibility is not a problem, we could get by without using a > > new nonce for hint bit changes, and in that case it doesn't matter if we > >

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Andrew Dunstan
On 5/25/21 4:31 PM, Tom Lane wrote: > Andrew Dunstan writes: >> On 5/25/21 4:10 PM, Tom Lane wrote: >>> Also, even if ZSON was "100% compatible with JSONB" back in 2016, >>> a whole lot of features have been added since then. Having to >>> duplicate all that code again for a different data

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 04:29:08PM -0400, Stephen Frost wrote: > Greetings, > > On Tue, May 25, 2021 at 14:56 Bruce Momjian wrote: > > On Tue, May 25, 2021 at 02:25:21PM -0400, Robert Haas wrote: > > One question here is whether we're comfortable saying that the nonce > > is

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, May 25, 2021 at 03:20:06PM -0400, Bruce Momjian wrote: > > Also, when you change hint bits, either you don't change the nonce/LSN, > > and don't re-encrypt the page (and the hint bit changes are visible), or > > you change the nonce

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 01:54:21PM -0700, Andres Freund wrote: > Hi, > > On 2021-05-25 15:34:04 -0400, Bruce Momjian wrote: > > My point is that we have to full-page-write cases where we change the > > nonce --- we get a new LSN/nonce for free if we are using the LSN as the > > nonce. What has

Re: storing an explicit nonce

2021-05-25 Thread Andres Freund
Hi, On 2021-05-25 15:34:04 -0400, Bruce Momjian wrote: > My point is that we have to full-page-write cases where we change the > nonce --- we get a new LSN/nonce for free if we are using the LSN as the > nonce. What has made this approach much easier is that you basically > tie a change of the

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Tom Lane
Matthias van de Meent writes: > I like the idea of the ZSON type, but I'm somewhat disappointed by its > current limitations: I've not read the code, so maybe this thought is completely off-point, but I wonder if anything could be learned from PostGIS. AIUI they have developed the

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, On Tue, May 25, 2021 at 15:09 Robert Haas wrote: > On Tue, May 25, 2021 at 2:45 PM Bruce Momjian wrote: > > Well, if we create a separate nonce counter, we still need to make sure > > it doesn't go backwards during a crash, so we have to WAL log it > > I think we don't really need a

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Tom Lane
Andrew Dunstan writes: > On 5/25/21 4:10 PM, Tom Lane wrote: >> Also, even if ZSON was "100% compatible with JSONB" back in 2016, >> a whole lot of features have been added since then. Having to >> duplicate all that code again for a different data type is not >> something I want to see us

Re: storing an explicit nonce

2021-05-25 Thread Stephen Frost
Greetings, On Tue, May 25, 2021 at 14:56 Bruce Momjian wrote: > On Tue, May 25, 2021 at 02:25:21PM -0400, Robert Haas wrote: > > One question here is whether we're comfortable saying that the nonce > > is entirely constant. I wasn't sure about that. It seems possible to > > me that different

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Andrew Dunstan
On 5/25/21 4:10 PM, Tom Lane wrote: > Magnus Hagander writes: >> On Tue, May 25, 2021 at 12:55 PM Aleksander Alekseev >> wrote: >>> Back in 2016 while being at PostgresPro I developed the ZSON extension [1]. >>> The extension introduces the new ZSON type, which is 100% compatible with >>>

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Tom Lane
Robert Haas writes: > On Tue, May 25, 2021 at 3:10 PM Tom Lane wrote: >> You're definitely confused, because reversing that choice is *exactly* >> what I'm on about. The question of whether SQL-level CALL should act >> differently from plpgsql CALL is pretty secondary. > I understood the

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Matthias van de Meent
On Tue, 25 May 2021 at 13:32, Magnus Hagander wrote: > > On Tue, May 25, 2021 at 12:55 PM Aleksander Alekseev > wrote: > > > > Hi hackers, > > > > Back in 2016 while being at PostgresPro I developed the ZSON extension [1]. > > The extension introduces the new ZSON type, which is 100% compatible

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Tom Lane
Magnus Hagander writes: > On Tue, May 25, 2021 at 12:55 PM Aleksander Alekseev > wrote: >> Back in 2016 while being at PostgresPro I developed the ZSON extension [1]. >> The extension introduces the new ZSON type, which is 100% compatible with >> JSONB but uses a shared dictionary of strings

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Andrew Dunstan
On 5/25/21 6:55 AM, Aleksander Alekseev wrote: > Hi hackers, > > Back in 2016 while being at PostgresPro I developed the ZSON extension > [1]. The extension introduces the new ZSON type, which is 100% > compatible with JSONB but uses a shared dictionary of strings most > frequently used in given

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Robert Haas
On Tue, May 25, 2021 at 3:10 PM Tom Lane wrote: > No, you misunderstand my proposal. The thing that I most urgently > want to do is to prevent that situation from ever arising, by not > allowing those two procedures to coexist, just as you can't have > two functions with such signatures. > > If

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 03:34:04PM -0400, Bruce Momjian wrote: > Let me go into more detail here. The general rule is that you never > encrypt _different_ data with the same key/nonce. Now, since a hint bit > change changes the data, it should get a new nonce, and since it is a > newly encrypted

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 03:20:06PM -0400, Bruce Momjian wrote: > Also, when you change hint bits, either you don't change the nonce/LSN, > and don't re-encrypt the page (and the hint bit changes are visible), or > you change the nonce and reencrypt the page, and you are then WAL > logging the

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-05-25 Thread Alvaro Herrera
On 2021-May-24, Noah Misch wrote: > prairiedog and wrasse failed a $SUBJECT test after this (commit 8aba932). > Also, some non-CLOBBER_CACHE_ALWAYS animals failed a test before the fix. > These IsolationCheck failures match detach-partition-concurrently[^\n]*FAILED: > > sysname │

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 03:09:03PM -0400, Robert Haas wrote: > On Tue, May 25, 2021 at 2:45 PM Bruce Momjian wrote: > > Well, if we create a separate nonce counter, we still need to make sure > > it doesn't go backwards during a crash, so we have to WAL log it > > I think we don't really need a

Re: pg_rewind fails if there is a read only file.

2021-05-25 Thread Andrew Dunstan
On 5/25/21 10:29 AM, Tom Lane wrote: > Andrew Dunstan writes: >> Perhaps we should add that read-only files can be particularly problematic. > Given the (legitimate, IMO) example of a read-only SSL key, I'm not > quite convinced that pg_rewind doesn't need to cope with this. > >

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Tom Lane
Robert Haas writes: > I'm also concerned about the behavior here. I noticed it when this > commit went in, and it seemed concerning to me then, and it still > does. Nevertheless, I'm not convinced that your proposal is an > improvement. Suppose we have foo(int, out int) and also foo(int). > Then,

Re: storing an explicit nonce

2021-05-25 Thread Robert Haas
On Tue, May 25, 2021 at 2:45 PM Bruce Momjian wrote: > Well, if we create a separate nonce counter, we still need to make sure > it doesn't go backwards during a crash, so we have to WAL log it I think we don't really need a global counter, do we? We could simply increment the nonce every time

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 10:37:32AM -0700, Andres Freund wrote: > The obvious concerns are issues around binary upgrades for cases that > already use the special space? Are you planning to address that by not > having that path? Or by storing the nonce at the "start" of the special > space (i.e.

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Tom Lane
Peter Eisentraut writes: > On 25.05.21 17:20, Tom Lane wrote: >> I don't really see how you can argue that the existing behavior is >> more spec-compliant than what I'm suggesting. What I read in the spec >> (SQL:2021 10.4 SR 9) h) iii) 1)) is >> 1) If Pi is an output SQL parameter, then XAi

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Robert Haas
On Tue, May 25, 2021 at 2:20 PM Tom Lane wrote: > Just to enlarge on that point a bit: > > regression=# create function foo(int, out int) language sql > regression-# as 'select $1'; > CREATE FUNCTION > regression=# create procedure foo(int, out int) language sql > regression-# as 'select $1'; >

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 02:25:21PM -0400, Robert Haas wrote: > One question here is whether we're comfortable saying that the nonce > is entirely constant. I wasn't sure about that. It seems possible to > me that different encryption algorithms might want nonces of different > sizes, either now or

Re: storing an explicit nonce

2021-05-25 Thread Bruce Momjian
On Tue, May 25, 2021 at 12:46:45PM -0400, Robert Haas wrote: > On Thu, Mar 18, 2021 at 2:59 PM Bruce Momjian wrote: > > > Ultimately, we need to make sure that LSNs aren't re-used. There's two > > > sources of LSNs today: those for relations which are being written into > > > the WAL and those

Re: Access root->simple_rte_array instead of Query->rtable for 2 more cases.

2021-05-25 Thread Tom Lane
Andy Fan writes: > When I am understanding the relationship between Query->rtable and > root->simple_rte_array, I'd like to assume that Query->rtable should be > never used > when root->simple_rte_array is ready. TBH, now that Lists are really arrays, there's basically no performance advantage

Re: storing an explicit nonce

2021-05-25 Thread Robert Haas
On Tue, May 25, 2021 at 1:37 PM Andres Freund wrote: > The obvious concerns are issues around binary upgrades for cases that > already use the special space? Are you planning to address that by not > having that path? Or by storing the nonce at the "start" of the special > space (i.e. [normal

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Tom Lane
I wrote: > * The rules for how to identify a target routine in ALTER, DROP, > etc are different for functions and procedures. That's especially > nasty in ALTER/DROP ROUTINE, where we don't have a syntax cue > as to whether or not to ignore OUT parameters. Just to enlarge on that point a bit:

Re: CALL versus procedures with output-only arguments

2021-05-25 Thread Peter Eisentraut
On 25.05.21 17:20, Tom Lane wrote: I don't really see how you can argue that the existing behavior is more spec-compliant than what I'm suggesting. What I read in the spec (SQL:2021 10.4 SR 9) h) iii) 1)) is 1) If Pi is an output SQL parameter, then XAi shall be a . (where more or less

Access root->simple_rte_array instead of Query->rtable for 2 more cases.

2021-05-25 Thread Andy Fan
When I am understanding the relationship between Query->rtable and root->simple_rte_array, I'd like to assume that Query->rtable should be never used when root->simple_rte_array is ready. I mainly checked two places, make_one_rel and create_plan with the below hacks. { List *l =

Re: Possible pointer var TupleDesc rettupdesc used not initialized (src/backend/optimizer/util/clauses.c)

2021-05-25 Thread Ranier Vilela
Em ter., 25 de mai. de 2021 às 14:35, Tom Lane escreveu: > Ranier Vilela writes: > > Following the guidelines, I provided a patch. > > Oh, I already pushed a fix, thanks. > No problem! Thank you. regards, Ranier Vilela

Re: storing an explicit nonce

2021-05-25 Thread Andres Freund
Hi, On 2021-05-25 12:46:45 -0400, Robert Haas wrote: > This approach has a few disadvantages. For example, right now, we only > need to WAL log hints for the first write to each page after a > checkpoint, but in this approach, if the same page is written multiple > times per checkpoint cycle,

Re: Possible pointer var TupleDesc rettupdesc used not initialized (src/backend/optimizer/util/clauses.c)

2021-05-25 Thread Tom Lane
Ranier Vilela writes: > Following the guidelines, I provided a patch. Oh, I already pushed a fix, thanks. regards, tom lane

Re: Possible pointer var TupleDesc rettupdesc used not initialized (src/backend/optimizer/util/clauses.c)

2021-05-25 Thread Ranier Vilela
Em ter., 25 de mai. de 2021 às 13:09, Tom Lane escreveu: > Ranier Vilela writes: > > Possible pointer TupleDesc rettupdesc used not initialized? > > if (!isNull) at line 4346 taking a true branch, the function > > check_sql_fn_retval at line 4448 can use rettupdesc uninitialized. > > This seems

Re: Test of a partition with an incomplete detach has a timing issue

2021-05-25 Thread Alvaro Herrera
On 2021-May-25, Tom Lane wrote: > Alvaro Herrera writes: > > The problem disappears completely if I add a sleep to the cancel query: > > step "s1cancel" { SELECT pg_cancel_backend(pid), pg_sleep(0.01) FROM > > d3_pid; } > > I suppose a 0.01 second sleep is not going to be sufficient to

Re: How can the Aggregation move to the outer query

2021-05-25 Thread Andy Fan
On Tue, May 25, 2021 at 10:23 PM Tom Lane wrote: > David Rowley writes: > > On Tue, 25 May 2021 at 22:28, Andy Fan wrote: > >> explain (costs off) select (select count(*) filter (where t2.b = 1) > from m1 t1) > >> from m1 t2 where t2.b % 2 = 1; > >> > >> This one is too confusing to me since

Re: array_cat anycompatible change is breaking xversion upgrade tests

2021-05-25 Thread Justin Pryzby
On Thu, May 20, 2021 at 07:35:10PM -0400, Tom Lane wrote: > Justin Pryzby writes: > > On Wed, Nov 04, 2020 at 07:43:51PM -0500, Tom Lane wrote: > >> As was discussed in the thread leading up to that commit, modifying the > >> signature of array_cat and friends could break user-defined operators >

Re: Possible pointer var TupleDesc rettupdesc used not initialized (src/backend/optimizer/util/clauses.c)

2021-05-25 Thread Tom Lane
Ranier Vilela writes: > Possible pointer TupleDesc rettupdesc used not initialized? > if (!isNull) at line 4346 taking a true branch, the function > check_sql_fn_retval at line 4448 can use rettupdesc uninitialized. This seems to have been introduced by the SQL-function-body patch. After some

Re: automatic analyze: readahead - add "IO read time" log message

2021-05-25 Thread Egor Rogov
Hi, On 11.02.2021 01:10, Stephen Frost wrote: Greetings, * Heikki Linnakangas (hlinn...@iki.fi) wrote: On 05/02/2021 23:22, Stephen Frost wrote: Unless there's anything else on this, I'll commit these sometime next week. One more thing: Instead of using 'effective_io_concurrency' GUC

  1   2   >