Re: Index Skip Scan

2019-07-10 Thread David Rowley
On Thu, 11 Jul 2019 at 14:50, Thomas Munro wrote: > > On Thu, Jul 11, 2019 at 2:40 AM Floris Van Nee > wrote: > > I verified that the backwards index scan is indeed functioning now. > > However, I'm afraid it's not that simple, as I think the cursor case is > > broken now. I think having just

Re: warning to publication created and wal_level is not set to logical

2019-07-10 Thread Thomas Munro
On Wed, Jul 10, 2019 at 12:47 PM Tom Lane wrote: > 1. > > + errmsg("insufficient wal_level to publish logical > changes"), > > Might read better as "wal_level is insufficient to publish logical changes"? > > 2. > > + errhint("Set wal_level to logical

Re: pg_receivewal documentation

2019-07-10 Thread Michael Paquier
On Wed, Jul 10, 2019 at 09:12:46PM +0200, Laurenz Albe wrote: > Are you talking about the replication connection from pg_receivewal > to the PostgreSQL server? That wouldn't do anything, because it is > the setting of "synchronous_commit" for an independent client > connection that is the

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2019-07-10 Thread Peter Geoghegan
On Sat, Jul 6, 2019 at 4:08 PM Peter Geoghegan wrote: > I took a closer look at this patch, and have some general thoughts on > its design, and specific feedback on the implementation. I have some high level concerns about how the patch might increase contention, which could make queries slower.

Re: POC: Cleaning up orphaned files using undo logs

2019-07-10 Thread Amit Kapila
On Wed, Jul 10, 2019 at 10:06 PM Robert Haas wrote: > > On Wed, Jul 10, 2019 at 2:32 AM Amit Kapila wrote: > > As of now, after we finish executing the rollback actions, the entry > > from the hash table is removed. Now, at a later time (when queues are > > full and we want to insert a new

Re: Implementing Incremental View Maintenance

2019-07-10 Thread Yugo Nagata
Hi Thomas, Thank you for your review and discussion on this patch! > > 2019年7月8日(月) 15:32 Thomas Munro : > > > > > On Fri, Jun 28, 2019 at 10:56 PM Yugo Nagata wrote: > > > > Attached is a WIP patch of IVM which supports some aggregate functions. > > > > > > Hi Nagata-san and Hoshiai-san, > >

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-10 Thread Michael Paquier
On Wed, Jul 10, 2019 at 09:44:14PM +0200, Julien Rouhaud wrote: > On Wed, Jul 10, 2019 at 4:15 PM Alvaro Herrera > wrote: >> Looking good! > > Thanks! Confirmed. The last set is much easier to go through. >> I'm not sure about the "Consume" word in ConsumeIdleSlot; >> maybe "Reserve"?

Re: POC: Cleaning up orphaned files using undo logs

2019-07-10 Thread Dilip Kumar
On Thu, Jul 11, 2019 at 12:38 AM Robert Haas wrote: > > On Tue, Jul 9, 2019 at 6:28 AM Dilip Kumar wrote: > > PFA, updated patch version which includes > > - One defect fix in undo interface related to undo page compression > > for handling persistence level > > - Implemented pending TODO

Re: Index Skip Scan

2019-07-10 Thread Thomas Munro
On Thu, Jul 11, 2019 at 2:40 AM Floris Van Nee wrote: > I verified that the backwards index scan is indeed functioning now. However, > I'm afraid it's not that simple, as I think the cursor case is broken now. I > think having just the 'scan direction' in the btree code is not enough to get >

Re: Implementing Incremental View Maintenance

2019-07-10 Thread Tatsuo Ishii
> I am quite interested to learn how IVM interacts with SERIALIZABLE. Just for a fun, I have added: SET TRANSACTION ISOLATION LEVEL SERIALIZABLE; right after every BEGIN; in incremental_matview.sql in regression test and it seems it works. > A couple of superficial review comments: > > +

Re: progress report for ANALYZE

2019-07-10 Thread Robert Haas
On Wed, Jul 10, 2019 at 9:26 AM Alvaro Herrera wrote: > On 2019-Jul-10, Robert Haas wrote: > > On Tue, Jul 9, 2019 at 6:12 PM Alvaro Herrera > > wrote: > > > Hmm, ok. In CREATE INDEX, we use the block counters multiple times. > > > > Why do we do that? > > Because we scan the table first, then

Re: Copy data to DSA area

2019-07-10 Thread Thomas Munro
On Wed, Jul 10, 2019 at 6:03 PM Thomas Munro wrote: > Hmm. I wonder if we should just make ShmContextFree() do nothing! And > make ShmContextAlloc() allocate (say) 8KB chunks (or larger if needed > for larger allocation) and then hand out small pieces from the > 'current' chunk as needed. Then

Re: using explicit_bzero

2019-07-10 Thread Michael Paquier
On Mon, Jun 24, 2019 at 02:08:50PM +0900, Michael Paquier wrote: > CreateRole() and AlterRole() can manipulate a password in plain format > in memory. The cleanup could be done just after calling > encrypt_password() in user.c. > > Could it be possible to add the new flag in pg_config.h.win32?

Re: [sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Michael Paquier
On Wed, Jul 10, 2019 at 11:26:09PM +0200, Tomas Vondra wrote: > Yeah, that's a bug. Will fix (not sure how yet). Please note that I have added an open item for it. -- Michael signature.asc Description: PGP signature

Re: using explicit_bzero

2019-07-10 Thread Thomas Munro
On Sun, Jul 7, 2019 at 1:11 AM Peter Eisentraut wrote: > I see. My premise, which should perhaps be explained in a comment at > least, is that on an operating system that does not provide > explicit_bzero() (or an obvious alternative), we don't care about > addressing this particular security

Re: pg_checksums (or checksums in general) vs tableam

2019-07-10 Thread Michael Paquier
On Wed, Jul 10, 2019 at 09:19:03AM -0700, Andres Freund wrote: > On July 10, 2019 9:12:18 AM PDT, Magnus Hagander wrote: >> That would be fine, if we actually knew. Should we (or have we already?) >> defined a rule that they are not allowed to use the same naming standard >> unless they have the

Re: buildfarm's typedefs list has gone completely nutso

2019-07-10 Thread Andres Freund
Hi, On 2019-07-10 16:40:20 -0400, Andrew Dunstan wrote: > On 7/10/19 1:34 PM, Andres Freund wrote: > > > > Hm, it has gotten gcc-9 installed recently, but calliphoridae isn't > > using that. So it's probably not the compiler side. But I also see a > > binutils upgrade: > > > > 2019-07-08 06:22:48

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Thu, Jul 11, 2019 at 12:18:47AM +0200, Tomas Vondra wrote: > On Wed, Jul 10, 2019 at 06:04:30PM -0400, Stephen Frost wrote: > > Greetings, > > > > * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > > > On Wed, Jul 10, 2019 at 04:11:21PM -0400, Alvaro Herrera wrote: > > > >On 2019-Jul-10,

Re: doc: minor update for description of "pg_roles" view

2019-07-10 Thread Ian Barwick
On 7/11/19 3:24 AM, Bruce Momjian wrote: On Wed, Jul 10, 2019 at 02:35:56PM +0900, Ian Barwick wrote: Hi Here: https://www.postgresql.org/docs/12/view-pg-roles.html we state: "This view explicitly exposes the OID column of the underlying table, since that is needed to do joins to

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 06:28:42PM -0400, Alvaro Herrera wrote: > On 2019-Jul-10, Joe Conway wrote: > > > On 7/10/19 3:53 PM, Alvaro Herrera wrote: > > > > (I do think you can have multiple writes of the same page with > > > different LSNs, if you change hint bits and don't write WAL about it, >

Re: Adversarial case for "many duplicates" nbtree split strategy in v12

2019-07-10 Thread Peter Geoghegan
On Tue, Jul 2, 2019 at 3:51 PM Peter Geoghegan wrote: > I've already written a rough patch that fixes the issue by taking this > second view of the problem. The patch makes nbtsplitloc.c more > skeptical about finishing with the "many duplicates" strategy, > avoiding the problem -- it can just

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-10 Thread Michael Paquier
On Wed, Jul 10, 2019 at 09:11:41AM -0700, Ashwin Agrawal wrote: > Will post patch for the tool, once I get in little decent shape. That would be nice! We may be able to get something into v13 this way then. -- Michael signature.asc Description: PGP signature

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Ryan Lambert
> As I understand, the reason we > want to avoid using the same IV for too many pages is to dodge a > cryptanalysis attack, which requires a large amount of data encrypted > with the same key/IV in order to be effective. But if we have two > copies of the same page encrypted with the same key/IV,

Re: Refactoring syslogger piping to simplify adding new log destinations

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-10, Tom Lane wrote: > Alvaro Herrera writes: > > Maybe we can use something like a shared memory queue, working in a > > similar way to wal_buffers -- where backends send over the shm queue to > > syslogger, and syslogger writes in order to the actual log file. > > No way that's

Re: Refactoring syslogger piping to simplify adding new log destinations

2019-07-10 Thread Tom Lane
Alvaro Herrera writes: > Maybe we can use something like a shared memory queue, working in a > similar way to wal_buffers -- where backends send over the shm queue to > syslogger, and syslogger writes in order to the actual log file. No way that's going to be acceptable for postmaster output.

Re: [sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Tom Lane
Oh ... while we're piling on here, it just sunk into me that mcv_get_match_bitmap is deciding what the semantics of an operator are by seeing what it's using for a selectivity estimator. That is just absolutely, completely wrong. For starters, it means that the whole mechanism fails for any

Re: Refactoring syslogger piping to simplify adding new log destinations

2019-07-10 Thread Alvaro Herrera
Hi Sehrope, On 2019-Jul-10, Sehrope Sarkuni wrote: > While working on adding a new log_destination I noticed that the > syslogger piping would need to be updated. At the moment both ends > only handle stderr/csvlog as the pipe message header has a char > "is_last" that is either t/f (stderr

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-10, Joe Conway wrote: > On 7/10/19 3:53 PM, Alvaro Herrera wrote: > > (I do think you can have multiple writes of the same page with > > different LSNs, if you change hint bits and don't write WAL about it, > > Do you mean "multiple writes of the same page without..."? Right,

Re: Tid scan improvements

2019-07-10 Thread David Rowley
On Sun, 7 Jul 2019 at 15:32, Edmund Horner wrote: > I'm not really sure how to proceed. I started with a fairly pragmatic > solution to "WHERE ctid > ? AND ctid < ?" for tables, and then tableam > came along. > > The options I see are: > > A. Continue to target only heapam tables, making the

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Tomas Vondra
On Wed, Jul 10, 2019 at 06:04:30PM -0400, Stephen Frost wrote: Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: On Wed, Jul 10, 2019 at 04:11:21PM -0400, Alvaro Herrera wrote: >On 2019-Jul-10, Bruce Momjian wrote: > >>Uh, what if a transaction modifies page 0 and page 1 of the

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > On Wed, Jul 10, 2019 at 04:11:21PM -0400, Alvaro Herrera wrote: > >On 2019-Jul-10, Bruce Momjian wrote: > > > >>Uh, what if a transaction modifies page 0 and page 1 of the same table > >>--- don't those pages have the same LSN. > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Tomas Vondra
On Wed, Jul 10, 2019 at 04:11:21PM -0400, Alvaro Herrera wrote: On 2019-Jul-10, Bruce Momjian wrote: Uh, what if a transaction modifies page 0 and page 1 of the same table --- don't those pages have the same LSN. No, because WAL being a physical change log, each page gets its own WAL record

Re: [sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Tomas Vondra
On Wed, Jul 10, 2019 at 05:45:24PM -0400, Tom Lane wrote: Tomas Vondra writes: Yeah, that's a bug. Will fix (not sure how yet). You could do worse than replace this: ok = (NumRelids(clause) == 1) && (is_pseudo_constant_clause(lsecond(expr->args)) || (varonleft

Re: [sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Tom Lane
Tomas Vondra writes: > Yeah, that's a bug. Will fix (not sure how yet). You could do worse than replace this: ok = (NumRelids(clause) == 1) && (is_pseudo_constant_clause(lsecond(expr->args)) || (varonleft = false,

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Joe Conway
On 7/10/19 3:53 PM, Alvaro Herrera wrote: > On 2019-Jul-10, Bruce Momjian wrote: > >> Good, so I think we all now agree we have to put the nonce >> (pg_class.oid, LSN, page-number) though the cipher using the secret. (been traveling -- just trying to get caught up on this thread) > Actually,

Re: [sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Tom Lane
Tomas Vondra writes: > BTW which other thread? I don't see any other threads mentioning this > function. https://www.postgresql.org/message-id/flat/CA%2Bu7OA65%2BjEFb_TyV5g%2BKq%2BonyJ2skMOPzgTgFH%2BqgLwszRqvw%40mail.gmail.com regards, tom lane

Re: [sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Tomas Vondra
On Wed, Jul 10, 2019 at 04:57:54PM -0400, Tom Lane wrote: Andreas Seltenreich writes: running sqlsmith on the regression database of REL_12_STABLE at ff597b656f yielded a crash in mcv_get_match_bitmap. I can reproduce it with the following query on the regression database: select filler1

Refactoring syslogger piping to simplify adding new log destinations

2019-07-10 Thread Sehrope Sarkuni
Hi, While working on adding a new log_destination I noticed that the syslogger piping would need to be updated. At the moment both ends only handle stderr/csvlog as the pipe message header has a char "is_last" that is either t/f (stderr last, stderr partial) or T/F (csvlog last, csvlog partial).

Re: [sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Tom Lane
Andreas Seltenreich writes: > running sqlsmith on the regression database of REL_12_STABLE at > ff597b656f yielded a crash in mcv_get_match_bitmap. I can reproduce it > with the following query on the regression database: > select filler1 from mcv_lists where a is not null and (select 42) <=

Re: buildfarm's typedefs list has gone completely nutso

2019-07-10 Thread Andrew Dunstan
On 7/10/19 1:34 PM, Andres Freund wrote: > > Hm, it has gotten gcc-9 installed recently, but calliphoridae isn't > using that. So it's probably not the compiler side. But I also see a > binutils upgrade: > > 2019-07-08 06:22:48 upgrade binutils-multiarch:amd64 2.31.1-16 > 2.32.51.20190707-1 >

[sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Andreas Seltenreich
Hi, running sqlsmith on the regression database of REL_12_STABLE at ff597b656f yielded a crash in mcv_get_match_bitmap. I can reproduce it with the following query on the regression database: select filler1 from mcv_lists where a is not null and (select 42) <= c; Backtrace below. regards,

Re: let's make the list of reportable GUCs configurable (was Re: Add %r substitution for psql prompts to show recovery status)

2019-07-10 Thread Tom Lane
Robert Haas writes: > On Wed, Jul 10, 2019 at 9:59 AM Dave Cramer wrote: >> I'm still a bit conflicted about what to do with search_path as I do believe >> this is potentially a security issue. >> It may be that we always want to report that and possibly back patch it. > I don't see that as a

Re: let's make the list of reportable GUCs configurable (was Re: Add %r substitution for psql prompts to show recovery status)

2019-07-10 Thread Robert Haas
On Wed, Jul 10, 2019 at 9:59 AM Dave Cramer wrote: > I'm still a bit conflicted about what to do with search_path as I do believe > this is potentially a security issue. > It may be that we always want to report that and possibly back patch it. I don't see that as a feasible option unless we

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-10, Bruce Momjian wrote: > Uh, what if a transaction modifies page 0 and page 1 of the same table > --- don't those pages have the same LSN. No, because WAL being a physical change log, each page gets its own WAL record with its own LSN. -- Álvaro Herrera

Re: coypu: "FATAL: sorry, too many clients already"

2019-07-10 Thread Rémi Zara
> Le 10 juil. 2019 à 04:09, Tom Lane a écrit : > > I don't see a really good reason to be using force_parallel_mode on > such a low-end box, and would recommend taking that out in any case. > If the box's SysV IPC limits can't be raised, it might be a good idea > to restrict the maximum test

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 03:53:55PM -0400, Alvaro Herrera wrote: > On 2019-Jul-10, Bruce Momjian wrote: > > > Good, so I think we all now agree we have to put the nonce > > (pg_class.oid, LSN, page-number) though the cipher using the secret. > > Actually, why do you need the page number in the

Re: POC: Cleaning up orphaned files using undo logs

2019-07-10 Thread Robert Haas
On Wed, Jul 10, 2019 at 12:36 PM Robert Haas wrote: > Broadly, you are correct to point out that you need to avoid chasing > stale pointers, and there are a bunch of ways to accomplish that: > approach #1 avoids using real pointers, and the rest just make sure > that any stale pointers don't

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Stephen Frost
Greetings, * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > On 2019-Jul-10, Bruce Momjian wrote: > > > Good, so I think we all now agree we have to put the nonce > > (pg_class.oid, LSN, page-number) though the cipher using the secret. > > Actually, why do you need the page number in the

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-10, Bruce Momjian wrote: > Good, so I think we all now agree we have to put the nonce > (pg_class.oid, LSN, page-number) though the cipher using the secret. Actually, why do you need the page number in the nonce? The LSN already distinguishes pages -- you can't have two pages with

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 02:57:54PM -0400, Stephen Frost wrote: > Greetings, > > * Ryan Lambert (r...@rustprooflabs.com) wrote: > > > I didn't either, except it was referenced above as "forward hash". I > > > don't know why that was suggested, which is why I listed it as an > > >

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-10 Thread Julien Rouhaud
Hi Alvaro, Thanks a lot for the review On Wed, Jul 10, 2019 at 4:15 PM Alvaro Herrera wrote: > > On 2019-Jul-09, Julien Rouhaud wrote: > > > I finished to do a better refactoring, and ended up with this API in > > scripts_parallel: > > Looking good! Thanks! > I'm not sure about the "Consume"

Re: pg_receivewal documentation

2019-07-10 Thread Laurenz Albe
On Wed, 2019-07-10 at 17:04 +0900, Michael Paquier wrote: > Hmm. synchronous_commit is user-settable, which means that it is > possible to enforce a value in the connection string doing the > connection. Isn't that something we had better enforce directly in > the tool? In this case what could

Re: POC: Cleaning up orphaned files using undo logs

2019-07-10 Thread Robert Haas
On Tue, Jul 9, 2019 at 6:28 AM Dilip Kumar wrote: > PFA, updated patch version which includes > - One defect fix in undo interface related to undo page compression > for handling persistence level > - Implemented pending TODO optimization in undo page compression. > - One defect fix in undo

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Stephen Frost
Greetings, * Ryan Lambert (r...@rustprooflabs.com) wrote: > > I didn't either, except it was referenced above as "forward hash". I > > don't know why that was suggested, which is why I listed it as an > > option/suggestion. > > My bad, sorry for the confusion! I meant to say "cipher" not

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Ryan Lambert
> I didn't either, except it was referenced above as "forward hash". I > don't know why that was suggested, which is why I listed it as an > option/suggestion. My bad, sorry for the confusion! I meant to say "cipher" not "hash". I was (trying to) refer to the method of generating unpredictable

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 02:44:30PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Wed, Jul 10, 2019 at 12:38:02PM -0600, Ryan Lambert wrote: > > > > > > what is it that gets stored in the page for > > > decryption use, the nonce or the IV

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-10 Thread Tom Lane
Amit Langote writes: > Attached updated patch. Thanks again. Pushed with a bit of further cleanup --- most notably, the way you had execute_sql_string(), it was still leaking any cruft ProcessUtility might generate. We can fix that by running ProcessUtility in the per-statement context too. I

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Wed, Jul 10, 2019 at 12:38:02PM -0600, Ryan Lambert wrote: > > > > what is it that gets stored in the page for > > decryption use, the nonce or the IV derived from it? > > > > > > I believe storing the IV is preferable and still

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 12:38:02PM -0600, Ryan Lambert wrote: > > what is it that gets stored in the page for > decryption use, the nonce or the IV derived from it? > > > I believe storing the IV is preferable and still secure per [1]: "The IV need > not be secret" > > Beyond needing

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Stephen Frost
Greetings, * Ryan Lambert (r...@rustprooflabs.com) wrote: > > what is it that gets stored in the page for > > decryption use, the nonce or the IV derived from it? > > I believe storing the IV is preferable and still secure per [1]: "The IV > need not be secret" > > Beyond needing the database

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Ryan Lambert
> what is it that gets stored in the page for > decryption use, the nonce or the IV derived from it? I believe storing the IV is preferable and still secure per [1]: "The IV need not be secret" Beyond needing the database oid, if every decrypt function has to regenerate the IV from the nonce

Re: doc: minor update for description of "pg_roles" view

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 02:35:56PM +0900, Ian Barwick wrote: > Hi > > Here: > > https://www.postgresql.org/docs/12/view-pg-roles.html > > we state: > > "This view explicitly exposes the OID column of the underlying table, >since that is needed to do joins to other catalogs." > > I

Re: block-level incremental backup

2019-07-10 Thread Anastasia Lubennikova
23.04.2019 14:08, Anastasia Lubennikova wrote: I'm volunteering to write a draft patch or, more likely, set of patches, which will allow us to discuss the subject in more detail. And to do that I wish we agree on the API and data format (at least broadly). Looking forward to hearing your

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 01:04:47PM -0400, Alvaro Herrera wrote: > On 2019-Jul-10, Bruce Momjian wrote: > > > * Using the LSN as part of the nonce fixes both problems, and has a > >third benefit: > > > > * We don't need to decrypt/re-encrypt during CREATE DATABASE since > >the

Re: buildfarm's typedefs list has gone completely nutso

2019-07-10 Thread Tom Lane
Andres Freund writes: > On 2019-07-10 12:57:08 -0400, Tom Lane wrote: >> Was there any change in calliphoridae's toolchain this month? > Hm, it has gotten gcc-9 installed recently, but calliphoridae isn't > using that. So it's probably not the compiler side. But I also see a > binutils upgrade:

Re: buildfarm's typedefs list has gone completely nutso

2019-07-10 Thread Andres Freund
Hi, On 2019-07-10 12:57:08 -0400, Tom Lane wrote: > The current HEAD typedefs list available from > https://buildfarm.postgresql.org/cgi-bin/typedefs.pl > has the following interesting additions compared to where > things were on July 1: > > 2 > ECPGt_bytea > connection_name >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-10, Bruce Momjian wrote: > * Using the LSN as part of the nonce fixes both problems, and has a >third benefit: > > * We don't need to decrypt/re-encrypt during CREATE DATABASE since >the page contents are the same in both places, and once one >database

Re: (select query)/relation as first class citizen

2019-07-10 Thread Dent John
Hi Roman, Pavel, I was interested in this post, as it’s a topic I’ve stumbled upon in the past. There are two topics at play here: 1. The ability to flexibly craft queries from procedural language functions 2. Support for pipelined access to SETOF/TABLEs from procedural language functions

buildfarm's typedefs list has gone completely nutso

2019-07-10 Thread Tom Lane
The current HEAD typedefs list available from https://buildfarm.postgresql.org/cgi-bin/typedefs.pl has the following interesting additions compared to where things were on July 1: 2 ECPGt_bytea connection_name in_addr pg_fprintf send_appname The

Re: [proposal] de-TOAST'ing using a iterator

2019-07-10 Thread Binguo Bao
This is the patch that fix warnings. Best Regards, Binguo Bao Binguo Bao 于2019年7月10日周三 下午10:18写道: > Hi Thomas, > I've fixed the warnings. > > Thomas Munro 于2019年7月5日周五 下午12:21写道: > >> On Thu, Jun 20, 2019 at 1:51 AM Binguo Bao wrote: >> > Hi hackers! >> > This proposal aims to provide the

Re: POC: Cleaning up orphaned files using undo logs

2019-07-10 Thread Robert Haas
On Wed, Jul 10, 2019 at 2:32 AM Amit Kapila wrote: > As of now, after we finish executing the rollback actions, the entry > from the hash table is removed. Now, at a later time (when queues are > full and we want to insert a new entry) when we access the queue entry > (to check whether we can

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 08:31:17AM -0400, Joe Conway wrote: > Please see my other reply (and > https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf > appendix C as pointed out by Ryan downthread). > > At least in my mind, I trust a published specification from the >

Re: pg_checksums (or checksums in general) vs tableam

2019-07-10 Thread Andres Freund
Hi, On July 10, 2019 9:12:18 AM PDT, Magnus Hagander wrote: >On Wed, Jul 10, 2019 at 3:05 PM Michael Paquier >wrote: > >> On Wed, Jul 10, 2019 at 11:42:34AM +0200, Magnus Hagander wrote: >> > pg_checksums enumerate the files. What if there are files there >from a >> > different tableam? Isn't

Re: pg_checksums (or checksums in general) vs tableam

2019-07-10 Thread Magnus Hagander
On Wed, Jul 10, 2019 at 3:05 PM Michael Paquier wrote: > On Wed, Jul 10, 2019 at 11:42:34AM +0200, Magnus Hagander wrote: > > pg_checksums enumerate the files. What if there are files there from a > > different tableam? Isn't pg_checksums just going to badly fail then, > since > > it assumes

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-10 Thread Ashwin Agrawal
On Wed, Jul 10, 2019 at 6:46 AM Tom Lane wrote: > Michael Paquier writes: > > On Wed, Jul 10, 2019 at 12:51:28PM +0530, Amit Kapila wrote: > >> It would be good if we can come up with something like that. It will > >> be helpful for zheap, where in some cases we get different row > >> ordering

Re: FETCH FIRST clause PERCENT option

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-08, Surafel Temesgen wrote: > Hi Thomas, > Thank you for informing me > > Hi Surafel, > > > > There's a call to adjust_limit_rows_costs() hiding under > > contrib/postgres_fdw, so this fails check-world. > > > > Fixed . I also include the review given by Ryan in attached patch

Re: pg_receivewal documentation

2019-07-10 Thread Jesper Pedersen
Hi, On 7/10/19 10:24 AM, Alvaro Herrera wrote: +1 to document this caveat. How about Note that while WAL will be flushed with this setting, pg_receivewal never applies it, so must not be set to remote_apply if pg_receivewal is the only synchronous

Re: Index Skip Scan

2019-07-10 Thread Dmitry Dolgov
> On Wed, Jul 10, 2019 at 5:00 PM Floris Van Nee > wrote: > > > > Thanks for testing! Could you provide a test case to show what exactly is > > the > > problem? > > Note that in the case of a regular non-skip scan, this cursor backwards works > because the Unique node on top does not support

Re: Index Skip Scan

2019-07-10 Thread Floris Van Nee
> Thanks for testing! Could you provide a test case to show what exactly is the > problem? Note that in the case of a regular non-skip scan, this cursor backwards works because the Unique node on top does not support backwards scanning at all. Therefore, when creating the cursor, the actual

Re: Index Skip Scan

2019-07-10 Thread Dmitry Dolgov
> On Wed, Jul 10, 2019 at 4:52 PM Floris Van Nee > wrote: > > > Thanks for testing! Could you provide a test case to show what exactly is > > the > > problem? > > create table a (a int, b int, c int); > insert into a (select vs, ks, 10 from generate_series(1,5) vs, > generate_series(1, 1)

Re: range_agg

2019-07-10 Thread Paul Jungwirth
On 7/9/19 11:24 PM, David Fetter wrote: I seem to recall that the usual convention (at least in math) is to use intervals that are generally represented as open on the infinity side, but that might not fit how we do things. I think it does, unless I'm misunderstanding? Oh, I was just

Re: Index Skip Scan

2019-07-10 Thread Floris Van Nee
> Thanks for testing! Could you provide a test case to show what exactly is the > problem? create table a (a int, b int, c int); insert into a (select vs, ks, 10 from generate_series(1,5) vs, generate_series(1, 1) ks); create index on a (a,b); analyze a; set enable_indexskipscan=1; //

Re: Index Skip Scan

2019-07-10 Thread Dmitry Dolgov
> On Tue, Jul 2, 2019 at 2:27 PM David Rowley > wrote: > > The more I think about these UniqueKeys, the more I think they need to > be a separate concept to PathKeys. For example, UniqueKeys: { x, y } > should be equivalent to { y, x }, but with PathKeys, that's not the > case, since the order

Re: Optimize partial TOAST decompression

2019-07-10 Thread Tomas Vondra
On Wed, Jul 10, 2019 at 01:35:25PM +0800, Binguo Bao wrote: Tomas Vondra 于2019年7月10日周三 上午5:12写道: On Sat, Jul 06, 2019 at 05:23:37PM +0200, Tomas Vondra wrote: >On Sat, Jul 06, 2019 at 02:27:56AM +0800, Binguo Bao wrote: >>Hi, Tomas! >>Thanks for your testing and the suggestion. >> >>That's

Re: Index Skip Scan

2019-07-10 Thread Floris Van Nee
> Here is finally a new version of the patch, where all the mentioned issues > seems to be fixed and the corresponding new tests should keep it like that > (I've skipped all the pubs at PostgresLondon for that). Thanks for the new patch! Really appreciate the work you're putting into it. I

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Tomas Vondra
On Wed, Jul 10, 2019 at 03:38:54PM +0900, Masahiko Sawada wrote: On Tue, Jul 9, 2019 at 9:01 PM Joe Conway wrote: On 7/9/19 6:07 AM, Peter Eisentraut wrote: > On 2019-07-08 18:09, Joe Conway wrote: >> In my mind, and in practice to a >> large extent, a postgres tablespace == a unique mount

Re: pg_receivewal documentation

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-09, Jesper Pedersen wrote: > + > +Note that while WAL will be flushed with this setting, > +it will never be applied, so linkend="guc-synchronous-commit"/> must > +not be set to remote_apply if > pg_receivewal > +is the only synchronous

Re: [proposal] de-TOAST'ing using a iterator

2019-07-10 Thread Binguo Bao
Hi Thomas, I've fixed the warnings. Thomas Munro 于2019年7月5日周五 下午12:21写道: > On Thu, Jun 20, 2019 at 1:51 AM Binguo Bao wrote: > > Hi hackers! > > This proposal aims to provide the ability to de-TOAST a fully TOAST'd > and compressed field using an iterator and then update the appropriate >

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-09, Julien Rouhaud wrote: > I finished to do a better refactoring, and ended up with this API in > scripts_parallel: Looking good! I'm not sure about the "Consume" word in ConsumeIdleSlot; maybe "Reserve"? "Obtain"? "Get"? Code commentary: I think the comment that sits atop the

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Tomas Vondra
On Tue, Jul 09, 2019 at 10:06:33PM -0400, Stephen Frost wrote: Greetings, * Ryan Lambert (r...@rustprooflabs.com) wrote: > What I think Tomas is getting at here is that we don't write a page only > once. Yes, that's what I meant. > A nonce of tableoid+pagenum will only be unique the first

Re: let's make the list of reportable GUCs configurable (was Re: Add %r substitution for psql prompts to show recovery status)

2019-07-10 Thread Dave Cramer
On Wed, 10 Jul 2019 at 09:11, Robert Haas wrote: > On Tue, Jul 9, 2019 at 2:32 PM Dave Cramer wrote: > > So did this die from lack of interest? > > > > I have proposed in another thread adding more GUC REPORT variables, but > I see this as a much better way. > > > > I'm willing to code the

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-10 Thread Tom Lane
Michael Paquier writes: > On Wed, Jul 10, 2019 at 12:51:28PM +0530, Amit Kapila wrote: >> It would be good if we can come up with something like that. It will >> be helpful for zheap, where in some cases we get different row >> ordering due to in-place updates. As of now, we try to add Order By

Re: Contribution to Perldoc for TestLib module in Postgres

2019-07-10 Thread Alvaro Herrera
On 2019-Apr-11, Iwata, Aya wrote: > In the above document, why not write a description after the function name? > I think it is better to write the function name first and then the > description. > In your code; > #Checks if all the tests passed or not > all_tests_passing() > > In my

Re: global / super barriers (for checksums)

2019-07-10 Thread Magnus Hagander
On Tue, Oct 30, 2018 at 6:16 AM Andres Freund wrote: > Hi, > > Magnus cornered me at pgconf.eu and asked me whether I could prototype > the "barriers" I'd been talking about in the online checksumming thread. > > The problem there was to make sure that all processes, backends and > auxiliary

Re: progress report for ANALYZE

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-10, Robert Haas wrote: > On Tue, Jul 9, 2019 at 6:12 PM Alvaro Herrera > wrote: > > Hmm, ok. In CREATE INDEX, we use the block counters multiple times. > > Why do we do that? Because we scan the table first, then the index, then the table again (last two for the validation phase

Re: progress report for ANALYZE

2019-07-10 Thread Robert Haas
On Tue, Jul 9, 2019 at 6:12 PM Alvaro Herrera wrote: > Hmm, ok. In CREATE INDEX, we use the block counters multiple times. Why do we do that? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: let's make the list of reportable GUCs configurable (was Re: Add %r substitution for psql prompts to show recovery status)

2019-07-10 Thread Robert Haas
On Tue, Jul 9, 2019 at 2:32 PM Dave Cramer wrote: > So did this die from lack of interest? > > I have proposed in another thread adding more GUC REPORT variables, but I see > this as a much better way. > > I'm willing to code the patch if we can get some buy in here ? It seemed like most people

Re: Index Skip Scan

2019-07-10 Thread Jesper Pedersen
Hi, On 7/9/19 10:14 PM, Thomas Munro wrote: Thomas, do you have any ideas for this ? I can see that MySQL did the functionality in two change sets (base and function support), but like you said we shouldn't paint ourselves into a corner. I think amskip() could be augmented by later patches to

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-10 Thread Melanie Plageman
On Wed, Jul 10, 2019 at 12:40 AM Michael Paquier wrote: > On Wed, Jul 10, 2019 at 12:51:28PM +0530, Amit Kapila wrote: > > It would be good if we can come up with something like that. It will > > be helpful for zheap, where in some cases we get different row > > ordering due to in-place

Re: pg_checksums (or checksums in general) vs tableam

2019-07-10 Thread Michael Paquier
On Wed, Jul 10, 2019 at 11:42:34AM +0200, Magnus Hagander wrote: > pg_checksums enumerate the files. What if there are files there from a > different tableam? Isn't pg_checksums just going to badly fail then, since > it assumes everything is heap? > > Also, do we allow AMs that don't support

  1   2   >