Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Heikki Linnakangas
On 05/08/2019 07:23, Thomas Munro wrote: On Mon, Aug 5, 2019 at 3:54 PM Amit Kapila wrote: On Sun, Aug 4, 2019 at 2:46 PM Heikki Linnakangas wrote: Could we leave out the UNDO and discard worker processes for now? Execute all UNDO actions immediately at rollback, and after crash recovery.

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-08-05 Thread Kyotaro Horiguchi
At Sun, 4 Aug 2019 00:04:01 -0700 (MST), legrand legrand wrote in <1564902241482-0.p...@n3.nabble.com> > > However having the nested queryid in > > pg_stat_activity would be convenient to track > > what is a long stored functions currently doing. > > +1 > > And this could permit to get wait

SSL Connection still showing TLSv1.3 even it is disabled in ssl_ciphers

2019-08-05 Thread tushar
Hi , While testing SSL version 1.1.1c , I only enabled TLSv1.2 and rest including TLSv1.3 has been disabled , like this - postgres=# show ssl_ciphers ; ssl_ciphers --  TLSv1.2:!aNULL:!SSLv2:!SSLv3:!TLSv1:!TLSv1.3 To cofirm the

Re: Parallel Append subplan order instability on aye-aye

2019-08-05 Thread Thomas Munro
On Wed, Jul 24, 2019 at 11:59 AM Thomas Munro wrote: > On Tue, Jul 16, 2019 at 12:21 PM Tom Lane wrote: > > In the meantime, we've had *lots* of buildfarm failures in the > > added pg_stat_all_tables query, which indicate that indeed the > > stats collector mechanism isn't terribly reliable.

Re: Ought to use heap_multi_insert() for pg_attribute/depend insertions?

2019-08-05 Thread Daniel Gustafsson
> On 2 Aug 2019, at 00:41, Thomas Munro wrote: > > On Tue, Jul 9, 2019 at 11:07 PM Daniel Gustafsson wrote: >> The attached v3 also has that fix in order to see if the cfbot is happier >> with >> this. > > Noticed while moving this to the next CF: > > heap.c: In function ‘StorePartitionKey’:

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Ian Barwick
On 8/4/19 4:13 AM, Tom Lane wrote: Ian Barwick writes: On 8/3/19 7:27 AM, Tom Lane wrote: Tomas Vondra writes: The main issue however is that no code was written yet. Seems like it ought to be relatively simple ... but I didn't look. The patch I originally sent does exactly this.

Re: pglz performance

2019-08-05 Thread Michael Paquier
On Fri, Aug 02, 2019 at 07:52:39PM +0200, Tomas Vondra wrote: > On Fri, Aug 02, 2019 at 10:12:58AM -0700, Andres Freund wrote: >> Why would they be stuck continuing to *compress* with pglz? As we >> fully retoast on write anyway we can just gradually switch over to the >> better algorithm.

Re: More issues with pg_verify_checksums and checksum verification in base backups

2019-08-05 Thread Michael Paquier
On Sat, Aug 03, 2019 at 06:47:48PM +0200, Michael Banck wrote: > first off, a bit of a meta-question: Did the whitelist approach die > completely, or are we going to tackle it again for v13 or later? At this stage, it is burried. Amen. > This is something I still have in the test suite of my

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-08-05 Thread Julien Rouhaud
On Mon, Aug 5, 2019 at 9:28 AM Kyotaro Horiguchi wrote: > > At Sun, 4 Aug 2019 00:04:01 -0700 (MST), legrand legrand > wrote in <1564902241482-0.p...@n3.nabble.com> > > > However having the nested queryid in > > > pg_stat_activity would be convenient to track > > > what is a long stored

Re: Remove HeapTuple and Buffer dependency for predicate locking functions

2019-08-05 Thread Thomas Munro
On Sat, Aug 3, 2019 at 11:56 AM Ashwin Agrawal wrote: > On Wed, Jul 31, 2019 at 2:06 PM Andres Freund wrote: >> I'm also a bit confused why we don't need to pass in the offset of the >> current tuple, rather than the HOT root tuple here. That's not related >> to this patch. But aren't we

Re: partition routing layering in nodeModifyTable.c

2019-08-05 Thread Amit Langote
Hi Andres, Fujita-san, On Sat, Aug 3, 2019 at 3:01 AM Andres Freund wrote: > On 2019-07-31 17:04:38 +0900, Amit Langote wrote: > > I looked into trying to do the things I mentioned above and it seems > > to me that revising BeginDirectModify()'s API to receive the > > ResultRelInfo directly as

Re: Ought to use heap_multi_insert() for pg_attribute/depend insertions?

2019-08-05 Thread Michael Paquier
On Mon, Aug 05, 2019 at 09:20:07AM +0200, Daniel Gustafsson wrote: > Thanks, the attached v4 updates to patch to handle a0555ddab9b672a046 as well. + if (referenced->numrefs == 1) + recordDependencyOn(depender, >refs[0], behavior); + else + recordMultipleDependencies(depender, +

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Ian Barwick
On 8/4/19 1:59 AM, Tom Lane wrote:> Tomas Vondra writes: >> On Fri, Aug 02, 2019 at 06:08:02PM -0700, Andres Freund wrote: >>> We're WAY WAY past feature freeze. This isn't the time to rewrite guc.c, >>> guc-file.l to be suitable for running outside of a backend environment. > >> Right. And

Re: concerns around pg_lsn

2019-08-05 Thread Michael Paquier
On Mon, Aug 05, 2019 at 09:15:02AM +0530, Jeevan Ladhe wrote: > Please find attached patch with the changes to RETURN_ERROR and > it's references in float.c Thanks. Committed after applying some tweaks to it. I have noticed that you forgot numeric_int4_opt_error() in the set. -- Michael

Re: concerns around pg_lsn

2019-08-05 Thread Jeevan Ladhe
> > Thanks. Committed after applying some tweaks to it. I have noticed > that you forgot numeric_int4_opt_error() in the set. Oops. Thanks for the commit, Michael. Regards, Jeevan Ladhe

Re: pglz performance

2019-08-05 Thread Andres Freund
Hi, On 2019-08-05 00:19:28 +0200, Petr Jelinek wrote: > It carries that information inside the compressed value, like I said in the > other reply, that's why the extra byte. I'm not convinced that that is a good plan - imo the reference to the compressed data should carry that information. I.e.

Re: Fix typos and inconsistencies for HEAD (take 9)

2019-08-05 Thread Alexander Lakhin
05.08.2019 8:40, Michael Paquier wrote: > On Mon, Aug 05, 2019 at 06:44:46AM +0300, Alexander Lakhin wrote: >> I believe that all the fixes in doc/ should be back-patched too. If it's >> not too late, I can produce such patches for the nearest releases. > I think that's unfortunately a bit too

Re: Identity columns should own only one sequence

2019-08-05 Thread Laurenz Albe
Peter Eisentraut wrote: > On 2019-05-08 16:49, Laurenz Albe wrote: > > I believe we should have both: > > > > - Identity columns should only use sequences with an INTERNAL dependency, > > as in Peter's patch. > > I have committed this. Since this is a bug fix, shouldn't it be backpatched?

Re: psql's meta command \d is broken as of 12 beta2.

2019-08-05 Thread Pavel Stehule
Hi po 5. 8. 2019 v 13:35 odesílatel Dmitry Igrishin napsal: > Hi, > > Sorry if this report is duplicate but there is no column relhasoids in > pg_catalog.pg_class required by the underlying query of the \d meta > command of psql. > do you use psql from this release? The psql client should be

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Tom Lane
David Fetter writes: > On Mon, Aug 05, 2019 at 03:52:07PM +0900, Ian Barwick wrote: >> On 8/4/19 1:59 AM, Tom Lane wrote: >>> I think we should just accept the facts on the ground, which are that >>> some tools modify pg.auto.conf by appending to it >> +1. It's just a text file... > So are

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Robert Haas
On Mon, Aug 5, 2019 at 12:25 PM Tom Lane wrote: > Perhaps as a future improvement, but it's not happening for v12, > at least not unless you accept "use ALTER SYSTEM in a standalone > backend" as a usable answer. I'm not sure that would even work for the cases at issue ... because we're talking

Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Andres Freund
Hi, On 2019-08-05 11:25:10 -0400, Robert Haas wrote: > The obvious thing to do seems to be to have UndoLogControl objects own > SmgrRelations. That would be something of a novelty, since it looks > like currently only a Relation ever owns an SMgrRelation, but the smgr > infrastructure seems to

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Robert Haas writes: > > On Mon, Aug 5, 2019 at 12:25 PM Tom Lane wrote: > >> Perhaps as a future improvement, but it's not happening for v12, > >> at least not unless you accept "use ALTER SYSTEM in a standalone > >> backend" as a usable

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2019-08-05 12:43:24 -0400, Tom Lane wrote: > > On the other hand, people have also opined that they want full error > > checking on the inserted values, and that seems out of reach with less > > than a full running system (mumble

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread David Fetter
On Mon, Aug 05, 2019 at 03:52:07PM +0900, Ian Barwick wrote: > On 8/4/19 1:59 AM, Tom Lane wrote:> Tomas Vondra > writes: > >> On Fri, Aug 02, 2019 at 06:08:02PM -0700, Andres Freund wrote: > >>> We're WAY WAY past feature freeze. This isn't the time to rewrite guc.c, > >>> guc-file.l to be

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-08-05 Thread legrand legrand
Kyotaro Horiguchi-4 wrote > At Sun, 4 Aug 2019 00:04:01 -0700 (MST), legrand legrand > legrand_legrand@ > wrote in < > 1564902241482-0.post@.nabble >> >> > However having the nested queryid in >> > pg_stat_activity would be convenient to track >> > what is a long stored functions currently

Re: Problem with default partition pruning

2019-08-05 Thread Alvaro Herrera
On 2019-Aug-05, yuzuko wrote: > So I proposed moving the if() block to the current place. > The latest patch can solve both queries but I found the latter > problem can be solved by setting constraint_exclusion = on. So we have three locations for that test; one is where it currently is, which

Re: pgbench - implement strict TPC-B benchmark

2019-08-05 Thread Fabien COELHO
Hello Andres, If not, do you think advisable to spend time improving the evaluator & variable stuff and possibly other places for an overall 15% gain? Also, what would be the likelyhood of such optimization patch to pass? I could do a limited variable management improvement patch,

Re: pgbench - implement strict TPC-B benchmark

2019-08-05 Thread Andres Freund
Hi, On 2019-08-05 17:38:23 +0200, Fabien COELHO wrote: > Which is a (somehow disappointing) * 3.3 speedup. The impact on the 3 > complex expressions tests is not measurable, though. I don't know why that could be disappointing. We put in much more work for much smaller gains in other places. >

Re: Adding column "mem_usage" to view pg_prepared_statements

2019-08-05 Thread Konstantin Knizhnik
On 31.07.2019 1:38, Daniel Migowski wrote: Am 31.07.2019 um 00:29 schrieb Tom Lane: Daniel Migowski writes: Ok, just have read about the commitfest thing. Is the patch OK for that? Or is there generally no love for a mem_usage column here? If it was, I really would add some memory

Re: Remove HeapTuple and Buffer dependency for predicate locking functions

2019-08-05 Thread Andres Freund
Hi, On 2019-08-05 20:58:05 +1200, Thomas Munro wrote: > On Sat, Aug 3, 2019 at 11:56 AM Ashwin Agrawal wrote: > > On Wed, Jul 31, 2019 at 2:06 PM Andres Freund wrote: > >> I'm also a bit confused why we don't need to pass in the offset of the > >> current tuple, rather than the HOT root

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Robert Haas
On Mon, Aug 5, 2019 at 10:21 AM Stephen Frost wrote: > Just to be clear, I brought up this exact concern back in *November*: > > https://www.postgresql.org/message-id/20181127153405.GX3415%40tamriel.snowman.net > > And now because this was pushed forward and the concerns that I raised > ignored,

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Aug 5, 2019 at 10:21 AM Stephen Frost wrote: > > Just to be clear, I brought up this exact concern back in *November*: > > > > https://www.postgresql.org/message-id/20181127153405.GX3415%40tamriel.snowman.net > > > > And now

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Tom Lane
Tomas Vondra writes: > On Mon, Aug 05, 2019 at 10:21:39AM -0400, Stephen Frost wrote: >> I'd be happier with one set of code at least being the recommended >> approach to modifying the file and only one set of code in our codebase >> that's messing with .auto.conf, so that, hopefully, it's done

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Tom Lane
Isaac Morland writes: > Here's a radical suggestion: replace postgresql.auto.conf with a directory > containing multiple files. Each file is named after a configuration > parameter, and its content is the value of the parameter. Hmm ... that seems like a lot of work and code churn --- in

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Tom Lane
Robert Haas writes: > On Mon, Aug 5, 2019 at 12:25 PM Tom Lane wrote: >> Perhaps as a future improvement, but it's not happening for v12, >> at least not unless you accept "use ALTER SYSTEM in a standalone >> backend" as a usable answer. > I'm not sure that would even work for the cases at

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Christoph Berg
Re: Tomas Vondra 2019-08-03 <20190803124111.2aaddumd7url5wmq@development> > If we really want to give external tools a sensible (and optional) API > to access the file, a simple command-line tool seems much better. Say we > have something like > > pg_config_file -f PATH --set KEY VALUE >

Re: [HACKERS] Cached plans and statement generalization

2019-08-05 Thread Konstantin Knizhnik
On 01.08.2019 19:56, Konstantin Knizhnik wrote: On 31.07.2019 19:56, Heikki Linnakangas wrote: On 09/07/2019 23:59, Konstantin Knizhnik wrote: Fixed patch version of the path is attached. Much of the patch and the discussion has been around the raw parsing, and guessing which literals

Re: Adding column "mem_usage" to view pg_prepared_statements

2019-08-05 Thread Andres Freund
Hi, On 2019-07-28 06:20:40 +, Daniel Migowski wrote: > how do you want to generalize it? Are you thinking about a view solely > for the display of the memory usage of different objects? I'm not quite sure. I'm just not sure that adding separate infrastructure for various objects is a

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Tomas Vondra
On Mon, Aug 05, 2019 at 10:21:39AM -0400, Stephen Frost wrote: Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: On Fri, Aug 02, 2019 at 06:38:46PM -0400, Stephen Frost wrote: >On Fri, Aug 2, 2019 at 18:27 Tom Lane wrote: >>Tomas Vondra writes: >>> There seems to be a consensus

Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Robert Haas
On Tue, Jul 30, 2019 at 4:02 AM Andres Freund wrote: > I'm a bit worried about expanding the use of > ReadBufferWithoutRelcache(). Not so much because of the relcache itself, > but because it requires doing separate smgropen() calls. While not > crazily expensive, it's also not free. Especially

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Andres Freund
Hi, On 2019-08-05 12:43:24 -0400, Tom Lane wrote: > Yeah, good point. There are a lot of other cases where you really > don't want system startup to happen, too. Agreed. > On the other hand, people have also opined that they want full error > checking on the inserted values, and that seems

Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Amit Kapila
On Mon, Aug 5, 2019 at 12:09 PM Heikki Linnakangas wrote: > > On 05/08/2019 07:23, Thomas Munro wrote: > > On Mon, Aug 5, 2019 at 3:54 PM Amit Kapila wrote: > >> On Sun, Aug 4, 2019 at 2:46 PM Heikki Linnakangas wrote: > >>> Could we leave out the UNDO and discard worker processes for now? >

Re: block-level incremental backup

2019-08-05 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Fri, Aug 2, 2019 at 9:13 AM vignesh C wrote: > > + rc = system(copycmd); > > I don't think this patch should be calling system() in the first place. +1. Thanks, Stephen signature.asc Description: PGP signature

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Isaac Morland
Here's a radical suggestion: replace postgresql.auto.conf with a directory containing multiple files. Each file is named after a configuration parameter, and its content is the value of the parameter. So to remove a special configuration parameter, delete its file. To set it, write the file,

Re: Problem with default partition pruning

2019-08-05 Thread Alvaro Herrera
On 2019-Aug-05, yuzuko wrote: > However, I'm still concerned that the block > - > ... > - > is written in the right place as Amit explained [1]. Yeah, I have that patch installed locally and I'm looking about it. Thanks for the reminder. I also have an eye on Horiguchi's patch elsewhere

Re: SSL Connection still showing TLSv1.3 even it is disabled in ssl_ciphers

2019-08-05 Thread Tom Lane
tushar writes: > when  i connect to psql terminal - > psql.bin (10.9) > SSL connection (protocol: TLSv1.3, cipher: *TLS_AES_256_GCM_SHA384*, > bits: 256, compression: off) > Type "help" for help. > postgres=# show ssl_ciphers ; > ssl_ciphers >

Re: Identity columns should own only one sequence

2019-08-05 Thread Laurenz Albe
Peter Eisentraut wrote: > On 2019-05-08 16:49, Laurenz Albe wrote: > > I believe we should have both: > > > > - Identity columns should only use sequences with an INTERNAL dependency, > > as in Peter's patch. > > I have committed this. Thanks! > > - When a column default is dropped, remove

psql's meta command \d is broken as of 12 beta2.

2019-08-05 Thread Dmitry Igrishin
Hi, Sorry if this report is duplicate but there is no column relhasoids in pg_catalog.pg_class required by the underlying query of the \d meta command of psql.

Re: psql's meta command \d is broken as of 12 beta2.

2019-08-05 Thread Pavel Stehule
po 5. 8. 2019 v 13:46 odesílatel Dmitry Igrishin napsal: > пн, 5 авг. 2019 г. в 14:42, Pavel Stehule : > > > > Hi > > > > > > po 5. 8. 2019 v 13:35 odesílatel Dmitry Igrishin > napsal: > >> > >> Hi, > >> > >> Sorry if this report is duplicate but there is no column relhasoids in > >>

Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Robert Haas
On Mon, Aug 5, 2019 at 6:16 AM Amit Kapila wrote: > For zheap, we collect all the records of a page, apply them together > and then write the entire page in WAL. The progress of transaction is > updated at either transaction end (rollback complete) or after > processing some threshold of undo

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > On Fri, Aug 02, 2019 at 06:38:46PM -0400, Stephen Frost wrote: > >On Fri, Aug 2, 2019 at 18:27 Tom Lane wrote: > >>Tomas Vondra writes: > >>> There seems to be a consensus that this this not a pg_basebackup issue > >>> (i.e.

Re: Optimize single tuple fetch from nbtree index

2019-08-05 Thread Floris Van Nee
Hi Peter, > Actually, having looked at the test case in more detail, that now > seems less likely. The test case seems designed to reward making it > cheaper to access one specific tuple among a fairly large group of > related tuples -- reducing the number of inner scans is not going to > be

Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Robert Haas
On Sun, Aug 4, 2019 at 5:16 AM Heikki Linnakangas wrote: > I feel that the level of abstraction is not quite right. There are a > bunch of fields, like uur_block, uur_offset, uur_tuple, that are > probably useful for some UNDO resource managers (zheap I presume), but > seem kind of arbitrary. How

Re: block-level incremental backup

2019-08-05 Thread Robert Haas
On Fri, Aug 2, 2019 at 9:13 AM vignesh C wrote: > + rc = system(copycmd); I don't think this patch should be calling system() in the first place. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Index Skip Scan

2019-08-05 Thread Floris Van Nee
Thanks for the new patch. I've reviewed the skip scan patch, but haven't taken a close look at the uniquekeys patch yet. In my previous review I mentioned that queries of the form: select distinct on(a) a,b from a where a=1; do not lead to a skip scan with the patch, even though the skip scan

Re: Removing unneeded self joins

2019-08-05 Thread Andrey Lepikhov
On 02/08/2019 04:54, Thomas Munro wrote: On Thu, Jun 27, 2019 at 6:42 PM Andrey Lepikhov wrote: Version v.17 of the patch that fix the bug see in attachment. While moving this to the September CF, I noticed that it needs to be updated for the recent pg_list.h API changes. The patch was

Re: psql's meta command \d is broken as of 12 beta2.

2019-08-05 Thread Dmitry Igrishin
пн, 5 авг. 2019 г. в 14:42, Pavel Stehule : > > Hi > > > po 5. 8. 2019 v 13:35 odesílatel Dmitry Igrishin napsal: >> >> Hi, >> >> Sorry if this report is duplicate but there is no column relhasoids in >> pg_catalog.pg_class required by the underlying query of the \d meta >> command of psql. > > >

Re: Unused header file inclusion

2019-08-05 Thread Alvaro Herrera
On 2019-Aug-04, vignesh C wrote: > Made the fixes based on your comments, updated patch has the changes > for the same. Well, you fixed the two things that seem to me quoted as examples of problems, but you didn't fix other occurrences of the same issues elsewhere. For example, you remove

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Robert Haas
On Mon, Aug 5, 2019 at 2:42 PM Tom Lane wrote: > I don't think we need to go on about it at great length, but it seems > to me that it'd be reasonable to point out that (a) you'd be well > advised not to touch the file while the postmaster is up, and (b) > last setting wins. Those things are

Cleanup of intro.sgml

2019-08-05 Thread Joshua D. Drake
-hackers, I went through and made some readability and modernization of the intro.sgml today. Patch attached. JD -- Command Prompt, Inc. || http://the.postgres.company/ || @cmdpromptinc Postgres centered full stack support, consulting and development. Advocate: @amplifypostgres || Get help:

Re: tableam vs. TOAST

2019-08-05 Thread Robert Haas
On Fri, Aug 2, 2019 at 6:42 PM Andres Freund wrote: > Hm, those all include writing, right? And for read-only we don't expect > any additional overhead, correct? The write overhead is probably too > large show a bit of function call overhead - but if so, it'd probably be > on unlogged tables? And

Re: pgbench - implement strict TPC-B benchmark

2019-08-05 Thread Robert Haas
On Fri, Aug 2, 2019 at 2:38 AM Fabien COELHO wrote: > Ok, one thread cannot feed an N core server if enough client are executed > per thread and the server has few things to do. Right ... where N is, uh, TWO. > The point I'm clumsily trying to make is that pgbench-specific overheads > are quite

Re: Redacting information from logs

2019-08-05 Thread Andres Freund
Hi, On 2019-08-05 14:32:36 -0700, Jeff Davis wrote: > On Mon, 2019-08-05 at 14:10 -0700, Andres Freund wrote: > > I was thinking that it'd just store a struct ErrParam, which'd > > reference > > the passed value and metadata like the name (for structured log > > output) and redaction category.

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Robert Haas
On Mon, Aug 5, 2019 at 1:29 PM Stephen Frost wrote: > No, that isn't the point of the controversy nor does it relate, at all, > to what you quoted above, so I don't think there's much value in > responding to the discussion about WARNING or not that you put together > below. Well, if that's not

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Robert Haas
On Mon, Aug 5, 2019 at 2:29 PM Tom Lane wrote: > Robert Haas writes: > > All we need to do to resolve this issue is have Tom commit his patch. > > I think Stephen is not being unreasonable to suggest that we need some > documentation about what external tools may safely do to pg.auto.conf. > So

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Tom Lane
Robert Haas writes: > On Mon, Aug 5, 2019 at 2:42 PM Tom Lane wrote: >> I don't think we need to go on about it at great length, but it seems >> to me that it'd be reasonable to point out that (a) you'd be well >> advised not to touch the file while the postmaster is up, and (b) >> last setting

Re: Adding column "mem_usage" to view pg_prepared_statements

2019-08-05 Thread Daniel Migowski
Am 05.08.2019 um 18:30 schrieb Konstantin Knizhnik: On 31.07.2019 1:38, Daniel Migowski wrote: Am 31.07.2019 um 00:29 schrieb Tom Lane: Daniel Migowski writes: Ok, just have read about the commitfest thing. Is the patch OK for that? Or is there generally no love for a mem_usage column here?

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Stephen Frost
Greetings, On Mon, Aug 5, 2019 at 14:11 Andres Freund wrote: > On 2019-08-05 13:34:39 -0400, Stephen Frost wrote: > > * Andres Freund (and...@anarazel.de) wrote: > > > On 2019-08-05 12:43:24 -0400, Tom Lane wrote: > > > > On the other hand, people have also opined that they want full error > >

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Tom Lane
Robert Haas writes: > On Mon, Aug 5, 2019 at 2:29 PM Tom Lane wrote: >> I think Stephen is not being unreasonable to suggest that we need some >> documentation about what external tools may safely do to pg.auto.conf. >> So somebody's got to write that. > I mean, really? We're going to document

Re: Cleanup of intro.sgml

2019-08-05 Thread Chapman Flack
On 8/5/19 3:20 PM, Joshua D. Drake wrote: > intro.sgml today. Patch attached. Things I noticed quickly: broken up in to categoriess/in to/into/ Unstructured data via JSON(or XML ?) s/Partioniing/Partitioning/ Regards, -Chap

Re: Adding column "mem_usage" to view pg_prepared_statements

2019-08-05 Thread Daniel Migowski
Am 05.08.2019 um 19:16 schrieb Andres Freund: On 2019-07-28 06:20:40 +, Daniel Migowski wrote: how do you want to generalize it? Are you thinking about a view solely for the display of the memory usage of different objects? I'm not quite sure. I'm just not sure that adding separate

Re: pgbench - implement strict TPC-B benchmark

2019-08-05 Thread Fabien COELHO
Hello Andres, Which is a (somehow disappointing) * 3.3 speedup. The impact on the 3 complex expressions tests is not measurable, though. I don't know why that could be disappointing. We put in much more work for much smaller gains in other places. Probably, but I thought I would have a

Re: Redacting information from logs

2019-08-05 Thread Jeff Davis
On Mon, 2019-08-05 at 14:10 -0700, Andres Freund wrote: > I was thinking that it'd just store a struct ErrParam, which'd > reference > the passed value and metadata like the name (for structured log > output) and redaction category. The bigger problem I see is handling > the different types of

Re: Redacting information from logs

2019-08-05 Thread Andres Freund
Hi, On 2019-08-05 14:26:44 -0700, Jeff Davis wrote: > On Sun, 2019-08-04 at 11:17 -0700, Andres Freund wrote: > > I'm imagining something like > > > > #define pg_printf(fmt, ...) \ > > do { \ > > if ( __builtin_constant_p(fmt)) \ > > { \ > > static processed_fmt

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Andres Freund
Hi, On 2019-08-05 13:34:39 -0400, Stephen Frost wrote: > * Andres Freund (and...@anarazel.de) wrote: > > On 2019-08-05 12:43:24 -0400, Tom Lane wrote: > > > On the other hand, people have also opined that they want full error > > > checking on the inserted values, and that seems out of reach with

Re: Unused header file inclusion

2019-08-05 Thread Tom Lane
Alvaro Herrera writes: > Then there's the removal, which is in tuplesort.c because of > INT_MAX as added by commit d26559dbf356 and still present ... One has to be especially wary of removing system-header inclusions; the fact that they don't seem to be needed on your own machine doesn't prove

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Stephen Frost
Greetings, On Mon, Aug 5, 2019 at 14:29 Tom Lane wrote: > Robert Haas writes: > > All we need to do to resolve this issue is have Tom commit his patch. > > I think Stephen is not being unreasonable to suggest that we need some > documentation about what external tools may safely do to

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-05 Thread Peter Geoghegan
On Fri, Aug 2, 2019 at 1:28 PM Julien Rouhaud wrote: > I'm fine with it! Pushed a version with similar wording just now. Thanks! -- Peter Geoghegan

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Robert Haas
On Mon, Aug 5, 2019 at 3:07 PM Tom Lane wrote: > Concretely, how about the attached? Works for me, for whatever that's worth. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Index Skip Scan

2019-08-05 Thread Dmitry Dolgov
> On Mon, Aug 5, 2019 at 12:05 PM Floris Van Nee > wrote: > > The root->distinct_pathkeys is already filtered for redundant keys, so column > 'a' is not in there anymore. Still, it'd be much faster to use the skip scan > here, because a regular scan will scan all entries with a=1, even though >

Re: Redacting information from logs

2019-08-05 Thread Jeff Davis
On Sat, 2019-08-03 at 19:14 -0400, Tom Lane wrote: > It seems to me that it'd be sufficient to do the annotation by > inserting wrapper functions, like the errparam() you suggest above. > If we just had errparam() choosing whether to return "..." instead of > its argument string, we'd have what we

Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Andres Freund
Hi, (as I was out of context due to dealing with bugs, I've switched to lookign at the current zheap/undoprocessing branch. On 2019-07-30 01:02:20 -0700, Andres Freund wrote: > +/* > + * Insert a previously-prepared undo records. > + * > + * This function will write the actual undo record into

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Tom Lane
Robert Haas writes: > All we need to do to resolve this issue is have Tom commit his patch. I think Stephen is not being unreasonable to suggest that we need some documentation about what external tools may safely do to pg.auto.conf. So somebody's got to write that. But I agree that we could go

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Robert Haas
On Mon, Aug 5, 2019 at 2:35 PM Stephen Frost wrote: > I dare say that if we had some documentation around what to expect and how to > deal with it, for our own tools as well as external ones, then maybe we > wouldn’t be in this situation in the first place. Clearly ALTER SYSTEM and > the

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-05 Thread Julien Rouhaud
On Mon, Aug 5, 2019 at 8:51 PM Peter Geoghegan wrote: > > Pushed a version with similar wording just now. Thanks!

Re: tableam vs. TOAST

2019-08-05 Thread Tom Lane
Robert Haas writes: > I don't feel entirely convinced that there's any rush to do all of > this right now, and the more I change the harder it is to make sure > that I haven't broken anything. How strongly do you feel about this > stuff? FWIW, I agree with your comment further up that this

Re: Redacting information from logs

2019-08-05 Thread Andres Freund
Hi, On 2019-08-05 13:37:50 -0700, Jeff Davis wrote: > On Sat, 2019-08-03 at 19:14 -0400, Tom Lane wrote: > > It seems to me that it'd be sufficient to do the annotation by > > inserting wrapper functions, like the errparam() you suggest above. > > If we just had errparam() choosing whether to

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Stephen Frost
Greetings, On Mon, Aug 5, 2019 at 14:38 Robert Haas wrote: > On Mon, Aug 5, 2019 at 2:29 PM Tom Lane wrote: > > Robert Haas writes: > > > All we need to do to resolve this issue is have Tom commit his patch. > > > > I think Stephen is not being unreasonable to suggest that we need some > >

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Stephen Frost
Greetings, On Mon, Aug 5, 2019 at 14:43 Tom Lane wrote: > Robert Haas writes: > > On Mon, Aug 5, 2019 at 2:29 PM Tom Lane wrote: > >> I think Stephen is not being unreasonable to suggest that we need some > >> documentation about what external tools may safely do to pg.auto.conf. > >> So

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Tom Lane
Robert Haas writes: > But that's not the problem. The problem is that ALTER SYSTEM modifies > the first match instead of the last one, when it's a well-established > principle that when reading from a PostgreSQL configuration file, the > system adopts the value from the last match, not the first

Re: Adding column "mem_usage" to view pg_prepared_statements

2019-08-05 Thread Andres Freund
Hi, On 2019-08-05 22:46:47 +0200, Daniel Migowski wrote: > > Arguably the proposed owning_object field would be a bit redundant with > > the already existing ident/MemoryContextSetIdentifier field, which > > e.g. already associates the query string with the contexts used for a > > prepared

Re: An out-of-date comment in nodeIndexonlyscan.c

2019-08-05 Thread Thomas Munro
On Fri, Jun 28, 2019 at 12:55 PM Ashwin Agrawal wrote: > On Thu, Jun 27, 2019 at 4:33 PM Thomas Munro wrote: >> I wonder if it might be possible to avoid page locks on both the heap >> *and* index in some limited cases, as I mentioned over here (just an >> idea, could be way off base): >> >>

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-05 Thread Ian Barwick
On 8/6/19 9:52 AM, Stephen Frost wrote:> Greetings, > > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> Robert Haas writes: >>> On Mon, Aug 5, 2019 at 2:42 PM Tom Lane wrote: I don't think we need to go on about it at great length, but it seems to me that it'd be reasonable to point out

Re: pg can create duplicated index without any errors even warnning

2019-08-05 Thread Peter Geoghegan
On Mon, Aug 5, 2019 at 7:34 PM Alex wrote: > is this by design? Yes. Being able to do this is useful for several reasons. For example, it's useful to be able to create a new, equivalent index before dropping the original when the original is bloated. (You could use REINDEX instead, but that has

Re: Redacting information from logs

2019-08-05 Thread Jeff Davis
On Mon, 2019-08-05 at 14:44 -0700, Andres Freund wrote: > at the cost of sometimes more complicated % syntax (i.e. %1$d to > refer > to the first argument). > > I think the probable loss of gcc format warnings would be the biggest > issue with this whole proposal, and potential translator trouble

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-05 Thread Bruce Momjian
On Wed, Jul 31, 2019 at 09:43:00AM -0400, Sehrope Sarkuni wrote: > On Wed, Jul 31, 2019 at 2:32 AM Masahiko Sawada wrote: > > Just to confirm, we have 21 bits left for nonce in CTR? We have LSN (8 > bytes), page-number (4 bytes) and counter (11 bits) in 16 bytes nonce > space. Even

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-05 Thread Bruce Momjian
On Wed, Jul 31, 2019 at 04:58:49PM +0900, Masahiko Sawada wrote: > On Wed, Jul 31, 2019 at 3:29 PM Masahiko Sawada wrote: > > > > > > For WAL encryption, before flushing WAL we encrypt whole 8k WAL page > > and then write only the encrypted data of the new WAL record using > > pg_pwrite() rather

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-05 Thread Bruce Momjian
On Wed, Jul 31, 2019 at 04:11:03PM +0900, Masahiko Sawada wrote: > On Wed, Jul 31, 2019 at 5:48 AM Bruce Momjian wrote: > > I am thinking for the heap/index IV, it would be: > > > > uint64 lsn; > > unint32 page number; > > /* only uses 11 bits for a zero-based CTR counter

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-05 Thread Bruce Momjian
On Wed, Jul 31, 2019 at 09:25:01AM -0400, Sehrope Sarkuni wrote: > On Tue, Jul 30, 2019 at 4:48 PM Bruce Momjian wrote: > > I had more time to think about the complexity of adding relfilenode to > the IV.  Since relfilenode is only unique within a database/tablespace, > we would need

  1   2   >