Re: Unused header file inclusion

2019-07-30 Thread Michael Paquier
On Wed, Jul 31, 2019 at 11:19:08AM +0530, vignesh C wrote: > I noticed that there are many header files being included which need > not be included. I have tried this in a few files and found the > compilation and regression to be working. I have attached the patch > for the files that I tried.

Re: Proposal to suppress errors thrown by to_reg*()

2019-07-30 Thread Takuma Hoshiai
On Tue, 30 Jul 2019 12:24:13 -0400 Tom Lane wrote: > Takuma Hoshiai writes: > > [ fix_to_reg_v2.patch ] > > I took a quick look through this patch. I'm on board with the goal > of not having schema-access violations throw an error in these > functions, but the implementation feels pretty ugly

Unused header file inclusion

2019-07-30 Thread vignesh C
Hi, I noticed that there are many header files being included which need not be included. I have tried this in a few files and found the compilation and regression to be working. I have attached the patch for the files that I tried. I tried this in CentOS, I did not find the header files to be

Re: [HACKERS] advanced partition matching algorithm for partition-wise join

2019-07-30 Thread Amit Langote
On Tue, Jul 30, 2019 at 6:00 PM Etsuro Fujita wrote: > On Fri, Jul 19, 2019 at 10:44 PM Robert Haas wrote: > > On Thu, Jul 18, 2019 at 2:55 AM Etsuro Fujita > > wrote: > > > I.e., partition_bounds_merge() is performed for each pair of input > > > partitioned relations for a join relation in

Re: Unused struct member in pgcrypto pgp.c

2019-07-30 Thread vignesh C
On Tue, Jul 30, 2019 at 9:19 PM Daniel Gustafsson wrote: > > Hi, > > In contrib/pgcrypto/pgp.c we have a struct member int_name in digest_info > which > isn’t used, and seems to have never been used (a potential copy/pasteo from > the > cipher_info struct?). Is there a reason for keeping this,

Re: Possible race condition in pg_mkdir_p()?

2019-07-30 Thread Michael Paquier
On Tue, Jul 30, 2019 at 10:19:45PM -0700, Andres Freund wrote: > I don't really have a problem fixing this case if we think it's > useful. But I'm a bit bothered by all the "fixes" being submitted that > don't matter for PG itself. They do eat up resources. Sure. In this particular case, we can

Re: Possible race condition in pg_mkdir_p()?

2019-07-30 Thread Andres Freund
Hi, On 2019-07-31 13:48:23 +0900, Michael Paquier wrote: > On Tue, Jul 30, 2019 at 09:11:44PM -0700, Andres Freund wrote: > > Hm. I'm not really seing much of a point in making mkdir_p safe against > > all of this. What's the scenario for pg where this matters? I assume > > you're using it for

Re: tap tests driving the database via psql

2019-07-30 Thread Craig Ringer
On Wed, 31 Jul 2019 at 10:20, Andres Freund wrote: > Hi, > > On 2019-07-31 09:32:10 +0800, Craig Ringer wrote: > > OK. So rather than building our own $everything from scratch, lets look > at > > solving that. > > IDK, a minimal driver that just does what we need it to do is a few > hundred

Re: Possible race condition in pg_mkdir_p()?

2019-07-30 Thread Ning Yu
On Wed, Jul 31, 2019 at 12:41 PM Michael Paquier wrote: > > On Wed, Jul 31, 2019 at 12:26:30PM +0800, Ning Yu wrote: > > Could I double confirm with you that you made a clean rebuild after > > applying the patches? pg_mkdir_p() is compiled as part of libpgport.a, > > and the postgres makefile

Re: Possible race condition in pg_mkdir_p()?

2019-07-30 Thread Michael Paquier
On Tue, Jul 30, 2019 at 09:11:44PM -0700, Andres Freund wrote: > Hm. I'm not really seing much of a point in making mkdir_p safe against > all of this. What's the scenario for pg where this matters? I assume > you're using it for somewhat different purposes, and that's why it is > problematic for

Re: POC: Cleaning up orphaned files using undo logs

2019-07-30 Thread Amit Kapila
On Tue, Jul 30, 2019 at 5:26 PM Thomas Munro wrote: > > Hi Amit > > I've been testing some undo worker workloads (more on that soon), > One small point, there is one small bug in the error queues which is that the element pushed into error queue doesn't have an updated value of to_urec_ptr which

Re: Possible race condition in pg_mkdir_p()?

2019-07-30 Thread Michael Paquier
On Wed, Jul 31, 2019 at 12:26:30PM +0800, Ning Yu wrote: > Could I double confirm with you that you made a clean rebuild after > applying the patches? pg_mkdir_p() is compiled as part of libpgport.a, > and the postgres makefile will not relink the initdb binary > automatically, for myself I must

Re: Possible race condition in pg_mkdir_p()?

2019-07-30 Thread Ning Yu
On Wed, Jul 31, 2019 at 12:11 PM Andres Freund wrote: > > Hi, > > On 2019-07-18 16:17:22 +0800, Ning Yu wrote: > > This seems buggy as it first checks the existence of the dir and makes the > > dir if it does not exist yet, however when executing concurrently a > > possible race condition can be

Re: SegFault on 9.6.14

2019-07-30 Thread Tom Lane
Amit Kapila writes: > On Wed, Jul 31, 2019 at 12:05 AM Robert Haas wrote: >> The other option is to do >> what I understand Amit and Thomas to be proposing, which is to do a >> better job identifying the case where we're "done for good" and can >> trigger the shutdown fearlessly. > Yes, this

Re: Possible race condition in pg_mkdir_p()?

2019-07-30 Thread Ning Yu
On Wed, Jul 31, 2019 at 12:04 PM Michael Paquier wrote: > > On Tue, Jul 30, 2019 at 06:22:59PM +0800, Ning Yu wrote: > > In fact personally I'm thinking that whether could we replace all uses of > > MakePGDirectory() with pg_mkdir_p(), so we could simplify > > TablespaceCreateDbspace() and

Re: tap tests driving the database via psql

2019-07-30 Thread Michael Paquier
On Tue, Jul 30, 2019 at 07:20:53PM -0700, Andres Freund wrote: > IDK, a minimal driver that just does what we need it to do is a few > hundred lines, not more. And there's plenty of stuff that we simply > won't be able to test with any driver that's not purposefully written > for testing. There's

Re: POC: Cleaning up orphaned files using undo logs

2019-07-30 Thread Amit Kapila
On Tue, Jul 30, 2019 at 1:32 PM Andres Freund wrote: > > Hi, > > Amit, short note: The patches aren't attached in patch order. Obviously > a miniscule thing, but still nicer if that's not the case. > Noted, I will try to ensure that patches are in order in future posts. -- With Regards, Amit

Re: Possible race condition in pg_mkdir_p()?

2019-07-30 Thread Andres Freund
Hi, On 2019-07-18 16:17:22 +0800, Ning Yu wrote: > This seems buggy as it first checks the existence of the dir and makes the > dir if it does not exist yet, however when executing concurrently a > possible race condition can be as below: > > A: does a/ exists? no > B: does a/ exists? no > A:

Re: SegFault on 9.6.14

2019-07-30 Thread Amit Kapila
On Wed, Jul 31, 2019 at 12:05 AM Robert Haas wrote: > > On Thu, Jul 18, 2019 at 9:45 AM Tom Lane wrote: > > I think this is going in the wrong direction. Nodes should *always* > > assume that a rescan is possible until ExecEndNode is called. > > If you want to do otherwise, you are going to be

Re: Possible race condition in pg_mkdir_p()?

2019-07-30 Thread Michael Paquier
On Tue, Jul 30, 2019 at 06:22:59PM +0800, Ning Yu wrote: > In fact personally I'm thinking that whether could we replace all uses of > MakePGDirectory() with pg_mkdir_p(), so we could simplify > TablespaceCreateDbspace() and PathNameCreateTemporaryDir() and other callers > significantly. I would

How to retain lesser paths at add_path()?

2019-07-30 Thread Kohei KaiGai
Hello, When we add a new path using add_path(), it checks estimated cost and path-keys, then it also removes dominated paths, if any. Do we have a reasonable way to retain these "dominated" paths? Once it is considered lesser paths at a level, however, it may have a combined cheaper cost with

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-07-30 Thread Peter Geoghegan
On Tue, Jul 30, 2019 at 8:07 PM Melanie Plageman wrote: > For the actual write to disk, I'm pretty sure I get that for free from > the BufFile API, no? > I was more thinking about optimizing when I call BufFileWrite at all. Right. Clearly several existing buffile.c users regularly have very

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-07-30 Thread Melanie Plageman
On Tue, Jul 30, 2019 at 4:36 PM Robert Haas wrote: > On Tue, Jul 30, 2019 at 2:47 PM Melanie Plageman > wrote: > > I did the "needlessly dumb implementation" Robert mentioned, though, > > I thought about it and couldn't come up with a much smarter way to > > write match bits to a file. I think

Re: Runtime pruning problem

2019-07-30 Thread Amit Langote
On Wed, Jul 31, 2019 at 8:31 AM Tom Lane wrote: > David Rowley writes: > > On Wed, 31 Jul 2019 at 10:56, Tom Lane wrote: > >> The portion of this below the Append is fine, but I argue that > >> the Vars above the Append should say "part", not "part_p1". > >> In that way they'd look the same

Re: tap tests driving the database via psql

2019-07-30 Thread Andres Freund
Hi, On 2019-07-31 09:32:10 +0800, Craig Ringer wrote: > OK. So rather than building our own $everything from scratch, lets look at > solving that. IDK, a minimal driver that just does what we need it to do is a few hundred lines, not more. And there's plenty of stuff that we simply won't be able

Re: Initdb failure

2019-07-30 Thread Michael Paquier
On Tue, Jul 30, 2019 at 03:30:12PM -0400, Tom Lane wrote: > On the whole though, I don't have a problem with the "do nothing" > answer. There's no security risk here, and no issue that seems > likely to arise in actual use cases rather than try-to-break-it > test scripts. +1. -- Michael

Re: idea: log_statement_sample_rate - bottom limit for sampling

2019-07-30 Thread Michael Paquier
On Tue, Jul 30, 2019 at 03:43:58PM -0400, Tom Lane wrote: > Well, we do not need to have a backwards-compatibility problem > here, because we have yet to release a version containing > log_statement_sample_rate. I do not think it's too late to decide > that v12's semantics for that are broken,

Re: extension patch of CREATE OR REPLACE TRIGGER

2019-07-30 Thread Michael Paquier
On Tue, Jul 30, 2019 at 04:44:11PM -0400, Tom Lane wrote: > We do not test \h output in any existing regression test, and we're > not going to start doing so in this one. For one thing, the expected > URL would break every time we forked off a new release branch. > (There would surely be value in

Re: tap tests driving the database via psql

2019-07-30 Thread Craig Ringer
On Tue, 30 Jul 2019 at 21:40, Tom Lane wrote: > Craig Ringer writes: > > On Sun, 28 Jul 2019 at 03:15, Andres Freund wrote: > >> 1) Just depend on DBD::Pg being installed. It's fairly common, after > >> all. It'd be somewhat annoying that we'd often end up using a > >> different version of

Re: Unused struct member in pgcrypto pgp.c

2019-07-30 Thread Michael Paquier
On Tue, Jul 30, 2019 at 05:48:49PM +0200, Daniel Gustafsson wrote: > In contrib/pgcrypto/pgp.c we have a struct member int_name in digest_info > which > isn’t used, and seems to have never been used (a potential copy/pasteo from > the > cipher_info struct?). Is there a reason for keeping this,

Re: concerns around pg_lsn

2019-07-30 Thread Michael Paquier
On Tue, Jul 30, 2019 at 02:22:30PM +0530, Jeevan Ladhe wrote: > On Tue, Jul 30, 2019 at 9:42 AM Michael Paquier wrote: >> Agreed about making the code more defensive as you do. I would keep >> the initialization in check_recovery_target_lsn and pg_lsn_in_internal >> though. That does not hurt

Re: Attached partition not considering altered column properties of root partition.

2019-07-30 Thread Amit Langote
On Wed, Jul 31, 2019 at 2:38 AM Robert Haas wrote: > > On Tue, Jul 2, 2019 at 9:53 PM Amit Langote wrote: > > Thanks for the report. This seems like a bug. Documentation claims > > that the child tables inherit column storage options from the parent > > table. That's actually enforced in only

Re: Adding column "mem_usage" to view pg_prepared_statements

2019-07-30 Thread David Fetter
On Tue, Jul 30, 2019 at 10:01:09PM +, Daniel Migowski wrote: > Hello, > > Will my patch be considered for 12.0? The calculation of the > mem_usage value might be improved later on but because the system > catalog is changed I would love to add it before 12.0 becomes > stable. Feature freeze

Re: pgbench - implement strict TPC-B benchmark

2019-07-30 Thread Tom Lane
Peter Geoghegan writes: > On Tue, Jul 30, 2019 at 3:00 PM Tom Lane wrote: >> I'm also highly dubious about labeling this script "standard TPC-B", >> when it resolves only some of the reasons why our traditional script >> is not really TPC-B. That's treading on being false advertising. > IANAL,

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-07-30 Thread Robert Haas
On Tue, Jul 30, 2019 at 2:47 PM Melanie Plageman wrote: > I did the "needlessly dumb implementation" Robert mentioned, though, > I thought about it and couldn't come up with a much smarter way to > write match bits to a file. I think there might be an optimization > opportunity in not writing the

Re: pgbench - implement strict TPC-B benchmark

2019-07-30 Thread Peter Geoghegan
On Tue, Jul 30, 2019 at 3:00 PM Tom Lane wrote: > TBH, I think we should reject this patch. Nobody cares about TPC-B > anymore, and they care even less about differences between one > sort-of-TPC-B test and another sort-of-TPC-B test. (As the lack > of response on this thread shows.) We don't

Re: Runtime pruning problem

2019-07-30 Thread Tom Lane
David Rowley writes: > On Wed, 31 Jul 2019 at 10:56, Tom Lane wrote: >> The portion of this below the Append is fine, but I argue that >> the Vars above the Append should say "part", not "part_p1". >> In that way they'd look the same regardless of which partitions >> have been pruned or not. >

Re: Runtime pruning problem

2019-07-30 Thread David Rowley
On Wed, 31 Jul 2019 at 10:56, Tom Lane wrote: > > OK, so experimenting, I see that it is a change: HEAD does > > regression=# explain verbose select * from part order by a; >QUERY PLAN >

Re: Runtime pruning problem

2019-07-30 Thread Tom Lane
I wrote: > This may be arguing for a change in ruleutils' existing behavior, > not sure. But when dealing with traditional-style inheritance, > I've always thought that Vars above the Append were referring to > the parent rel in its capacity as the parent, not in its capacity > as the first

Re: Runtime pruning problem

2019-07-30 Thread Tom Lane
David Rowley writes: > On Wed, 31 Jul 2019 at 10:27, Tom Lane wrote: >> What I had in mind was to revert 1cc29fe7c's ruleutils changes >> entirely, so that ruleutils deals only in Plans not PlanStates. >> Perhaps we've grown some code since then that really needs the >> PlanStates, but what is

Re: Adding column "mem_usage" to view pg_prepared_statements

2019-07-30 Thread Daniel Migowski
Am 31.07.2019 um 00:29 schrieb Tom Lane: Daniel Migowski writes: Ok, just have read about the commitfest thing. Is the patch OK for that? Or is there generally no love for a mem_usage column here? If it was, I really would add some memory monitoring in our app regarding this. You should

Re: Runtime pruning problem

2019-07-30 Thread David Rowley
On Wed, 31 Jul 2019 at 10:27, Tom Lane wrote: > > David Rowley writes: > > The part I wouldn't mind another set of eyes on is the ruleutils.c > > changes. > > Um, sorry for not getting to this sooner. > > What I had in mind was to revert 1cc29fe7c's ruleutils changes > entirely, so that

Re: Adding column "mem_usage" to view pg_prepared_statements

2019-07-30 Thread Daniel Migowski
Am 31.07.2019 um 00:17 schrieb Tomas Vondra: FWIW not sure what mail client you're using, but it seems to be breaking the threads for some reason, splitting it into two - see [1]. Also, please stop top-posting. It makes it way harder to follow the discussion. Was using Outlook because it's my

Re: AW: AW: Adding column "mem_usage" to view pg_prepared_statements

2019-07-30 Thread Tom Lane
Daniel Migowski writes: > Ok, just have read about the commitfest thing. Is the patch OK for that? Or > is there generally no love for a mem_usage column here? If it was, I really > would add some memory monitoring in our app regarding this. You should certainly put it into the next

Re: Runtime pruning problem

2019-07-30 Thread Tom Lane
David Rowley writes: > The part I wouldn't mind another set of eyes on is the ruleutils.c > changes. Um, sorry for not getting to this sooner. What I had in mind was to revert 1cc29fe7c's ruleutils changes entirely, so that ruleutils deals only in Plans not PlanStates. Perhaps we've grown some

Re: Adding column "mem_usage" to view pg_prepared_statements

2019-07-30 Thread Tomas Vondra
On Tue, Jul 30, 2019 at 10:01:09PM +, Daniel Migowski wrote: Hello, Will my patch be considered for 12.0? The calculation of the mem_usage value might be improved later on but because the system catalog is changed I would love to add it before 12.0 becomes stable. Nope. Code freeze for

AW: AW: Adding column "mem_usage" to view pg_prepared_statements

2019-07-30 Thread Daniel Migowski
Ok, just have read about the commitfest thing. Is the patch OK for that? Or is there generally no love for a mem_usage column here? If it was, I really would add some memory monitoring in our app regarding this. -Ursprüngliche Nachricht- Von: Tom Lane Gesendet: Mittwoch, 31. Juli 2019

Re: AW: Adding column "mem_usage" to view pg_prepared_statements

2019-07-30 Thread Tom Lane
Daniel Migowski writes: > Will my patch be considered for 12.0? The calculation of the mem_usage value > might be improved later on but because the system catalog is changed I would > love to add it before 12.0 becomes stable. v12 has been feature-frozen for months, and it's pretty hard to

AW: Adding column "mem_usage" to view pg_prepared_statements

2019-07-30 Thread Daniel Migowski
Hello, Will my patch be considered for 12.0? The calculation of the mem_usage value might be improved later on but because the system catalog is changed I would love to add it before 12.0 becomes stable. Regards, Daniel Migowski -Ursprüngliche Nachricht- Von: Daniel Migowski

Re: pgbench - implement strict TPC-B benchmark

2019-07-30 Thread Tom Lane
Fabien COELHO writes: > [ pgbench-strict-tpcb-2.patch ] TBH, I think we should reject this patch. Nobody cares about TPC-B anymore, and they care even less about differences between one sort-of-TPC-B test and another sort-of-TPC-B test. (As the lack of response on this thread shows.) We don't

Re: Remove HeapTuple and Buffer dependency for predicate locking functions

2019-07-30 Thread Thomas Munro
On Tue, Jun 25, 2019 at 6:02 AM Andres Freund wrote: > On 2019-06-24 10:41:06 -0700, Ashwin Agrawal wrote: > > Proposing following changes to make predicate locking and checking > > functions generic and remove dependency on HeapTuple and Heap AM. We > > made these changes to help with Zedstore.

Re: [Patch] Adding CORRESPONDING/CORRESPONDING BY to set operation

2019-07-30 Thread David Fetter
On Tue, Jul 30, 2019 at 02:43:05PM -0700, 毛瑞嘉 wrote: > Hi, > > > I wrote a patch for adding CORRESPONDING/CORRESPONDING BY to set operation. > It is a task in the todo list. This is how the patch works: > > I modified transformSetOperationStmt() to get an intersection target list > which is the

Re: Patch to document base64 encoding

2019-07-30 Thread Fabien COELHO
My 0.02 € It seems entirely crazy that encode() and decode() are no longer in the same section, likewise that convert_from() and convert_to() aren't documented together anymore. Awkward, yes. But findable if you know what the categories are. I suppose there could be 3 different

[Patch] Adding CORRESPONDING/CORRESPONDING BY to set operation

2019-07-30 Thread 毛瑞嘉
Hi, I wrote a patch for adding CORRESPONDING/CORRESPONDING BY to set operation. It is a task in the todo list. This is how the patch works: I modified transformSetOperationStmt() to get an intersection target list which is the intersection of the target lists of the left clause and right

Re: idea: log_statement_sample_rate - bottom limit for sampling

2019-07-30 Thread Tomas Vondra
On Tue, Jul 30, 2019 at 03:43:58PM -0400, Tom Lane wrote: Tomas Vondra writes: I've started reviewing this patch, thinking that maybe I could get it committed as it's marked as RFC. In general I agree with having this fuature, but I think we need to rethink the GUC because the current approach

Re: heapam_index_build_range_scan's anyvisible

2019-07-30 Thread Ashwin Agrawal
On Tue, Jul 16, 2019 at 10:22 AM Andres Freund wrote: > Hi, > > On 2019-07-11 17:27:46 -0700, Ashwin Agrawal wrote: > > Please find attached the patch to remove IndexBuildCallback's dependency > on > > HeapTuple, as discussed. Changed to have the argument as ItemPointer > > instead of HeapTuple.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-30 Thread Bruce Momjian
On Tue, Jul 30, 2019 at 10:14:14AM -0400, Sehrope Sarkuni wrote: > > In general it's fine to use the same IV with different keys. Only reuse > of Key > > + IV is a problem and the entire set of possible counter values (IV + 0, > IV + > > 1, ...) generated with a key must be

Re: extension patch of CREATE OR REPLACE TRIGGER

2019-07-30 Thread Tom Lane
"osumi.takami...@fujitsu.com" writes: > [ CREATE_OR_REPLACE_TRIGGER_v03.patch ] I took a quick look through this just to see what was going on. A few comments: * Upthread you asked about changing the lock level to be AccessExclusiveLock if the trigger already exists, but the patch doesn't

Re: idea: log_statement_sample_rate - bottom limit for sampling

2019-07-30 Thread Tom Lane
Tomas Vondra writes: > I've started reviewing this patch, thinking that maybe I could get it > committed as it's marked as RFC. In general I agree with having this > fuature, but I think we need to rethink the GUC because the current > approach is just confusing. > ... > What I think we should do

Re: Initdb failure

2019-07-30 Thread Tom Lane
Robert Haas writes: > Agreed, but I think we should just do nothing. To actually fix this > in general, we'd have to get rid of every instance of MAXPGPATH in the > source tree: > [rhaas pgsql]$ git grep MAXPGPATH | wc -l > 611 I don't think it'd really be necessary to go that far. One of

Re: Allow table AM's to cache stuff in relcache

2019-07-30 Thread Heikki Linnakangas
On 12/07/2019 16:07, Julien Rouhaud wrote: On Fri, Jun 14, 2019 at 5:40 PM Tom Lane wrote: Heikki Linnakangas writes: In the patch, I documented that rd_amcache must be allocated in CacheMemoryContext, or in rd_indexcxt if it's an index. It works, but it's a bit weird. Given the way the

Redacting information from logs

2019-07-30 Thread Jeff Davis
Logs are important to diagnose problems or monitor operations, but logs can contain sensitive information which is often unnecessary for these purposes. Redacting the sensitive information would enable easier access and simpler integration with analysis tools without compromising the sensitive

Re: Initdb failure

2019-07-30 Thread Robert Haas
On Sat, Jul 27, 2019 at 2:22 AM Peter Eisentraut wrote: > I think if you want to make this more robust, get rid of the fixed-size > array, use dynamic allocation with PQExpBuffer, and let the operating > system complain if it doesn't like the directory name length. Agreed, but I think we should

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-07-30 Thread Melanie Plageman
So, I've rewritten the patch to use a BufFile for the outer table batch file tuples' match statuses and write bytes to and from the file which start as 0 and, upon encountering a match for a tuple, I set its bit in the file to 1 (also rebased with current master). It, of course, only works for

Re: tap tests driving the database via psql

2019-07-30 Thread Alvaro Herrera
On 2019-Jul-30, Tom Lane wrote: > OK, so just lifting DBD::Pg in toto is out for license reasons. > However, maybe we could consider writing a new DBD driver from > scratch (while using a platform-provided DBI layer) rather than > doing everything from scratch. I'm not sure how much actual >

Re: SegFault on 9.6.14

2019-07-30 Thread Robert Haas
On Thu, Jul 18, 2019 at 9:45 AM Tom Lane wrote: > I think this is going in the wrong direction. Nodes should *always* > assume that a rescan is possible until ExecEndNode is called. > If you want to do otherwise, you are going to be inventing a whole > bunch of complicated and

Re: Patch to document base64 encoding

2019-07-30 Thread Karl O. Pinc
On Tue, 30 Jul 2019 11:40:03 -0400 Tom Lane wrote: > "Karl O. Pinc" writes: > > On Mon, 15 Jul 2019 23:00:55 +0200 (CEST) > > Fabien COELHO wrote: > >> The patch clarifies the documentation about encode/decode and > >> other text/binary string conversion functions. > > > Other notable

typesafe printf hackery

2019-07-30 Thread Andres Freund
Hi, I've occasionally wished for a typesafe version of pg_printf() and other varargs functions. The compiler warnings are nice, but also far from complete. Here's a somewhat crazy hack/prototype for how printf could get actual argument types. I'm far from certain it's worth pursuing this

Re: ANALYZE: ERROR: tuple already updated by self

2019-07-30 Thread Tomas Vondra
On Mon, Jul 29, 2019 at 12:18:33PM +0200, Tomas Vondra wrote: On Sun, Jul 28, 2019 at 09:53:20PM -0700, Andres Freund wrote: Hi, On 2019-07-28 21:21:51 +0200, Tomas Vondra wrote: AFAICS it applies to 10+ versions, because that's where extended stats were introduced. We certainly can't mess

Re: TopoSort() fix

2019-07-30 Thread Tom Lane
Robert Haas writes: > On Tue, Jul 30, 2019 at 1:44 PM Tom Lane wrote: >> Well, there'd be an actual isolation test that they work ;-), if you >> override the marking. Admittedly, one test case does not prove that >> there's no way to crash the system, but that can be said of most >> parts of

Re: TopoSort() fix

2019-07-30 Thread Robert Haas
On Tue, Jul 30, 2019 at 1:44 PM Tom Lane wrote: > Well, there'd be an actual isolation test that they work ;-), if you > override the marking. Admittedly, one test case does not prove that > there's no way to crash the system, but that can be said of most > parts of Postgres. True. I'm just

Re: TopoSort() fix

2019-07-30 Thread Tom Lane
Robert Haas writes: > On Tue, Jul 30, 2019 at 1:36 PM Tom Lane wrote: >> In any case, my question at the moment is whether we need the belt-and- >> suspenders-too approach of having both non-parallel-safe marking and an >> explicit check inside these functions. We've largely moved away from >>

Re: TopoSort() fix

2019-07-30 Thread Robert Haas
On Tue, Jul 30, 2019 at 1:36 PM Tom Lane wrote: > No, there's a sufficient reason why we should force advisory locks > to be taken in the leader process, namely that the behavior is totally > different if we don't: they will disappear at the end of the parallel > worker run, not at end of

Re: Attached partition not considering altered column properties of root partition.

2019-07-30 Thread Robert Haas
On Tue, Jul 2, 2019 at 9:53 PM Amit Langote wrote: > Thanks for the report. This seems like a bug. Documentation claims > that the child tables inherit column storage options from the parent > table. That's actually enforced in only some cases. I realize I'm just repeating the same argument

Re: TopoSort() fix

2019-07-30 Thread Tom Lane
Robert Haas writes: > On Tue, Jul 30, 2019 at 10:27 AM Tom Lane wrote: >> (BTW, why aren't these functions just "parallel restricted"?) > ... > But it is really pretty arguable whether we should feel responsible > for that problem. We could just decide that if you're doing that, and > you

Re: Avoid full GIN index scan when possible

2019-07-30 Thread Tom Lane
Marc Cousin writes: > By the way, while preparing this, I noticed that it seems that during this > kind of index scan, the interrupt signal is masked > for a very long time. Control-C takes a very long while to cancel the query. > But it's an entirely different problem :) Yeah, that seems like

Re: make installcheck-world in a clean environment

2019-07-30 Thread Tom Lane
Alexander Lakhin writes: > 01.07.2019 13:47, Thomas Munro wrote: >> A new CF is here and this is in "Needs Review". Would you like to >> provide a rebased patch, or should it really be withdrawn? > The rebased patch is attached, but I still can't find anyone interested > in reviewing it. > So

Re: TopoSort() fix

2019-07-30 Thread Robert Haas
On Tue, Jul 30, 2019 at 10:27 AM Tom Lane wrote: > I also looked into whether one could use SELECT FOR UPDATE/SHARE to get > stronger locks at a tuple level, but that's been blocked off as well. > You guys really did a pretty good job of locking that down. Thanks. We learned from the master. >

Re: Proposal to suppress errors thrown by to_reg*()

2019-07-30 Thread Tom Lane
Takuma Hoshiai writes: > [ fix_to_reg_v2.patch ] I took a quick look through this patch. I'm on board with the goal of not having schema-access violations throw an error in these functions, but the implementation feels pretty ugly and bolted-on. Nobody who had designed the code to do this from

Unused struct member in pgcrypto pgp.c

2019-07-30 Thread Daniel Gustafsson
Hi, In contrib/pgcrypto/pgp.c we have a struct member int_name in digest_info which isn’t used, and seems to have never been used (a potential copy/pasteo from the cipher_info struct?). Is there a reason for keeping this, or can it be removed as per the attached? cheers ./daniel

Re: Patch to document base64 encoding

2019-07-30 Thread Tom Lane
"Karl O. Pinc" writes: > On Mon, 15 Jul 2019 23:00:55 +0200 (CEST) > Fabien COELHO wrote: >> The patch clarifies the documentation about encode/decode and other >> text/binary string conversion functions. > Other notable changes: > Corrects categorization of functions as string or binary. >

Re: pg_upgrade version checking questions

2019-07-30 Thread Daniel Gustafsson
> On 27 Jul 2019, at 08:42, Peter Eisentraut > wrote: > I have committed 0002, 0003, and 0004. Thanks! > The implementation in 0001 (Only allow upgrades by the same exact > version new bindir) has a problem. It compares (new_cluster.bin_version > != PG_VERSION_NUM), but

Re: Parallel grouping sets

2019-07-30 Thread Tomas Vondra
On Tue, Jul 30, 2019 at 03:50:32PM +0800, Richard Guo wrote: On Wed, Jun 12, 2019 at 10:58 AM Richard Guo wrote: Hi all, Paul and I have been hacking recently to implement parallel grouping sets, and here we have two implementations. Implementation 1 Attached is the patch

Re: tap tests driving the database via psql

2019-07-30 Thread Tom Lane
Andres Freund writes: > On 2019-07-30 09:40:54 -0400, Tom Lane wrote: >> Now, none of these things are really a problem with DBD/DBI as such >> --- rather, they are reasons not to depend on a pre-packaged build >> of DBD::Pg that depends on a pre-packaged build of libpq.so. >> I haven't looked at

Re: tap tests driving the database via psql

2019-07-30 Thread Andres Freund
Hi, On 2019-07-30 09:40:54 -0400, Tom Lane wrote: > Craig Ringer writes: > > On Sun, 28 Jul 2019 at 03:15, Andres Freund wrote: > >> 1) Just depend on DBD::Pg being installed. It's fairly common, after > >> all. It'd be somewhat annoying that we'd often end up using a > >> different version of

Re: TopoSort() fix

2019-07-30 Thread Tom Lane
Robert Haas writes: > On Mon, Jul 29, 2019 at 9:48 PM Tom Lane wrote: >> I believe the only accessible route to taking any sort of new lock >> in a parallel worker is catalog lookups causing AccessShareLock on >> a catalog. > Can't the worker just query a previously-untouched table, maybe by >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-30 Thread Sehrope Sarkuni
On Tue, Jul 30, 2019 at 10:06 AM Masahiko Sawada wrote: > On Tue, Jul 30, 2019 at 5:03 AM Sehrope Sarkuni > wrote: > > > > On Mon, Jul 29, 2019 at 9:44 AM Bruce Momjian wrote: > >> > >> > Checking that all buffers using a single LSN are from the same > >> > relation would be a good idea but I

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-30 Thread Sehrope Sarkuni
On Tue, Jul 30, 2019 at 8:16 AM Bruce Momjian wrote: > On Tue, Jul 30, 2019 at 07:44:20AM -0400, Sehrope Sarkuni wrote: > > If each relation file has its own derived key, the derived TDEK for that > > relation file, then there is no issue with reusing the same IV = LSN || > Page > > Number. The

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-30 Thread Masahiko Sawada
On Tue, Jul 30, 2019 at 5:03 AM Sehrope Sarkuni wrote: > > On Mon, Jul 29, 2019 at 9:44 AM Bruce Momjian wrote: >> >> > Checking that all buffers using a single LSN are from the same >> > relation would be a good idea but I think it's hard to test it and >> > regard the test result as okay. Even

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-30 Thread Sehrope Sarkuni
On Tue, Jul 30, 2019 at 8:16 AM Masahiko Sawada wrote: > On Mon, Jul 29, 2019 at 8:18 PM Sehrope Sarkuni > wrote: > > > > On Mon, Jul 29, 2019 at 6:42 AM Masahiko Sawada > wrote: > > > > An argument could be made to push that problem upstream, i.e. let the > > > > supplier of the passphrase

Re: tap tests driving the database via psql

2019-07-30 Thread Tom Lane
Craig Ringer writes: > On Sun, 28 Jul 2019 at 03:15, Andres Freund wrote: >> 1) Just depend on DBD::Pg being installed. It's fairly common, after >> all. It'd be somewhat annoying that we'd often end up using a >> different version of libpq than what we're testing against. But in >> most cases

Re: block-level incremental backup

2019-07-30 Thread Ibrar Ahmed
On Tue, Jul 30, 2019 at 1:28 AM Robert Haas wrote: > On Wed, Jul 10, 2019 at 2:17 PM Anastasia Lubennikova > wrote: > > In attachments, you can find a prototype of incremental pg_basebackup, > > which consists of 2 features: > > > > 1) To perform incremental backup one should call pg_basebackup

Re: [proposal] de-TOAST'ing using a iterator

2019-07-30 Thread Binguo Bao
John Naylor 于2019年7月29日周一 上午11:49写道: > On Thu, Jul 25, 2019 at 10:21 PM Binguo Bao wrote: > My goal for this stage of review was to understand more fully what the > code is doing, and make it as simple and clear as possible, starting > at the top level. In doing so, it looks like I found some

Re: Built-in connection pooler

2019-07-30 Thread Tomas Vondra
On Tue, Jul 30, 2019 at 01:01:48PM +0300, Konstantin Knizhnik wrote: On 30.07.2019 4:02, Tomas Vondra wrote: My idea (sorry if it wasn't too clear) was that we might handle some cases more gracefully. For example, if we only switch between transactions, we don't quite care about 'SET LOCAL'

Re: SQL:2011 PERIODS vs Postgres Ranges?

2019-07-30 Thread Ibrar Ahmed
Hi Paul, I have rebased the patch to master (1e2fddfa33d3c7cc93ca3ee0f32852699bd3e012) and fixed some compilation warning. Now I am reviewing the actual code. On Fri, Jul 26, 2019 at 6:35 PM Ibrar Ahmed wrote: > The patch requires to rebase on the master branch. > > The new status of this

Re: concerns around pg_lsn

2019-07-30 Thread Robert Haas
On Tue, Jul 30, 2019 at 4:52 AM Jeevan Ladhe wrote: > My only concern was something that we internally treat as invalid, why do > we allow, that as a valid value for that type. While I am not trying to > reinvent the wheel here, I am trying to understand if there had been any > idea behind this

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-07-30 Thread Kyotaro Horiguchi
At Tue, 30 Jul 2019 21:30:45 +0900 (Tokyo Standard Time), Kyotaro Horiguchi wrote in <20190730.213045.221405075.horikyota@gmail.com> > I attach the revised patch. I'll repost the polished version > sooner. (Mainly TAP test and documentation, code comments will be revised) regards. --

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-07-30 Thread Kyotaro Horiguchi
Thanks for reviewing! At Thu, 27 Jun 2019 16:22:56 +0200, Jehan-Guillaume de Rorthais wrote in <20190627162256.4f4872b8@firost> > Hi all, > > Being interested by this feature, I did a patch review. > > This features adds the GUC "max_slot_wal_keep_size". This is the maximum > amount > of WAL

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-30 Thread Bruce Momjian
On Tue, Jul 30, 2019 at 07:44:20AM -0400, Sehrope Sarkuni wrote: > On Mon, Jul 29, 2019 at 8:35 PM Bruce Momjian wrote: > From the patch: > > /* > ! * The initialization vector (IV) is used for page-level > ! * encryption.  We use the LSN and page number as the IV, and IV > ! * values must never

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-30 Thread Masahiko Sawada
On Mon, Jul 29, 2019 at 8:18 PM Sehrope Sarkuni wrote: > > On Mon, Jul 29, 2019 at 6:42 AM Masahiko Sawada wrote: > > > An argument could be made to push that problem upstream, i.e. let the > > > supplier of the passphrase deal with the indirection. You would still > > > need to verify the

  1   2   >