Re: POC: Cleaning up orphaned files using undo logs

2019-08-14 Thread Andres Freund
Hi, On 2019-08-06 14:18:42 -0700, Andres Freund wrote: > Here's the last section of my low-leve review. Plan to write a higher > level summary afterwards, now that I have a better picture of the code. General comments: - For a feature of this complexity, there's very little architectural

Re: use of the term "verifier" with SCRAM

2019-08-14 Thread Heikki Linnakangas
On 14/08/2019 08:59, Peter Eisentraut wrote: I'm confused by how the code uses the term "verifier" in relation to SCRAM. ISTM that the code uses the term as meaning whatever is or would be stored in pg_auth.rolpassword. I don't see this usage supported in the RFCs. In RFC 5802, verifier

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-08-14 Thread David Rowley
On Thu, 25 Jul 2019 at 05:49, Tom Lane wrote: > On the whole, I don't especially like this approach, because of the > confusion between peak lock count and end-of-xact lock count. That > seems way too likely to cause problems. Thanks for having a look at this. I've not addressed the points

Re: Custom table AMs need to include heapam.h because of BulkInsertState

2019-08-14 Thread David Rowley
On Wed, 17 Jul 2019 at 06:46, Andres Freund wrote: > 2) Add a execPartition.c function that returns all the used tables from >a PartitionTupleRouting*. Here's a patch which implements it that way. I struggled a bit to think of a good name for the execPartition.c function. I ended up with

Re: progress report for ANALYZE

2019-08-14 Thread Etsuro Fujita
Hi, On Tue, Aug 13, 2019 at 11:01 PM Alvaro Herrera wrote: > On the subject of FDW support: I did look into supporting that before > submitting this. I think it's not academically difficult: just have the > FDW's acquire_sample_rows callback invoke the update_param functions > once in a while.

use of the term "verifier" with SCRAM

2019-08-14 Thread Peter Eisentraut
I'm confused by how the code uses the term "verifier" in relation to SCRAM. ISTM that the code uses the term as meaning whatever is or would be stored in pg_auth.rolpassword. I don't see this usage supported in the RFCs. In RFC 5802, verifier= "v=" base64 ;;

Re: accounting for memory used for BufFile during hash joins

2019-08-14 Thread Hubert Zhang
On Fri, Jul 12, 2019 at 1:16 AM Robert Haas wrote: > On Mon, May 6, 2019 at 9:49 PM Thomas Munro > wrote: > > Stepping back a bit, I think there is something fishy about the way we > > detect extreme skew. Is that a factor in this case? Right now we > > wait until we have a batch that gets

Don't like getObjectDescription results for pg_amop/pg_amproc

2019-08-14 Thread Tom Lane
I notice that for a pg_amop object, getObjectDescription does this: /*-- translator: %d is the operator strategy (a number), the first two %s's are data type names, the third %s is the description of the operator family,

Re: Add "password_protocol" connection parameter to libpq

2019-08-14 Thread Jeff Davis
On Wed, 2019-08-14 at 11:38 +0900, Michael Paquier wrote: > What I got in mind was a comma-separated list of authorized protocols > which can be specified as a connection parameter, which extends to > all > the types of AUTH_REQ requests that libpq can understand, plus an > extra for channel

Re: Don't like getObjectDescription results for pg_amop/pg_amproc

2019-08-14 Thread Alexander Korotkov
On Thu, Aug 15, 2019 at 2:08 AM Tom Lane wrote: > Or maybe we're just being too ambitious here and we should discard some of > this information. I'm not really sure that the format_operator result > can be included without complete loss of intelligibility. > > Thoughts? I'm particularly unclear

Re: Change ereport level for QueuePartitionConstraintValidation

2019-08-14 Thread Alvaro Herrera
On 2019-Jul-23, David Rowley wrote: > I don't know my way around the tap tests that well, but I started to > look at this and ended up a bit stuck on where the test should be > located. I see src/test/modules/brin has some brin related tests, so > I thought that src/test/modules/alter_table

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-14 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> Perhaps we could put some of these details into the Notes section of the > >> ALTER SYSTEM ref page. But I wonder how much of this is needed at all. > > > I'd be alright

Re: progress report for ANALYZE

2019-08-14 Thread Tatsuro Yamada
Hi Alvaro, On 2019/08/13 23:01, Alvaro Herrera wrote: Hello, On 2019-Jul-03, Tatsuro Yamada wrote: My ex-colleague Vinayak created same patch in 2017 [1], and he couldn't get commit because there are some reasons such as the patch couldn't handle analyzing Foreign table. Therefore, I wonder

Re: Improve Assert output

2019-08-14 Thread Tom Lane
Peter Eisentraut writes: > This kind of output is usually not helpful: > TRAP: BadArgument("((context) != ((void *)0) && (const > Node*)((context)))->type) == T_AllocSetContext) || const > Node*)((context)))->type) == T_SlabContext) || const > Node*)((context)))->type) ==

Re: progress report for ANALYZE

2019-08-14 Thread Alvaro Herrera
On 2019-Aug-14, Etsuro Fujita wrote: > On Tue, Aug 13, 2019 at 11:01 PM Alvaro Herrera > wrote: > > On the subject of FDW support: I did look into supporting that before > > submitting this. I think it's not academically difficult: just have the > > FDW's acquire_sample_rows callback invoke the

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-14 Thread Bruce Momjian
On Wed, Aug 14, 2019 at 04:36:35PM +0200, Antonin Houska wrote: > I can work on it right away but don't know where to start. I think the big open question is whether there will be acceptance of an all-cluster encyption feature. I guess if no one objects, we can move forward. > First, I think we

Re: [HACKERS] CLUSTER command progress monitor

2019-08-14 Thread Tatsuro Yamada
Hi Michael, Alvaro and Robert! On 2019/08/14 11:52, Michael Paquier wrote: On Wed, Aug 14, 2019 at 11:38:01AM +0900, Tatsuro Yamada wrote: On 2019/08/13 14:40, Tatsuro Yamada wrote: On 2019/08/02 3:43, Alvaro Herrera wrote: Hmm, I'm trying this out now and I don't see the index_rebuild_count

Re: Why is infinite_recurse test suddenly failing?

2019-08-14 Thread Tom Lane
Thomas Munro writes: > Here's another crash like that. > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=cavefish=2019-07-13%2003%3A49%3A38 > 2019-07-13 04:01:23.437 UTC [9365:70] LOG: server process (PID 12951) > was terminated by signal 11: Segmentation fault > 2019-07-13 04:01:23.437

Improve Assert output

2019-08-14 Thread Peter Eisentraut
This kind of output is usually not helpful: TRAP: BadArgument("((context) != ((void *)0) && (const Node*)((context)))->type) == T_AllocSetContext) || const Node*)((context)))->type) == T_SlabContext) || const Node*)((context)))->type) == T_GenerationContext)))", File:

Re: Zedstore - compressed in-core columnar storage

2019-08-14 Thread Ashutosh Sharma
Hi Ashwin, I tried playing around with the zedstore code a bit today and there are couple questions that came into my mind. 1) Can zedstore tables be vacuumed? If yes, does VACUUM on zedstore table set the VM bits associated with it. 2) Is there a chance that IndexOnlyScan would ever be

Re: POC: Cleaning up orphaned files using undo logs

2019-08-14 Thread Dilip Kumar
On Wed, Aug 14, 2019 at 12:27 PM Andres Freund wrote: > > Hi, > > - I think there's two fairly fundamental, and related, problems with > the sequence outlined above: > > - We can't search for target buffers to store undo data, while holding > the "table" page content locked. > > The

Re: POC: Cleaning up orphaned files using undo logs

2019-08-14 Thread Andres Freund
Hi, On 2019-08-13 13:53:59 +0530, Dilip Kumar wrote: > On Tue, Jul 30, 2019 at 1:32 PM Andres Freund wrote: > > > + /* Loop until we have fetched all the buffers in which we need to > > > write. */ > > > + while (size > 0) > > > + { > > > + bufidx =

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-14 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> Perhaps we could put some of these details into the Notes section of the >> ALTER SYSTEM ref page. But I wonder how much of this is needed at all. > I'd be alright with that too, but I'd be just as fine with even a README > or

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-14 Thread Antonin Houska
Bruce Momjian wrote: > On Sat, Aug 10, 2019 at 08:06:17AM -0400, Bruce Momjian wrote: > > So, I just had an indea if we use separate encryption keys for > > heap/index and for WAL --- we already know we will have an offline tool > > that can rotate the passphrase or encryption keys. If we allow

Re: use valgrind --leak-check=yes to detect memory leak

2019-08-14 Thread Tom Lane
Alex writes: > I want to use valgrind to detect memory leak issue. Then I run into 2 > problems I want to confirm them here. > 1. In https://wiki.postgresql.org/wiki/Valgrind, the wiki indicates to > use '--leak-check=no' That's just a sample configuration. > 2. in

Re: Auxiliary Processes and MyAuxProc

2019-08-14 Thread Yuli Khodorkovskiy
On Mon, Feb 25, 2019 at 5:25 PM Mike Palmiotto wrote: > > On Mon, Feb 25, 2019 at 1:41 PM Mike Palmiotto > wrote: > > > > > > > > > > If memory serves, StartChildProcess already was an attempt to unify > > > the treatment of postmaster children. It's possible that another > > > round of

Re: POC: Cleaning up orphaned files using undo logs

2019-08-14 Thread Andres Freund
Hi, On 2019-08-14 14:48:07 +0530, Dilip Kumar wrote: > On Wed, Aug 14, 2019 at 12:27 PM Andres Freund wrote: > > - I think there's two fairly fundamental, and related, problems with > > the sequence outlined above: > > > > - We can't search for target buffers to store undo data, while

Re: Zedstore - compressed in-core columnar storage

2019-08-14 Thread Ashwin Agrawal
On Wed, Aug 14, 2019 at 2:51 AM Ashutosh Sharma wrote: > Hi Ashwin, > > I tried playing around with the zedstore code a bit today and there > are couple questions that came into my mind. > Great! Thank You. > > 1) Can zedstore tables be vacuumed? If yes, does VACUUM on zedstore > table set

Re: Feature: Use DNS SRV records for connecting

2019-08-14 Thread Andres Freund
Hi, On 2019-08-13 10:43:07 -0400, Tom Lane wrote: > How would we get at that data without writing our own DNS client? > (AFAIK, our existing DNS interactions are all handled by getnameinfo() > or other library-supplied functions.) > Maybe that'd be worth doing, but it sounds like a lot of work

Re: proposal: make NOTIFY list de-duplication optional

2019-08-14 Thread Tom Lane
I wrote: > I think that we'd probably be better off fixing the root performance issue > than adding semantic complexity to bypass it. ... > Accordingly, I looked into making a hash table when there are more than > a small number of notifications pending, and attached is a lightly-tested > version

Re: "long" type is not appropriate for counting tuples

2019-08-14 Thread Andres Freund
Hi, On 2019-05-22 16:40:28 +0200, Peter Eisentraut wrote: > On 2019-04-29 19:52, Andres Freund wrote: > > Hm. It appears that gettext supports expanding PRId64 PRIu64 etc in > > translated strings. > > That won't work in non-GNU gettext. Which of those do we actually support? We already depend

Re: getting ERROR "relation 16401 has no triggers" with partition foreign key alter

2019-08-14 Thread Tom Lane
Alvaro Herrera writes: > On 2019-Jul-22, Alvaro Herrera wrote: >> After looking at the code some more, I think calling the new function in >> the Prep phase is correct. The attached patch is pretty much final form >> for this bugfix. I decided to unwrap a couple of error messages (I did >> get

Re: Removing unneeded downlink field from nbtree stack struct

2019-08-14 Thread Peter Geoghegan
On Mon, Aug 12, 2019 at 9:43 AM Anastasia Lubennikova wrote: > The refactoring is clear, so I set Ready for committer status. > I have just a couple of notes about comments: > > 1) I think that it's worth to add explanation of the case when we use > right sibling to this comment: > +

Re: getting ERROR "relation 16401 has no triggers" with partition foreign key alter

2019-08-14 Thread Alvaro Herrera
On 2019-Aug-14, Tom Lane wrote: > Alvaro Herrera writes: > > On 2019-Jul-22, Alvaro Herrera wrote: > >> After looking at the code some more, I think calling the new function in > >> the Prep phase is correct. The attached patch is pretty much final form > >> for this bugfix. I decided to

Re: POC: Cleaning up orphaned files using undo logs

2019-08-14 Thread Andres Freund
Hi, On 2019-08-13 17:05:27 +0530, Dilip Kumar wrote: > On Mon, Aug 5, 2019 at 11:59 PM Andres Freund wrote: > > (as I was out of context due to dealing with bugs, I've switched to > > lookign at the current zheap/undoprocessing branch. > > > > On 2019-07-30 01:02:20 -0700, Andres Freund wrote: >