Re: Have an encrypted pgpass file

2018-07-18 Thread Alvaro Herrera
On 2018-Jul-18, Marco van Eck wrote: > Since .pgpass files contain plain-text passwords, I searched for an > alternative. > In the attached patch I've added the possibility to run a command to > produce the content of the pgpass file, in exactly the same format. In this > way I could use gpg or

Re: print_path is missing GatherMerge and CustomScan support

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 12:15:25PM +0530, Ashutosh Bapat wrote: > Yes that's right. Thanks for taking care of it. Okay, I have pushed a fix for this one as that's wrong and back-patched to v11. The coverage of reparameterize_path_by_child is actually quite poor if you look at the reports:

Re: Add SKIP LOCKED to VACUUM and ANALYZE

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 04:58:48PM +, Bossart, Nathan wrote: > On 7/17/18, 1:22 AM, "Michael Paquier" wrote: > Perhaps we could extend RangeVarGetRelidExtended() to only lock if > has_subclass() is true. However, I also understand Robert's position > on calling RangeVarGetRelidExtended()

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2018-07-18 Thread Heikki Linnakangas
On 18/07/18 23:29, Fabien COELHO wrote: Hmm. How about we just remove this special case from doCustom(): case CSTATE_START_THROTTLE: // ... if (duration > 0 && st->txn_scheduled > end_time) { st->state = CSTATE_FINISHED; break; } That way, we let the

Re: Possible bug in logical replication.

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 02:30:53PM -0400, Alvaro Herrera wrote: > In the immortal words of Julian Bream: "yeah, I didn't like any of > that". One wikipedia lookup later, I still don't know where this quote comes from, but at least I understand who the man is. I may be missing something, but I

Re: YA race condition in 001_stream_rep.pl (was Re: pgsql: Allow using the updated tuple while moving it to a different par)

2018-07-18 Thread Tom Lane
Amit Kapila writes: > On Sat, Jul 14, 2018 at 11:46 PM, Tom Lane wrote: >> So the issue boils down to this: the test script is, effectively, >> assuming that it's guaranteed that the walreceiver will send a feedback >> message before it shuts down; but there is no such guarantee. Is this >> a

Re: psql's \d versus included-index-column feature

2018-07-18 Thread Alvaro Herrera
On 2018-Jul-18, Tom Lane wrote: > I can sympathize with the eyestrain argument against t/f, but the > above doesn't seem like an improvement --- in particular, "Data" > as the column header seems quite content-free. My counterproposal > is to keep "Key" as the header and use "Yes"/"No" as the

Re: Have an encrypted pgpass file

2018-07-18 Thread Tom Lane
Alvaro Herrera writes: > Seems to me that passing %-specifiers to the command would make it more > useful (%u for "user", "host" etc) -- your command could refuse to give > you a password for the superuser account for instance but grant one for > a read-only user. It would also provide a *very*

Re: untrusted PLs should be GRANTable

2018-07-18 Thread Stephen Frost
Greetings, * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote: > On 17.07.18 07:20, Craig Ringer wrote: > > A user has raised the point that our refusal to GRANT rights to > > untrusted PLs is counterproductive and inconsistent with how we behave > > elsewhere. > > Previous discussion:

Re: Possible bug in logical replication.

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 10:45:27PM -0400, Alvaro Herrera wrote: > On 2018-Jul-19, Michael Paquier wrote: > >> On Wed, Jul 18, 2018 at 02:30:53PM -0400, Alvaro Herrera wrote: >> > In the immortal words of Julian Bream: "yeah, I didn't like any of >> > that". >> >> One wikipedia lookup later, I

Re: Have an encrypted pgpass file

2018-07-18 Thread Tom Lane
"Joshua D. Drake" writes: > On 07/18/2018 04:25 PM, Tom Lane wrote: >> This is exactly the kind of area in which I'm concerned for the >> possibility of sloppily-written scripts being a net negative for >> security. > Although I appreciate the concern, can we not worried about this? Your >

Re: patch to allow disable of WAL recycling

2018-07-18 Thread Kyotaro HORIGUCHI
At Tue, 17 Jul 2018 21:01:03 -0400, Robert Haas wrote in > On Tue, Jul 17, 2018 at 3:12 PM, Peter Eisentraut > wrote: > > The actual implementation could use another round of consideration. I > > wonder how this should interact with min_wal_size. Wouldn't > > min_wal_size = 0 already do what

RE: GSOC 2018 Project - A New Sorting Routine

2018-07-18 Thread Kefan Yang
Hey Tomas! I am trying to reproduce the results on my machine. Could you please share the script to generate .ods files? Regards, Kefan From: Tomas Vondra Sent: July 18, 2018 2:05 AM To: Andrey Borodin Cc: Peter Geoghegan; Kefan Yang; PostgreSQL Hackers Subject: Re: GSOC 2018 Project - A New

Re: GiST VACUUM

2018-07-18 Thread Heikki Linnakangas
On 18/07/18 21:27, Andrey Borodin wrote: Hi! 18 июля 2018 г., в 16:02, Heikki Linnakangas написал(а): , but I think it would be better to split this into two patches as follows: 1st patch: Scan the index in physical rather than logical order. No attempt at deleting empty pages yet. 2nd

Re: Have an encrypted pgpass file

2018-07-18 Thread Thomas Munro
On Thu, Jul 19, 2018 at 9:52 AM, Tom Lane wrote: > Thomas Munro writes: >> On Thu, Jul 19, 2018 at 5:46 AM, Marco van Eck >> wrote: >>> Since .pgpass files contain plain-text passwords, I searched for an >>> alternative. >>> In the attached patch I've added the possibility to run a command to

RE: Recovery performance of standby for multiple concurrent truncates on large tables

2018-07-18 Thread Jamison, Kirk
Hi, Thank you for your replies. On Tue, July 10, 2018 4:15 PM, Andres Freund wrote: >I think you'd run into a lot of very hairy details with this approach. >Consider what happens if client processes need fresh buffers and need to write >out a victim buffer. You'll need to know that the relevant

Re: Possible bug in logical replication.

2018-07-18 Thread Alvaro Herrera
On 2018-Jul-19, Michael Paquier wrote: > On Wed, Jul 18, 2018 at 02:30:53PM -0400, Alvaro Herrera wrote: > > In the immortal words of Julian Bream: "yeah, I didn't like any of > > that". > > One wikipedia lookup later, I still don't know where this quote comes > from, but at least I understand

Re: More consistency for some file-related error message

2018-07-18 Thread Tom Lane
Michael Paquier writes: > On Wed, Jul 18, 2018 at 10:57:16AM -0400, Tom Lane wrote: >> BTW, isn't the initial "errno = 0" dead code now? > Hm. I have not bothered touching those as it could be possible that > read() may not initialize errno to 0, so errno would remain set to any > previous

Re: Postgres, fsync, and OSs (specifically linux)

2018-07-18 Thread Thomas Munro
On Thu, Jul 19, 2018 at 7:23 AM, Robert Haas wrote: > 2. I don't like promote_ioerr_to_panic() very much, partly because the > same pattern gets repeated over and over, and partly because it would > be awkwardly-named if we discovered that another 2 or 3 errors needed > similar handling (or some

Re: untrusted PLs should be GRANTable

2018-07-18 Thread Stephen Frost
Greetings, * Craig Ringer (cr...@2ndquadrant.com) wrote: > Untrusted PLs should be GRANTable with a NOTICE or WARNING telling the > admin that GRANTing an untrusted PL effectively gives the user the ability > to escape to superuser. I don't know that we really want to get into the business of

Re: Have an encrypted pgpass file

2018-07-18 Thread Joshua D. Drake
On 07/18/2018 04:25 PM, Tom Lane wrote: Alvaro Herrera writes: Seems to me that passing %-specifiers to the command would make it more useful (%u for "user", "host" etc) -- your command could refuse to give you a password for the superuser account for instance but grant one for a read-only

Re: [PATCH] pg_hba.conf : new auth option : clientcert=verify-full

2018-07-18 Thread Thomas Munro
On Sun, Jul 15, 2018 at 12:47 AM, Julian Markwort wrote: > Also, while writing this part of the docs, I tried to stay below 80 > characters, but I've exceeded it in some places. > There are several other places (several in the .sgml files touched by this > patch), where 80 characters are

Re: patch to allow disable of WAL recycling

2018-07-18 Thread Kyotaro HORIGUCHI
At Thu, 19 Jul 2018 12:37:26 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180719.123726.00899102.horiguchi.kyot...@lab.ntt.co.jp> > While considering this, I found a bug in 4b0d28de06, which > removed prior checkpoint from control file. It actually trims the > segments before the

Re: psql's \d versus included-index-column feature

2018-07-18 Thread Tom Lane
Alvaro Herrera writes: > On 2018-Jul-18, David G. Johnston wrote: >> -1 for printing a boolean t/f; would rather spell it out: >> >> CASE WHEN "Key" THEN 'Key' ELSE 'Included' END AS "Data" > +1 I can sympathize with the eyestrain argument against t/f, but the above doesn't seem like an

Re: Have an encrypted pgpass file

2018-07-18 Thread Thomas Munro
On Thu, Jul 19, 2018 at 5:46 AM, Marco van Eck wrote: > Since .pgpass files contain plain-text passwords, I searched for an > alternative. > In the attached patch I've added the possibility to run a command to produce > the content of the pgpass file, in exactly the same format. In this way I >

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2018-07-18 Thread Fabien COELHO
Hello Heikki, [...] So threadRun() would not have the opportunity to stop the scheduled transaction, even if beyond the end of run, because it would not have got out of doCustom, in the case I outlined above. I see. Instead of moving to FINISHED state, then, we could stay in THROTTLE state,

Re: More consistency for some file-related error message

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 10:57:16AM -0400, Tom Lane wrote: > +1. Okay, thanks. I can always get that pushed first if there are no objections. More can be always done, but that's already a nice cut. > The short-read ereport has no errcode() call, meaning it will report > XX000, which seems like

Re: GSOC 2018 Project - A New Sorting Routine

2018-07-18 Thread Tomas Vondra
I don't have any script for that - load the files into a spreadsheet, create pivot tables and you're done. regards On 07/18/2018 11:13 PM, Kefan Yang wrote: > Hey Tomas! > >   > > I am trying to reproduce the results on my machine. Could you please > share the script to generate .ods files? >

Re: Have an encrypted pgpass file

2018-07-18 Thread Christophe Pettus
> On Jul 18, 2018, at 14:33, Thomas Munro wrote: > Here you side step those questions completely and make that the end > user's problem. I like it. +1. This is a clever solution, since any kind of key vault or other system could be dropped in there. -- -- Christophe Pettus

Re: Have an encrypted pgpass file

2018-07-18 Thread Tom Lane
Thomas Munro writes: > On Thu, Jul 19, 2018 at 5:46 AM, Marco van Eck wrote: >> Since .pgpass files contain plain-text passwords, I searched for an >> alternative. >> In the attached patch I've added the possibility to run a command to produce >> the content of the pgpass file, in exactly the

Re: [HACKERS] possible self-deadlock window after bad ProcessStartupPacket

2018-07-18 Thread Asim R P
On Thu, Jun 22, 2017 at 10:50 AM, Andres Freund wrote: > > Or, probably more robust: Simply _exit(2) without further ado, and rely > on postmaster to output an appropriate error message. Arguably it's not > actually useful to see hundreds of "WARNING: terminating connection because of > crash of

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-18 Thread Heikki Linnakangas
On 18/07/18 16:29, Robert Haas wrote: On Wed, Jul 18, 2018 at 9:06 AM, Michael Paquier wrote: What's wrong with the approach proposed in http://postgr.es/m/55afc302.1060...@iki.fi ? For back-branches that's very invasive so that seems risky to me particularly seeing the low number of

Re: Make foo=null a warning by default.

2018-07-18 Thread David Fetter
On Wed, Jul 18, 2018 at 08:25:46AM -0400, Fabien COELHO wrote: > > Hello David, > > >I assure you that you expression yourself in English a good deal > >better than I do in Portuguese. > > Alas, despite a Portuguese "rabbit" name, I cannot speak the language which > got lost between

One transaction and several processes

2018-07-18 Thread Valery Kuzmin
Hi, Can multiple processes participate in a single transaction's execution? I need to do the following actions sequence: 1. Calling service begins transaction. At this step transaction manager generate new XID. 2. Calling service updates some data. 3. Need to detach a transaction from a service

Re: One transaction and several processes

2018-07-18 Thread David G. Johnston
On Wed, Jul 18, 2018 at 8:31 AM, Valery Kuzmin wrote: > Hi, > > Can multiple processes participate in a single transaction's execution? > I need to do the following actions sequence: > 1. Calling service begins transaction. At this step transaction > manager generate new XID. > 2. Calling

Re: One transaction and several processes

2018-07-18 Thread Tomas Vondra
On 07/18/2018 05:31 PM, Valery Kuzmin wrote: Hi, Can multiple processes participate in a single transaction's execution? I need to do the following actions sequence: 1. Calling service begins transaction. At this step transaction manager generate new XID. 2. Calling service updates some data.

Re: Segfault logical replication PG 10.4

2018-07-18 Thread Mai Peng
Hello, Here the backtrace ```Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `postgres: bgworker: logical replication worker for subscription'. Program terminated with signal SIGSEGV, Segmentation fault. #0 GetActiveSnapshot () at

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-18 Thread Robert Haas
On Wed, Jul 18, 2018 at 10:08 AM, Tomas Vondra wrote: > The problem is you don't know if a transaction does DDL sometime later, in > the part that you might not have decoded yet (or perhaps concurrently with > the decoding). So I don't see how you could easily exclude such transactions > from the

Re: More consistency for some file-related error message

2018-07-18 Thread Tom Lane
Michael Paquier writes: > While looking at the source code for more consistency work with error > messages, I have bumped into a couple of messages which could be > simplified, as those include in the name of the file manipulated > basically the same information as the context added. > I have

Re: Segfault logical replication PG 10.4

2018-07-18 Thread Tom Lane
Mai Peng writes: > Here the backtrace Hmm .. so this can be summarized as "logical replication workers should provide an ActiveSnapshot in case the user functions they call want one". Makes me wonder how much other transactional infrastructure is needed but not present.

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-18 Thread Tomas Vondra
On 07/17/2018 08:10 PM, Robert Haas wrote: On Mon, Jul 16, 2018 at 3:25 PM, Tomas Vondra wrote: Oh, right, I forgot the patch also adds the leader into the group, for some reason (I agree it's unclear why that would be necessary, as you pointed out later). But all this is happening while

Re: partition tree inspection functions

2018-07-18 Thread Jesper Pedersen
Hi Amit, On 06/28/2018 01:49 AM, Amit Langote wrote: OK, I've added an example below the table of functions added by the patch. Attached updated patch. You forgot to remove the test output in create_table.out, so check-world is failing. In pg_partition_parent + else + /* Not

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-18 Thread Tomas Vondra
On 07/18/2018 04:56 PM, Robert Haas wrote: On Wed, Jul 18, 2018 at 10:08 AM, Tomas Vondra wrote: The problem is you don't know if a transaction does DDL sometime later, in the part that you might not have decoded yet (or perhaps concurrently with the decoding). So I don't see how you could

RE: [bug fix] Produce a crash dump before main() on Windows

2018-07-18 Thread Tsunakawa, Takayuki
From: Haribabu Kommi [mailto:kommi.harib...@gmail.com] > I reviewed patch and it works as per the subject, but I am not able to verify > the actual > bug that is reported in the upthread. The moving of setErrorMode() call > to the start > of the main function can handle all the cases that can lead

Re: print_path is missing GatherMerge and CustomScan support

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 02:35:23PM +0900, Masahiko Sawada wrote: > Hi, > > While debugging planner I realized that print_path() function is not > aware of both GatherMerge path and CustomScan path. Attached small > patch fixes it. Good catch. Those should be backpatched. While I am looking at

Re: [bug fix] Produce a crash dump before main() on Windows

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 06:09:57AM +, Tsunakawa, Takayuki wrote: > From: Haribabu Kommi [mailto:kommi.harib...@gmail.com] >> I reviewed patch and it works as per the subject, but I am not able to verify >> the actual >> bug that is reported in the upthread. The moving of setErrorMode() call >>

Re: print_path is missing GatherMerge and CustomScan support

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 12:15:25PM +0530, Ashutosh Bapat wrote: > On Wed, Jul 18, 2018 at 11:52 AM, Michael Paquier wrote: >> On Wed, Jul 18, 2018 at 02:35:23PM +0900, Masahiko Sawada wrote: >>> Hi, >>> >>> While debugging planner I realized that print_path() function is not >>> aware of both

RE: Temporary tables prevent autovacuum, leading to XID wraparound

2018-07-18 Thread Tsunakawa, Takayuki
From: Michael Paquier [mailto:mich...@paquier.xyz] > + /* Does the backend own the temp schema? */ > + if (proc->tempNamespaceId != namespaceID) > + return false; > I have a very hard time believing that this is safe lock-less, and a spin > lock would be enough it seems. The lwlock in

Re: [bug fix] Produce a crash dump before main() on Windows

2018-07-18 Thread Haribabu Kommi
On Wed, Jul 18, 2018 at 4:10 PM Tsunakawa, Takayuki < tsunakawa.ta...@jp.fujitsu.com> wrote: > From: Haribabu Kommi [mailto:kommi.harib...@gmail.com] > > I reviewed patch and it works as per the subject, but I am not able to > verify > > the actual > > bug that is reported in the upthread. The

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-07-18 Thread Konstantin Knizhnik
On 18.07.2018 02:58, Tomas Vondra wrote: On 07/18/2018 12:41 AM, Konstantin Knizhnik wrote: ... Teodor Sigaev has proposed an alternative approach for calculating selectivity of multicolumn join or compound index search. Usually DBA creates compound indexes which can be used  by optimizer

Re: print_path is missing GatherMerge and CustomScan support

2018-07-18 Thread Ashutosh Bapat
On Wed, Jul 18, 2018 at 11:52 AM, Michael Paquier wrote: > On Wed, Jul 18, 2018 at 02:35:23PM +0900, Masahiko Sawada wrote: >> Hi, >> >> While debugging planner I realized that print_path() function is not >> aware of both GatherMerge path and CustomScan path. Attached small >> patch fixes it. >

Re: [bug fix] Produce a crash dump before main() on Windows

2018-07-18 Thread Craig Ringer
On 18 July 2018 at 14:34, Michael Paquier wrote: > On Wed, Jul 18, 2018 at 06:09:57AM +, Tsunakawa, Takayuki wrote: > > From: Haribabu Kommi [mailto:kommi.harib...@gmail.com] > >> I reviewed patch and it works as per the subject, but I am not able to > verify > >> the actual > >> bug that is

Re: [HACKERS] PoC: full merge join on comparison clause

2018-07-18 Thread Ashutosh Bapat
On Tue, Jul 10, 2018 at 10:06 PM, Alexander Kuzmenkov wrote: > I tried to fix the things you mentioned and improve the comments. Among > other changes, there is now a description of how merge join works with > inequalities at the top of nodeMergejoin.c. It also explains why we only > support one

Re: ENOSPC FailedAssertion("!(RefCountErrors == 0)"

2018-07-18 Thread Tom Lane
Kyotaro HORIGUCHI writes: > Hello. I confirmed that this patch fixes the crash. Thanks for double-checking. > At Tue, 17 Jul 2018 20:01:05 -0400, Tom Lane wrote in > <14892.1531872...@sss.pgh.pa.us> >> Further investigation showed that the part of that code that was >> actually needed was not

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2018-07-18 Thread Heikki Linnakangas
On 18/07/18 16:01, Fabien COELHO wrote: I don't think you want to wait in that situation. I think we should wait at the end only if there some threads still alive, with nothing to do only because of --rate. Yep. The attached version does only the tailing stuff under -R and not all threads were

Re: GiST VACUUM

2018-07-18 Thread Andrey Borodin
Hi! > 18 июля 2018 г., в 16:02, Heikki Linnakangas написал(а): > > In the corresponding B-tree code, we use don't do actual recursion, but a > hand-optimized "tail recursion", to avoid stack overflow if there are a lot > of splits. I think we need to do something like tha there, too. I don't

Have an encrypted pgpass file

2018-07-18 Thread Marco van Eck
Hi, Since .pgpass files contain plain-text passwords, I searched for an alternative. In the attached patch I've added the possibility to run a command to produce the content of the pgpass file, in exactly the same format. In this way I could use gpg or any other command to decrypt a pgpass file.

Re: ENOSPC FailedAssertion("!(RefCountErrors == 0)"

2018-07-18 Thread Tom Lane
Robert Haas writes: > On Wed, Jul 18, 2018 at 10:33 AM, Tom Lane wrote: >>> So couldn't we use TopTransactionResourceOwner instead of >>> AuxProcessResrouceOwner? I feel a bit uneasy that bootstrap and >>> standalone-backend have *AuxProcess*ResourceOwner. >> Since the aux processes aren't

Re: Faster str to int conversion (was Table with large number of int columns, very slow COPY FROM)

2018-07-18 Thread Robert Haas
On Sat, Jul 7, 2018 at 4:01 PM, Andres Freund wrote: > FWIW, here's a rebased version of this patch. Could probably be polished > further. One might argue that we should do a bit more wide ranging > changes, to convert scanint8 and pg_atoi to be also unified. But it > might also just be

Background worker/idle sessions and caching

2018-07-18 Thread Jeremy Finzel
I have a background worker running SQL functions, and I believe I have noticed that when I do things like change function definitions, or even add tables, the background worker does not pick up the schema changes until I restart the worker. Is this expected behavior? If I use background workers

Re: [PG-11] Potential bug related to INCLUDE clause of CREATE INDEX

2018-07-18 Thread Tom Lane
Yugo Nagata writes: > To fix this, we agree with Tom about getting rid of "must not intersect" > restriction. > A patch is attached for this Pushed, after fixing documentation and regression tests to match. regards, tom lane

Re: missing toast table for pg_policy

2018-07-18 Thread John Naylor
On 7/17/18, Michael Paquier wrote: > [... digging ...] > This comes from get_rel_infos where large objects are treated as user > data. Rather than the comment you added, I would rather do the > following: > "Large object catalogs and toast tables are mutually exclusive and large > object data is

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2018-07-18 Thread Heikki Linnakangas
On 18/07/18 01:43, Fabien COELHO wrote: The more reasonable alternative could be to always last 2 seconds under -T 2, even if the execution can be shorten because there is nothing to do at all, i.e. remove the environment-based condition but keep the sleep. That sounds reasonable. It's a bit

Re: cursors with prepared statements

2018-07-18 Thread Heikki Linnakangas
On 16/07/18 15:56, Peter Eisentraut wrote: On 11.07.18 19:07, Heikki Linnakangas wrote: It's confusing, and risks conflicting with future additions to the standard. ECPG supports the actual standard syntax, with OPEN, right? So this wouldn't be consistent with ECPG, either. It would be

Re: why partition pruning doesn't work?

2018-07-18 Thread Amit Langote
On 2018/07/16 2:02, Tom Lane wrote: > Amit Langote writes: >> On 2018/06/19 2:05, Tom Lane wrote: >>> Or maybe what we should do is drop ExecLockNonLeafAppendTables/ >>> ExecOpenAppendPartitionedTables entirely and teach InitPlan to do it. > >> Hmm, for InitPlan to do what

Re: Bug in gin insert redo code path during re-compression of empty gin data leaf pages

2018-07-18 Thread Alexander Korotkov
Hi! On Wed, Jul 18, 2018 at 1:40 AM R, Siva wrote: > We came across an issue during replay of a gin insert record on a pre-9.4 > uncompressed data leaf page that does not have any items in it. The engine > version where the replay is done is 9.6.3. The redo logic attempts to > compress the

RE: Speeding up INSERTs and UPDATEs to partitioned tables

2018-07-18 Thread Kato, Sho
On 2018-Jul-11, Alvaro Herrera wrote: > That commit is also in pg11, though -- just not in beta2. So we still don't > know how much of an improvement patch2 is by itself :-) Oops! I benchmarked with 11beta2 + 0001-Speed​​-up-INSERT-and-UPDATE-on-partitioned-tables.patch. Results are as

Re[2]: Alter index rename concurrently to

2018-07-18 Thread Andrey Klychkov
>Среда, 18 июля 2018, 12:21 +03:00 от Peter Eisentraut >: > >If we think the lower lock level is OK, then we should just use >it always. > Hi, I absolutely agree with you. If lower locking is safe and possible to be used by default in renaming it will be great. What stage is solving of this

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-07-18 Thread Amit Langote
Hi David, Thanks for taking a look. On 2018/07/15 17:34, David Rowley wrote: > I've looked over the code and the ExecUseUpdateResultRelForRouting() > function is broken. Your while loop only skips partitions for the > current partitioned table, it does not skip ModifyTable subnodes that >

RE: [bug fix] Produce a crash dump before main() on Windows

2018-07-18 Thread Tsunakawa, Takayuki
From: Haribabu Kommi [mailto:kommi.harib...@gmail.com] > May be I can give a try by modifying the source code to get the crash. Thank you, that would be great if you could come up with a good way! > My point is, With this patch, in case if the postgres crashses > before reaching main(), does it

Re: Alter index rename concurrently to

2018-07-18 Thread Peter Eisentraut
On 17.07.18 13:48, Andrey Klychkov wrote: > Please, have a look at previous discussions on the subject: > - 2012 > > https://www.postgresql.org/message-id/cab7npqtys6juqdxuczbjb0bnw0kprw8wdzuk11kaxqq6o98...@mail.gmail.com > - > 2013  >

Re: Incorrect error handling for two-phase state files resulting in data loss

2018-07-18 Thread Michael Paquier
On Mon, Jul 09, 2018 at 02:03:09PM +0900, Michael Paquier wrote: > I think that we really need to harden things, by making > ReadTwoPhaseFile() fail hard is it finds something unexpected, which is > in this case anything except trying to open a file which fails on > ENOENT, and that this stuff

Re: PG 10: could not generate random cancel key

2018-07-18 Thread Dean Rasheed
On 18 July 2018 at 03:17, Michael Paquier wrote: >> [1] https://wiki.openssl.org/index.php/Random_Numbers > > This quote from the wiki is scary so that's not quite clean either for > Windows: > "Be careful when deferring to RAND_poll on some Unix systems because it > does not seed the generator.

Re: Add SKIP LOCKED to VACUUM and ANALYZE

2018-07-18 Thread Bossart, Nathan
Hi Michael, Thanks for taking a look. On 7/17/18, 1:22 AM, "Michael Paquier" wrote: > The first thing which is striking me is that we may actually *not* want > to check for lock skipping within expand_vacuum_rel() as that's mainly a > function aimed at building the relations which are going to

Re: ENOSPC FailedAssertion("!(RefCountErrors == 0)"

2018-07-18 Thread Robert Haas
On Wed, Jul 18, 2018 at 10:33 AM, Tom Lane wrote: >> So couldn't we use TopTransactionResourceOwner instead of >> AuxProcessResrouceOwner? I feel a bit uneasy that bootstrap and >> standalone-backend have *AuxProcess*ResourceOwner. > > Since the aux processes aren't running transactions, I didn't

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-18 Thread Robert Haas
On Wed, Jul 18, 2018 at 11:27 AM, Tomas Vondra wrote: >> One idea is that maybe the running transaction could communicate with >> the decoding process through shared memory. For example, suppose that >> before you begin decoding an ongoing transaction, you have to send >> some kind of

Re: Possible bug in logical replication.

2018-07-18 Thread Alvaro Herrera
On 2018-Jul-19, Michael Paquier wrote: > I am fine either way if you want to have the last call. So please feel > free to choose what you prefer here. That's no big deal. Okay. You want to push it, or shall I? -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL

Re: Background worker/idle sessions and caching

2018-07-18 Thread Craig Ringer
On 19 July 2018 at 04:30, Jeremy Finzel wrote: My use case is similar to the example of worker_spi. A plpgsql function > runs every 1 minute and processes records in audit tables in order to > update fact tables with records that have changed. I noticed for example > renaming a column in the

Re: Possible bug in logical replication.

2018-07-18 Thread Michael Paquier
On Thu, Jul 19, 2018 at 12:38:53AM -0400, Alvaro Herrera wrote: > On 2018-Jul-19, Michael Paquier wrote: >> I am fine either way if you want to have the last call. So please feel >> free to choose what you prefer here. That's no big deal. > > Okay. You want to push it, or shall I? It seems to

Re: More consistency for some file-related error message

2018-07-18 Thread Kyotaro HORIGUCHI
Hello. At Thu, 19 Jul 2018 12:33:30 +0900, Michael Paquier wrote in <2018071900.gh3...@paquier.xyz> > On Wed, Jul 18, 2018 at 11:24:05PM -0400, Tom Lane wrote: > > read() is required by spec to set errno when returning a negative result. > > I think the previous coding paid attention to

Re: [HACKERS] PATCH: Keep one postmaster monitoring pipe per process

2018-07-18 Thread Thomas Munro
On Wed, Jul 18, 2018 at 8:30 PM, Kyotaro HORIGUCHI wrote: > At Wed, 18 Jul 2018 14:02:47 +1200, Thomas Munro > wrote in > >> Here are some of the places I had to add WL_EXIT_ON_PM_DEATH: >> gather_readnext(), shm_mq_send_bytes(), shm_mq_receive_bytes(), >> shm_mq_wait_internal(), ProcSleep(),

project updates

2018-07-18 Thread Charles Cui
Hi mentors and hackers, Here is my current working status. 1. Complete the thrift_binary_in and thrift_binary_out functions, so that users can express their thrift struct using json. These two functions support both simple data struct and complex data structure like struct and map. 2. added

Re: print_path is missing GatherMerge and CustomScan support

2018-07-18 Thread Ashutosh Bapat
On Thu, Jul 19, 2018 at 5:37 AM, Michael Paquier wrote: > On Wed, Jul 18, 2018 at 12:15:25PM +0530, Ashutosh Bapat wrote: >> Yes that's right. Thanks for taking care of it. > > Okay, I have pushed a fix for this one as that's wrong and > back-patched to v11. The coverage of

Re: More consistency for some file-related error message

2018-07-18 Thread Michael Paquier
On Thu, Jul 19, 2018 at 02:05:51PM +0900, Kyotaro HORIGUCHI wrote: > Agreed to it's not necessary and a developer ought to know about > the errno behavior. However, I can sympathize with Michael. I am fine to remove them if folks here push for that. > CopyGetData has a variant of it. > > |

Re: Make foo=null a warning by default.

2018-07-18 Thread David Fetter
On Wed, Jul 18, 2018 at 04:15:30PM -0400, Fabien COELHO wrote: > > >[...] Done. > > All looks well, but I just noticed a warning: > > gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement > -Wendif-labels -Wmissing-format-attribute -Wformat-security >

Re: untrusted PLs should be GRANTable

2018-07-18 Thread Craig Ringer
On 19 July 2018 at 08:23, Stephen Frost wrote: > Greetings, > > * Craig Ringer (cr...@2ndquadrant.com) wrote: > > Untrusted PLs should be GRANTable with a NOTICE or WARNING telling the > > admin that GRANTing an untrusted PL effectively gives the user the > ability > > to escape to superuser. >

Re: documentation about explicit locking

2018-07-18 Thread Amit Langote
On 2018/07/18 18:30, Peter Eisentraut wrote: > On 06.07.18 04:00, Amit Langote wrote: >> On 2018/07/05 23:02, Robert Haas wrote: >>> On Wed, Jul 4, 2018 at 3:09 AM, Amit Langote >>> wrote: I wonder why we mention on the following page that CREATE COLLATION requires SHARE ROW EXCLUSIVE

Re: patch to allow disable of WAL recycling

2018-07-18 Thread Kyotaro HORIGUCHI
At Thu, 19 Jul 2018 12:37:26 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180719.123726.00899102.horiguchi.kyot...@lab.ntt.co.jp> > At Tue, 17 Jul 2018 21:01:03 -0400, Robert Haas wrote > in > > On Tue, Jul 17, 2018 at 3:12 PM, Peter Eisentraut > > wrote: > > > The actual

Re: patch to allow disable of WAL recycling

2018-07-18 Thread Kyotaro HORIGUCHI
At Thu, 19 Jul 2018 12:59:26 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180719.125926.257896670.horiguchi.kyot...@lab.ntt.co.jp> > At Thu, 19 Jul 2018 12:37:26 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in >

Re: Expression errors with "FOR UPDATE" and postgres_fdw with partition wise join enabled.

2018-07-18 Thread Etsuro Fujita
(2018/07/13 23:05), Ashutosh Bapat wrote: On Thu, Jul 12, 2018 at 4:32 PM, Etsuro Fujita wrote: In this example, the value of the whole-row reference to the child table ptp1 for that record is ('foo',1), and that of the index expression for that record is (1,'foo'). Those have different

Re: Possible bug in logical replication.

2018-07-18 Thread Erik Rijkers
On 2018-07-19 03:42, Michael Paquier wrote: On Wed, Jul 18, 2018 at 02:30:53PM -0400, Alvaro Herrera wrote: In the immortal words of Julian Bream: "yeah, I didn't like any of that". One wikipedia lookup later, I still don't know where this quote comes from, but at least I understand who the

Re: psql's \d versus included-index-column feature

2018-07-18 Thread Alexander Korotkov
On Wed, Jul 18, 2018 at 11:14 PM David G. Johnston < david.g.johns...@gmail.com> wrote: > On Wed, Jul 18, 2018 at 12:55 PM, Tom Lane wrote: > >> >> regression=# \d tbl_include_reg_idx >> Index "public.tbl_include_reg_idx" >> Column | Type | Key | Definition >>

Re: YA race condition in 001_stream_rep.pl (was Re: pgsql: Allow using the updated tuple while moving it to a different par)

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 05:42:40PM -0400, Tom Lane wrote: > Hearing nobody speaking in favor of the other alternatives, > I've removed the test. I had this problem running in the background for a couple of days, but I could not come up with a solution cleaner than just removing the test. Thanks

Re: print_path is missing GatherMerge and CustomScan support

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 03:22:02PM +0900, Michael Paquier wrote: > Good catch. Those should be backpatched. While I am looking at this > stuff, I have noticed that pathnode.c/reparameterize_path_by_child uses > T_MergeAppend and not T_MergeAppendPath. Okay, I have checked the full list of path

Re: More consistency for some file-related error message

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 11:24:05PM -0400, Tom Lane wrote: > read() is required by spec to set errno when returning a negative result. > I think the previous coding paid attention to errno regardless of the sign > of the result, which would justify pre-zeroing it ... but the new coding > definitely

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2018-07-18 Thread Fabien COELHO
Hello Heikki, Yep. The attached version does only the tailing stuff under -R and not all threads were stopped on errors, with comments to tell about the why. Hmm. How about we just remove this special case from doCustom(): case CSTATE_START_THROTTLE: // ... if (duration > 0 &&

psql's \d versus included-index-column feature

2018-07-18 Thread Tom Lane
I noticed that psql's \d command doesn't do very well with included index columns. Given the regression db's test case, CREATE INDEX tbl_include_reg_idx ON tbl_include_reg (c1, c2) INCLUDE (c3, c4); we get regression=# \d tbl_include_reg_idx Index "public.tbl_include_reg_idx" Column | Type

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2018-07-18 Thread Heikki Linnakangas
On 18/07/18 22:56, Fabien COELHO wrote: Hello Heikki, Yep. The attached version does only the tailing stuff under -R and not all threads were stopped on errors, with comments to tell about the why. Hmm. How about we just remove this special case from doCustom(): case

Re: Background worker/idle sessions and caching

2018-07-18 Thread Tom Lane
Jeremy Finzel writes: > I have a background worker running SQL functions, and I believe I have > noticed that when I do things like change function definitions, or even add > tables, the background worker does not pick up the schema changes until I > restart the worker. Maybe you need some

Re: psql's \d versus included-index-column feature

2018-07-18 Thread David G. Johnston
On Wed, Jul 18, 2018 at 12:55 PM, Tom Lane wrote: > > regression=# \d tbl_include_reg_idx > Index "public.tbl_include_reg_idx" > Column | Type | Key | Definition > +-+-- > c1 | integer | t | c1 > c2 | integer | t | c2 > c3 | integer | f |

  1   2   >