Re: Have an encrypted pgpass file

2018-07-20 Thread Tom Lane
Isaac Morland writes: >>> It would also provide a *very* fertile source of shell-script-injection >>> vulnerabilities. (Whaddya mean, you tried to use a user name with a >>> quote mark in it?) > If I understand the proposal correctly, the pgpass program would run on the > client, invoked by

Re: small development tip: Consider using the gold linker

2018-07-20 Thread Peter Geoghegan
On Mon, Jul 9, 2018 at 4:40 PM, Andres Freund wrote: > FWIW, I've lately noticed that I spend a fair time waiting for the > linker during edit-compile-test cycles. Due to an independent issue I > just used the gold linker, and the speedup is quite noticable. > > Just relinking the backend,

Re: Have an encrypted pgpass file

2018-07-20 Thread Isaac Morland
On 20 July 2018 at 17:22, Tels wrote: > Moin, > > > It would also provide a *very* fertile source of shell-script-injection > > vulnerabilities. (Whaddya mean, you tried to use a user name with a > > quote mark in it?) > > Little Bobby Tables, we call him. :) > > I'm also concerned that that

Re: patch to allow disable of WAL recycling

2018-07-20 Thread Jerry Jelinek
Thomas, Thanks for your offer to run some tests on different OSes and filesystems that you have. Anything you can provide here would be much appreciated. I don't have anything other than our native SmartOS/ZFS based configurations, but I might be able to setup some VMs and get results that way.

Re: BUG #15182: Canceling authentication due to timeout aka Denial of Service Attack

2018-07-20 Thread Marko Tiikkaja
On Fri, Jul 20, 2018 at 2:17 AM, Jeremy Schneider wrote: > I'd like to bump this old bug that Lloyd filed for more discussion. It > seems serious enough to me that we should at least talk about it. > > Anyone with simply the login privilege and the ability to run SQL can > instantly block all

Re: patch to allow disable of WAL recycling

2018-07-20 Thread Jerry Jelinek
Peter, Thanks for your feedback. I'm happy to change the name of the tunable or to update the man page in any way. I have already posted an updated patch with changes to the man page which I think may address your concerns there, but please let me know if that still needs more work. It looks

Re: [HACKERS] possible self-deadlock window after bad ProcessStartupPacket

2018-07-20 Thread Ashwin Agrawal
On Fri, Jul 20, 2018 at 1:54 AM Heikki Linnakangas wrote: > On 19/07/18 23:13, Andres Freund wrote: > > On 2018-07-19 14:54:52 -0500, Nico Williams wrote: > >> On Thu, Jul 19, 2018 at 12:20:53PM -0700, Andres Freund wrote: > >>> On 2018-07-19 11:57:25 +0300, Heikki Linnakangas wrote: > Ugh.

Re: patch to allow disable of WAL recycling

2018-07-20 Thread Jerry Jelinek
Hi Robert, I'm new to the Postgresql community, so I'm not familiar with how patches are accepted here. Thanks for your detailed explanation. I do want to keep pushing on this. I'll respond separately to Peter and to Tomas regarding their emails. Thanks again, Jerry On Wed, Jul 18, 2018 at

Re: [HACKERS] Two pass CheckDeadlock in contentent case

2018-07-20 Thread Yura Sokolov
11.07.2018 17:01, Ashutosh Bapat пишет: > The patch still applies and it's part of this commitfest. > > On Tue, Oct 3, 2017 at 8:36 PM, Sokolov Yura wrote: >> On 2017-10-03 17:30, Sokolov Yura wrote: >>> >>> Good day, hackers. >>> >>> During hard workload sometimes process reaches deadlock

Re: Have an encrypted pgpass file

2018-07-20 Thread Tels
Moin, On Wed, July 18, 2018 7:25 pm, Tom Lane wrote: > Alvaro Herrera writes: >> Seems to me that passing %-specifiers to the command would make it more >> useful (%u for "user", "host" etc) -- your command could refuse to give >> you a password for the superuser account for instance but grant

Re: Non-portable shell code in pg_upgrade tap tests

2018-07-20 Thread Tels
Moin, On Fri, July 20, 2018 10:55 am, Victor Wagner wrote: > On Fri, 20 Jul 2018 10:25:47 -0400 > Tom Lane wrote: > >> Victor Wagner writes: >> > I've discovered that in the branch REL_11_STABLE there is shell >> > script src/bin/pg_upgrade/test.sh which doesn't work under Solaris >> > 10. (it

Re: pread() and pwrite()

2018-07-20 Thread Daniel Gustafsson
> On 20 Jul 2018, at 17:34, Tom Lane wrote: > > Heikki Linnakangas writes: >> No objections, if you want to make the effort. But IMHO the lseek+read >> fallback is good enough on Windows. Unless you were thinking that we >> could then remove the !HAVE_PREAD fallback altogether. Are there any

Re: Possible performance regression in version 10.1 with pgbench read-write tests.

2018-07-20 Thread Andres Freund
Hi, On 2018-07-20 16:43:33 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2018-07-20 15:35:39 -0400, Tom Lane wrote: > >> In any case, I strongly resist making performance-based changes on > >> the basis of one test on one kernel and one hardware platform. > > > Sure, it'd be good to do

Re: Possible performance regression in version 10.1 with pgbench read-write tests.

2018-07-20 Thread Tom Lane
Andres Freund writes: > On 2018-07-20 15:35:39 -0400, Tom Lane wrote: >> In any case, I strongly resist making performance-based changes on >> the basis of one test on one kernel and one hardware platform. > Sure, it'd be good to do more of that. But from a theoretical POV it's > quite logical

Re: buildfarm: could not read block 3 in file "base/16384/2662": read only 0 of 8192 bytes

2018-07-20 Thread Andres Freund
On 2018-07-20 16:15:14 -0400, Tom Lane wrote: > We've seen several occurrences of $subject in the buildfarm in the past > month or so. Scraping the logs, I find > > coypu| 2018-06-14 21:17:49 | HEAD | Check | 2018-06-14 > 23:31:44.505 CEST [5b22deb8.30e1:124] ERROR: could not

Re: [HACKERS][PATCH] Applying PMDK to WAL operations for persistent memory

2018-07-20 Thread Heikki Linnakangas
On 01/03/18 12:40, Heikki Linnakangas wrote: On 16/01/18 15:00, Yoshimi Ichiyanagi wrote: These patches enable to use Persistent Memory Development Kit(PMDK)[1] for reading/writing WAL logs on persistent memory(PMEM). PMEM is next generation storage and it has a number of nice features: fast,

buildfarm: could not read block 3 in file "base/16384/2662": read only 0 of 8192 bytes

2018-07-20 Thread Tom Lane
We've seen several occurrences of $subject in the buildfarm in the past month or so. Scraping the logs, I find coypu| 2018-06-14 21:17:49 | HEAD | Check | 2018-06-14 23:31:44.505 CEST [5b22deb8.30e1:124] ERROR: could not read block 3 in file "base/16384/2662": read only 0 of

Re: Possible performance regression in version 10.1 with pgbench read-write tests.

2018-07-20 Thread Andres Freund
On 2018-07-20 15:35:39 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2018-07-21 00:53:28 +0530, Mithun Cy wrote: > >> I did a quick test applying the patch with same settings as initial mail I > >> have reported (On postgresql 10 latest code) > >> 72 clients > >> > >> CASE 1: > >>

Re: Possible performance regression in version 10.1 with pgbench read-write tests.

2018-07-20 Thread Tom Lane
Andres Freund writes: > On 2018-07-21 00:53:28 +0530, Mithun Cy wrote: >> I did a quick test applying the patch with same settings as initial mail I >> have reported (On postgresql 10 latest code) >> 72 clients >> >> CASE 1: >> Without Patch : TPS 29269.823540 >> >> With Patch : TPS

Re: Possible performance regression in version 10.1 with pgbench read-write tests.

2018-07-20 Thread Mithun Cy
On Fri, Jul 20, 2018 at 10:52 AM, Thomas Munro < thomas.mu...@enterprisedb.com> wrote: > On Fri, Jul 20, 2018 at 7:56 AM, Tom Lane wrote: > > > > It's not *that* noticeable, as I failed to demonstrate any performance > > difference before committing the patch. I think some more investigation >

Re: Segfault logical replication PG 10.4

2018-07-20 Thread Alvaro Herrera
On 2018-Jul-19, Quan TRAN wrote: > Hello, > > In worker.c: > >     - in apply_handle_insert, slot_store_cstrings is called before > PushActiveSnapshot >     - in apply_handle_update & apply_handle_delete, slot_store_cstrings is > called after PushActiveSnapshot > >     /* Process and store

Re: Add constraint in a Materialized View

2018-07-20 Thread Nico Williams
On Wed, Jul 18, 2018 at 09:28:19AM +0200, Kaye Ann Ignacio wrote: > I'm trying to add a foreign constraint in my local table to reference a > column in a materialized view, is it possible to alter this materialized > view by adding a primary key constraint? It's not, but I'm working on a patch

Re: Making "COPY partitioned_table FROM" faster

2018-07-20 Thread Karen Huddleston
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested This patch applied cleanly and worked as expected. Patch

Add constraint in a Materialized View

2018-07-20 Thread Kaye Ann Ignacio
Hi, I'm trying to add a foreign constraint in my local table to reference a column in a materialized view, is it possible to alter this materialized view by adding a primary key constraint? Thank you. -- Kaye Ann Ignacio, Programmer proceedit "the BPaaS company" kaye.igna...@proceedit.com +34

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-07-20 Thread Peter Geoghegan
On Thu, Jul 19, 2018 at 4:29 AM, Masahiko Sawada wrote: >> One area that might be worth investigating is retail index tuple >> deletion performed within the executor in the event of non-HOT >> updates. Maybe LP_REDIRECT could be repurposed to mean "ghost record", >> at least in unique index

Re: Negotiating the SCRAM channel binding type

2018-07-20 Thread Heikki Linnakangas
On 12/07/18 16:08, Michael Paquier wrote: On Thu, Jul 12, 2018 at 12:34:51PM +0300, Heikki Linnakangas wrote: Hmm. That is actually in a section called "Default Channel Binding". And the first paragraph says "A default channel binding type agreement process for all SASL application protocols

Re: Faster str to int conversion (was Table with large number of int columns, very slow COPY FROM)

2018-07-20 Thread Andres Freund
Hi, On 2018-07-20 08:27:34 -0400, Robert Haas wrote: > On Thu, Jul 19, 2018 at 4:32 PM, Andres Freund wrote: > >> 1. Why the error message changes? If there's a good reason, it should > >> be done as a separate commit, or at least well-documented in the > >> commit message. > > > > Because

Re: [bug fix] Produce a crash dump before main() on Windows

2018-07-20 Thread Robert Haas
On Wed, Jul 18, 2018 at 4:38 AM, Tsunakawa, Takayuki wrote: > IIRC, the pop up doesn't appear under Windows service. If you start the > database server with pg_ctl start on the command prompt, the pop up will > appear, which I think is not bad. Hmm. I think that might be bad. What makes you

Re: Expression errors with "FOR UPDATE" and postgres_fdw with partition wise join enabled.

2018-07-20 Thread Robert Haas
On Fri, Jul 20, 2018 at 8:38 AM, Robert Haas wrote: > This looks like a terrible design to me. If somebody in future > invents a new plan type that is not projection-capable, then this > could fail an assertion here and there won't be any simple fix. And > if you reach here and the child

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-20 Thread David Rowley
On 20 July 2018 at 21:44, Amit Langote wrote: > But I don't think the result of make_partition_pruneinfo itself has to be > List of PartitionedRelPruneInfo nested under PartitionPruneInfo. I gather > that each PartitionPruneInfo corresponds to each root partitioned table > and a

Re: Non-portable shell code in pg_upgrade tap tests

2018-07-20 Thread Tom Lane
Victor Wagner writes: > Tom Lane wrote: >> Please send a patch. Most of us do not have access to old shells > Here it goes. Previous letter was written before fixed tests were > completed, because this old machine is slow. Thanks. Will check on my own dinosaurs, and push if I don't find a

Re: [HACKERS] possible self-deadlock window after bad ProcessStartupPacket

2018-07-20 Thread Andres Freund
On 2018-07-20 11:53:32 +0300, Heikki Linnakangas wrote: > ISTM that no-one has any great ideas on what to do about the ereport() in > quickdie(). I think we actually have some decent ideas how to make that less problematic in a number of cases. Avoid translation (thereby avoiding direct malloc())

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-20 Thread Andres Freund
On 2018-07-20 12:13:19 +0530, Nikhil Sontakke wrote: > Hi Andres, > > > > So what if we, at the begin / end of cache miss handling, re-check if > > the to-be-decoded transaction is still in-progress (or has > > committed). And we throw an error if that happened. That error is then > > caught in

Re: Making "COPY partitioned_table FROM" faster

2018-07-20 Thread David Rowley
(This email seemed to only come to me and somehow missed the mailing list. Including the message here in its entirety) On 20 July 2018 at 13:26, Karen Huddleston wrote: > The following review has been posted through the commitfest application: > make installcheck-world: tested, passed >

Re: Non-portable shell code in pg_upgrade tap tests

2018-07-20 Thread Victor Wagner
On Fri, 20 Jul 2018 10:25:47 -0400 Tom Lane wrote: > Victor Wagner writes: > > I've discovered that in the branch REL_11_STABLE there is shell > > script src/bin/pg_upgrade/test.sh which doesn't work under Solaris > > 10. (it uses $(command) syntax with is not compatible with original > >

Re: Fw: Windows 10 got stuck with PostgreSQL at starting up. Adding delay lets it avoid.

2018-07-20 Thread Tom Lane
Yugo Nagata writes: > Recently, one of our clients reported a problem that Windows 10 sometime > (approximately once in 300 tries) hung up at OS starting up while PostgreSQL > 9.3.x service is starting up. My co-worker analyzed this and found that > PostgreSQL's auxiliary process and Windows'

Re: Non-portable shell code in pg_upgrade tap tests

2018-07-20 Thread Tom Lane
Victor Wagner writes: > I've discovered that in the branch REL_11_STABLE there is shell script > src/bin/pg_upgrade/test.sh which doesn't work under Solaris 10. > (it uses $(command) syntax with is not compatible with original > Solaris /bin/sh) OK ... > It is quite easy to replace $() syntax

Non-portable shell code in pg_upgrade tap tests

2018-07-20 Thread Victor Wagner
Collegues, I've discovered that in the branch REL_11_STABLE there is shell script src/bin/pg_upgrade/test.sh which doesn't work under Solaris 10. (it uses $(command) syntax with is not compatible with original Solaris /bin/sh) I was quite surprised that this problem goes unnoticed on big

Re: Expression errors with "FOR UPDATE" and postgres_fdw with partition wise join enabled.

2018-07-20 Thread Robert Haas
On Wed, Jul 18, 2018 at 8:00 AM, Etsuro Fujita wrote: > [ new patch ] + /* + * If the child plan is an Append or MergeAppend, the targetlists of its + * subplans would have already been adjusted before we get here, so need + * to work here. + */ + if (IsA(subplan, Append) || IsA(subplan,

Re: Faster str to int conversion (was Table with large number of int columns, very slow COPY FROM)

2018-07-20 Thread Robert Haas
On Thu, Jul 19, 2018 at 4:32 PM, Andres Freund wrote: >> 1. Why the error message changes? If there's a good reason, it should >> be done as a separate commit, or at least well-documented in the >> commit message. > > Because there's a lot of "invalid input syntax for type %s: \"%s\"", > error

Re: partition tree inspection functions

2018-07-20 Thread Jesper Pedersen
Hi Amit, On 07/19/2018 10:27 PM, Amit Langote wrote: On 2018/07/19 23:18, Jesper Pedersen wrote: I'm thinking about how to best use these functions to generate a graph that represents the partition hierarchy. What about renaming pg_partition_tree_tables() to pg_partition_children(), and have

Re: Flexible configuration for full-text search

2018-07-20 Thread Alexander Korotkov
Hi, Aleksandr! On Mon, Jul 9, 2018 at 10:26 AM Aleksandr Parfenov < a.parfe...@postgrespro.ru> wrote: > A new version of the patch in the attachment. There are no changes since > the last version except refreshing it to current HEAD. > I took a look at this patch. It applied cleanly, but

Re: Fw: Windows 10 got stuck with PostgreSQL at starting up. Adding delay lets it avoid.

2018-07-20 Thread Michael Paquier
On Fri, Jul 20, 2018 at 05:58:13PM +0900, Yugo Nagata wrote: > He reported this problem to pgsql-general list as below. Also, he created a > patch > to add a build-time option for adding 0.5 or 3.0 seconds delay after each sub > process starts. The attached is the same one. Our client

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-20 Thread Amit Langote
On 2018/07/19 22:03, David Rowley wrote: > v3-0001-Fix-run-time-partition-pruning-for-UNION-ALL-pare.patch Thanks for updating the patch. I studied this patch today and concluded that it's going a bit too far by carrying the nested partition pruning info structures from the planner all the way

Re: project updates

2018-07-20 Thread Aleksander Alekseeev
Hello Charles, > Here is my current working status. > 1. Complete the thrift_binary_in and thrift_binary_out functions, so > that users can express their thrift struct using json. These two > functions support both simple data struct and complex data structure > like struct and map. 2. added

Adding TCP_USER_TIMEOUT support for libpq/psqlodbc

2018-07-20 Thread AYahorau
Hello PostgreSQL Community! Not long ago I faced the situation concerning ODBC/libpq client hanging in case of some network problems. I had a discussion regarding this issue within pgsql-o...@postgresql.org and got some suggestions. Here is this discussion:

Fw: Windows 10 got stuck with PostgreSQL at starting up. Adding delay lets it avoid.

2018-07-20 Thread Yugo Nagata
Hi, Recently, one of our clients reported a problem that Windows 10 sometime (approximately once in 300 tries) hung up at OS starting up while PostgreSQL 9.3.x service is starting up. My co-worker analyzed this and found that PostgreSQL's auxiliary process and Windows' logon processes are in a

Re: de-deduplicate code in DML execution hooks in postgres_fdw

2018-07-20 Thread Etsuro Fujita
(2018/07/20 13:49), Michael Paquier wrote: On Thu, Jul 19, 2018 at 05:35:11PM +0900, Etsuro Fujita wrote: +1 for the general idea. (Actually, I also thought the same thing before.) But since this is definitely a matter of PG12, ISTM that it's wise to work on this after addressing the issue in

Re: [HACKERS] possible self-deadlock window after bad ProcessStartupPacket

2018-07-20 Thread Heikki Linnakangas
On 19/07/18 23:13, Andres Freund wrote: On 2018-07-19 14:54:52 -0500, Nico Williams wrote: On Thu, Jul 19, 2018 at 12:20:53PM -0700, Andres Freund wrote: On 2018-07-19 11:57:25 +0300, Heikki Linnakangas wrote: Ugh. Yeah, in wal_quickdie, and other aux process *_quickdie() handlers, I agree we

Re: [HACKERS] possible self-deadlock window after bad ProcessStartupPacket

2018-07-20 Thread Heikki Linnakangas
On 19/07/18 23:04, Nico Williams wrote: On Thu, Jun 22, 2017 at 03:10:31PM -0400, Tom Lane wrote: Andres Freund writes: Or, probably more robust: Simply _exit(2) without further ado, and rely on postmaster to output an appropriate error message. Arguably it's not actually useful to see

Re: pread() and pwrite()

2018-07-20 Thread Oskari Saarenmaa
On Thu, Jul 12, 2018 at 01:55:31PM +1200, Thomas Munro wrote: > A couple of years ago, Oskari Saarenmaa proposed a patch[1] to adopt > $SUBJECT. Last year, Tobias Oberstein argued again that we should do > that[2]. At the end of that thread there was a +1 from multiple > committers in support of

Re: pread() and pwrite()

2018-07-20 Thread Heikki Linnakangas
On 20/07/18 01:50, Thomas Munro wrote: An idea for how to handle Windows, in a follow-up patch: add a file src/backend/port/win32/file.c that defines pgwin32_pread() and pgwin32_pwrite(), wrapping WriteFile()/ReadFile() and passing in an "OVERLAPPED" struct with the offset and sets errno on

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-20 Thread Nikhil Sontakke
Hi Andres, > So what if we, at the begin / end of cache miss handling, re-check if > the to-be-decoded transaction is still in-progress (or has > committed). And we throw an error if that happened. That error is then > caught in reorderbuffer, the in-progress-xact aborted callback is > called,

Re: More consistency for some file-related error message

2018-07-20 Thread Michael Paquier
On Thu, Jul 19, 2018 at 12:33:30PM +0900, Michael Paquier wrote: > On Wed, Jul 18, 2018 at 11:24:05PM -0400, Tom Lane wrote: >> read() is required by spec to set errno when returning a negative result. >> I think the previous coding paid attention to errno regardless of the sign >> of the result,