Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Nico Williams
On Thu, Jun 21, 2018 at 10:05:41AM +0900, Masahiko Sawada wrote: > On Thu, Jun 21, 2018 at 6:57 AM, Nico Williams wrote: > > On Wed, Jun 20, 2018 at 05:16:46PM -0400, Bruce Momjian wrote: > >> On Mon, Jun 18, 2018 at 12:29:57PM -0500, Nico Williams wrote: > >> > Note that unless the pg_catalog is

Re: Speedup of relation deletes during recovery

2018-06-20 Thread Michael Paquier
On Wed, Jun 20, 2018 at 08:43:11PM -0700, Andres Freund wrote: > On 2018-06-18 11:13:47 -0700, Andres Freund wrote: >> We could do that - but add_to_unowned_list() is actually a bottleneck in >> other places during recovery too. We pretty much never (outside of >> dropping relations / databases)

Re: PATCH: backtraces for error messages

2018-06-20 Thread Michael Paquier
On Thu, Jun 21, 2018 at 12:35:10PM +0800, Craig Ringer wrote: > I wrote it because I got sick of Assert(false) debugging, and I was chasing > down some "ERROR: 08P01: insufficient data left in message" errors. Then I > got kind of caught up in it... you know how it is. Yes, I know that feeling!

Re: PATCH: backtraces for error messages

2018-06-20 Thread Craig Ringer
This is what the stacks look like btw [2018-06-21 12:26:45.309 AWST] [7293] [] [] [:0] DEBUG: 0: find_in_dynamic_libpath: trying "/home/craig/pg/10/lib/postgresql/pglogical.so" [2018-06-21 12:26:45.309 AWST] [7293] [] [] [:0] LOCATION: find_in_dynamic_libpath, dfmgr.c:639 [2018-06-21

Re: Speedup of relation deletes during recovery

2018-06-20 Thread Andres Freund
On 2018-06-18 11:13:47 -0700, Andres Freund wrote: > On 2018-06-19 03:06:54 +0900, Fujii Masao wrote: > > On Sat, Jun 16, 2018 at 3:28 AM, Andres Freund wrote: > > > Hi, > > > > > > On 2018-06-15 10:45:04 -0700, Andres Freund wrote: > > >> > + > > >> > + srels = palloc(sizeof(SMgrRelation) *

Re: PATCH: backtraces for error messages

2018-06-20 Thread Craig Ringer
On 21 June 2018 at 01:15, Andres Freund wrote: > On 2018-06-20 13:10:57 -0400, Robert Haas wrote: > > On Wed, Jun 20, 2018 at 12:10 PM, Andres Freund > wrote: > > > If we instead had a backtrace enabled for all PANICs and some FATALs by > > > default (and perhaps a SIGSEGV handler too), we'd be

Re: Speedup of relation deletes during recovery

2018-06-20 Thread Thomas Munro
On Tue, Jun 19, 2018 at 6:13 AM, Fujii Masao wrote: > On Sat, Jun 16, 2018 at 2:54 AM, Teodor Sigaev wrote: >>> We just had a customer hit this issue. I kind of wonder whether this >>> shouldn't be backpatched: Currently the execution on the primary is >>> O(NBuffers * log(ndrels)) whereas it's

Re: Keeping temporary tables in shared buffers

2018-06-20 Thread Amit Kapila
On Wed, Jun 20, 2018 at 8:47 PM, Bruce Momjian wrote: > On Sat, Jun 2, 2018 at 05:18:17PM -0400, Asim Praveen wrote: >> Hi Amit >> >> On Mon, May 28, 2018 at 4:25 AM, Amit Kapila wrote: >> > >> > This is one way, but I think there are other choices as well. We can >> > identify and flush all

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Masahiko Sawada
On Thu, Jun 21, 2018 at 6:57 AM, Nico Williams wrote: > On Wed, Jun 20, 2018 at 05:16:46PM -0400, Bruce Momjian wrote: >> On Mon, Jun 18, 2018 at 12:29:57PM -0500, Nico Williams wrote: >> > Note that unless the pg_catalog is protected against manipulation by >> > remote storage, then TDE for user

Re: Fast default stuff versus pg_upgrade

2018-06-20 Thread Andres Freund
Hi, On 2018-06-20 20:53:34 -0400, Andrew Dunstan wrote: > This version adds a lock on the table owning the attribute. Cool. > > /* > + * in binary upgrade mode, update the catalog with any missing > values > + * that might be present. > +

Re: Fast default stuff versus pg_upgrade

2018-06-20 Thread Andrew Dunstan
On 06/20/2018 01:46 PM, Andrew Dunstan wrote: Attached deals with all those issues except locking. This version adds a lock on the table owning the attribute. cheers andrew -- Andrew Dunstanhttps://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA,

RE: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Tsunakawa, Takayuki
From: Bruce Momjian [mailto:br...@momjian.us] > On Fri, May 25, 2018 at 08:41:46PM +0900, Moon, Insung wrote: > > BTW, I want to support CBC mode encryption[3]. However, I'm not sure how > to use the IV in CBC mode for this proposal. > > I'd like to hear opinions by security engineer. > > Well,

RE: PATCH: backtraces for error messages

2018-06-20 Thread Tsunakawa, Takayuki
From: Alvaro Herrera [mailto:alvhe...@2ndquadrant.com] > I have no idea how expensive backtrace() and libunwind are, but I doubt > we want to pay the cost for every message before we know that error > requires the backtrace to be collected. Something like PGC_USERSET >

Re: changing xpath() and xpath_exists()

2018-06-20 Thread Andrew Dunstan
On 06/20/2018 07:35 PM, Alvaro Herrera wrote: Hello Per discussion at https://postgr.es/m/0684a598-002c-42a2-ae12-f024a324e...@winand.at I intend to change xpath() and xpath_exists() in a subtly backwards- incompatible way, so that they match the behavior of SQL-standard- specified XMLTABLE.

Re: partition -> partitioned

2018-06-20 Thread Amit Langote
On 2018/06/21 0:45, Alvaro Herrera wrote: > On 2018-Jun-19, Amit Langote wrote: > >> Noticed that the relevant code changed, so I rebased the patch. Also, >> made a minor update to a nearby comment. > > Pushed, thanks. I made a couple of comments one or two words shorter > while (IMO) not

Re: Avoiding Tablespace path collision for primary and standby

2018-06-20 Thread Craig Ringer
On Thu., 21 Jun. 2018, 04:30 Tom Lane, wrote: > Ashwin Agrawal writes: > > Okay just bouncing another approach, how about generating UUID for a > > postgres instance during initdb and pg_basebackup ? > > There's no uuid generation code in core postgres, for excellent reasons > (lack of

changing xpath() and xpath_exists()

2018-06-20 Thread Alvaro Herrera
Hello Per discussion at https://postgr.es/m/0684a598-002c-42a2-ae12-f024a324e...@winand.at I intend to change xpath() and xpath_exists() in a subtly backwards- incompatible way, so that they match the behavior of SQL-standard- specified XMLTABLE. It seems sane to keep them in sync. Anybody

Re: Excessive CPU usage in StandbyReleaseLocks()

2018-06-20 Thread Andres Freund
Hi, On 2018-06-21 00:25:03 +1200, David Rowley wrote: > On 19 June 2018 at 17:43, Thomas Munro wrote: > > The problem is that StandbyReleaseLocks() does a linear search of all > > known AccessExclusiveLocks when a transaction ends. Luckily, since > > v10 (commit 9b013dc2) that is skipped for

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Nico Williams
On Wed, Jun 20, 2018 at 06:19:40PM -0400, Joe Conway wrote: > On 06/20/2018 05:12 PM, Bruce Momjian wrote: > > On Mon, Jun 18, 2018 at 11:06:20AM -0400, Joe Conway wrote: > > Even if they are encrypted with the same key, they use different > > initialization vectors that are stored inside the

Re: Excessive CPU usage in StandbyReleaseLocks()

2018-06-20 Thread Thomas Munro
Hi David, On Thu, Jun 21, 2018 at 12:25 AM, David Rowley wrote: > On 19 June 2018 at 17:43, Thomas Munro wrote: >> The problem is that StandbyReleaseLocks() does a linear search of all >> known AccessExclusiveLocks when a transaction ends. Luckily, since >> v10 (commit 9b013dc2) that is

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Joe Conway
On 06/20/2018 05:12 PM, Bruce Momjian wrote: > On Mon, Jun 18, 2018 at 11:06:20AM -0400, Joe Conway wrote: >>> At the same time, having to have a bunch of independently-decipherable >>> short field values is not real secure either, especially if they're known >>> to all be encrypted with the same

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Joe Conway
On 06/20/2018 05:03 PM, Bruce Momjian wrote: > On Wed, Jun 13, 2018 at 09:20:58AM -0400, Joe Conway wrote: >> The idea has not been extensively fleshed out yet, but the thought was >> that we create column level POLICY, which would transparently apply some >> kind of transform on input and/or

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Nico Williams
On Wed, Jun 20, 2018 at 06:06:56PM -0400, Joe Conway wrote: > On 06/20/2018 05:09 PM, Bruce Momjian wrote: > > On Mon, Jun 18, 2018 at 09:49:20AM -0400, Robert Haas wrote: > >> know the ordering of the values under whatever ordering semantics > >> apply to that index. It's unclear to me how

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Joe Conway
On 06/20/2018 05:05 PM, Bruce Momjian wrote: > On Mon, Jun 18, 2018 at 08:29:32AM -0400, Joe Conway wrote: Or maybe key management is really tied into the separately discussed effort to create SQL VARIABLEs somehow. >>> >>> Could you elaborate on how key management is tied into SQL

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Joe Conway
On 06/20/2018 05:09 PM, Bruce Momjian wrote: > On Mon, Jun 18, 2018 at 09:49:20AM -0400, Robert Haas wrote: >> know the ordering of the values under whatever ordering semantics >> apply to that index. It's unclear to me how useful such information > > I don't think an ordered index is possible,

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Nico Williams
On Wed, Jun 20, 2018 at 05:16:46PM -0400, Bruce Momjian wrote: > On Mon, Jun 18, 2018 at 12:29:57PM -0500, Nico Williams wrote: > > Note that unless the pg_catalog is protected against manipulation by > > remote storage, then TDE for user tables might be possible to > > compromise. Like so: the

Postgres 10.4 crashing when using PLV8

2018-06-20 Thread Mukesh Chhatani
Hello Team, I am trying to use the PLV8 via function and while using the function created via PLV8 in one of the create materialized view, postgres crashes, attached is the log file with DEBUG5 turned on. SQL which is breaking the code and SQL function is attached. Creating materialized view -

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Bruce Momjian
On Mon, Jun 18, 2018 at 12:29:57PM -0500, Nico Williams wrote: > On Mon, Jun 11, 2018 at 06:22:22PM +0900, Masahiko Sawada wrote: > > As per discussion at PGCon unconference, I think that firstly we need > > to discuss what threats we want to defend database data against. If > > We call that a

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Bruce Momjian
On Mon, Jun 18, 2018 at 11:06:20AM -0400, Joe Conway wrote: > > At the same time, having to have a bunch of independently-decipherable > > short field values is not real secure either, especially if they're known > > to all be encrypted with the same key. But what you know or can guess > > about

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Bruce Momjian
On Mon, Jun 18, 2018 at 09:49:20AM -0400, Robert Haas wrote: > figure stuff out. You can infer something about the length of > particular values. Perhaps you can find cases where the same > encrypted value appears multiple times. If there's a btree index, you Most encryption methods use a

Re: Push down Aggregates below joins

2018-06-20 Thread Tomas Vondra
On 06/20/2018 10:12 PM, Heikki Linnakangas wrote: > Currently, the planner always first decides the scan/join order, and > adds Group/Agg nodes on top of the joins. Sometimes it would be legal, > and beneficial, to perform the aggregation below a join. I've been > hacking on a patch to allow that.

Re: Push down Aggregates below joins

2018-06-20 Thread Tomas Vondra
On 06/20/2018 10:12 PM, Heikki Linnakangas wrote: > Currently, the planner always first decides the scan/join order, and > adds Group/Agg nodes on top of the joins. Sometimes it would be legal, > and beneficial, to perform the aggregation below a join. I've been > hacking on a patch to allow that.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Bruce Momjian
On Mon, Jun 18, 2018 at 08:29:32AM -0400, Joe Conway wrote: > >> Or > >> maybe key management is really tied into the separately discussed effort > >> to create SQL VARIABLEs somehow. > > > > Could you elaborate on how key management is tied into SQL VARIABLEs? > > Well, the key management

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Bruce Momjian
On Wed, Jun 13, 2018 at 09:20:58AM -0400, Joe Conway wrote: > On 06/11/2018 05:22 AM, Masahiko Sawada wrote: > > As per discussion at PGCon unconference, I think that firstly we need > > to discuss what threats we want to defend database data against. > > Exactly. While certainly there is demand

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Bruce Momjian
On Fri, May 25, 2018 at 08:41:46PM +0900, Moon, Insung wrote: > BTW, I want to support CBC mode encryption[3]. However, I'm not sure how to > use the IV in CBC mode for this proposal. > I'd like to hear opinions by security engineer. Well, CBC makes sense, and since AES uses a 16 byte block

Re: libpq compression

2018-06-20 Thread Robbie Harwood
Konstantin Knizhnik writes: > On 20.06.2018 00:04, Robbie Harwood wrote: >> Konstantin Knizhnik writes: >>> On 18.06.2018 23:34, Robbie Harwood wrote: >>> I also don't like that you've injected into the *startup* path - before authentication takes place. Fundamentally, authentication

Re: Avoiding Tablespace path collision for primary and standby

2018-06-20 Thread Tom Lane
Ashwin Agrawal writes: > Okay just bouncing another approach, how about generating UUID for a > postgres instance during initdb and pg_basebackup ? There's no uuid generation code in core postgres, for excellent reasons (lack of portability and lack of failure modes are the main objections).

Re: ERROR: ORDER/GROUP BY expression not found in targetlist

2018-06-20 Thread Tom Lane
I wrote: > Thanks for the report. I traced through this, and the problem seems to > be that split_pathtarget_at_srfs() is neglecting to attach sortgroupref > labeling to the extra PathTargets it constructs. I don't remember > whether that code is my fault or Andres', but I'll take a look at >

Re: Postgres 10.4 crashing when using PLV8

2018-06-20 Thread Mukesh Chhatani
Thanks David for the response, I have opened a issue with PLV8 team. Let me know if I should report this bug to postgres or not, since I was not sure thus I sent email earlier. Regards, Mukesh On Wed, Jun 20, 2018 at 3:02 PM, David G. Johnston < david.g.johns...@gmail.com> wrote: > On Wed, Jun

Re: Postgres 10.4 crashing when using PLV8

2018-06-20 Thread David G. Johnston
On Wed, Jun 20, 2018 at 12:46 PM, Mukesh Chhatani wrote: > I am trying to use the PLV8 via function and while using the function > created via PLV8 in one of the create materialized view, postgres crashes, > attached is the log file with DEBUG5 turned on. > ​These are not the correct place to

Re: [PATCH] Include application_name in "connection authorized" log message

2018-06-20 Thread Stephen Frost
Greetings, * Don Seiler (d...@seiler.us) wrote: > In trying to troubleshoot the source of a recent connection storm, I was > frustrated to find that the app name was not included in the connection > messages. It is there in the log_line_prefix on the disconnection messages > but I would prefer it

memory leak when serializing TRUNCATE in reorderbuffer

2018-06-20 Thread Tomas Vondra
Hi, While rebasing the logical replication patches on top of PG11, I've noticed that ReorderBufferSerializeChange claims this: case REORDER_BUFFER_CHANGE_TRUNCATE: ... /* ReorderBufferChange contains everything important */ That is not quite correct, though - the OIDs of

[PATCH] Include application_name in "connection authorized" log message

2018-06-20 Thread Don Seiler
Good afternoon, all. In trying to troubleshoot the source of a recent connection storm, I was frustrated to find that the app name was not included in the connection messages. It is there in the log_line_prefix on the disconnection messages but I would prefer it be directly visible with the

Re: Avoiding Tablespace path collision for primary and standby

2018-06-20 Thread Ashwin Agrawal
On Wed, Jun 20, 2018 at 10:50 AM Andres Freund wrote: > > > On June 20, 2018 10:31:05 AM PDT, Ashwin Agrawal > wrote: > >On Wed, Jun 20, 2018 at 9:39 AM Bruce Momjian wrote: > > > >> On Fri, May 25, 2018 at 02:17:23PM -0700, Ashwin Agrawal wrote: > >> > > >> > On Fri, May 25, 2018 at 7:33 AM,

Re: Allow auto_explain to log to NOTICE

2018-06-20 Thread Daniel Gustafsson
> On 27 Apr 2018, at 04:24, Andres Freund wrote: > > On 2018-04-27 11:52:18 +0930, Tom Dunstan wrote: >>> I'd argue this should contain the non-error cases. It's just as >>> reasonable to want to add this as a debug level or such. >> >> So all of warning, info, debug and debug1-5? > > Yea.

Re: [HACKERS] WIP: long transactions on hot standby feedback replica / proof of concept

2018-06-20 Thread Alexander Korotkov
Hi! Thank you for your work on this issue. On Wed, Feb 28, 2018 at 5:25 PM Ivan Kartyshov wrote: > Thank you for your valuable comments. I've made a few adjustments. > > The main goal of my changes is to let long read-only transactions run on > replica if hot_standby_feedback is turned on. > >

Re: PATCH: backtraces for error messages

2018-06-20 Thread Robert Haas
On Wed, Jun 20, 2018 at 1:15 PM, Andres Freund wrote: > I don't think that's ok. It's perfectly possible to hit > ERRCODE_INTERNAL_ERROR at a high frequency in some situations, Really? How? > and > there's plenty cases that aren't ERRCODE_INTERNAL_ERROR where we'd want > this. E.g. a lot of

Re: Avoiding Tablespace path collision for primary and standby

2018-06-20 Thread Andres Freund
On June 20, 2018 10:31:05 AM PDT, Ashwin Agrawal wrote: >On Wed, Jun 20, 2018 at 9:39 AM Bruce Momjian wrote: > >> On Fri, May 25, 2018 at 02:17:23PM -0700, Ashwin Agrawal wrote: >> > >> > On Fri, May 25, 2018 at 7:33 AM, Tom Lane >wrote: >> > >> > Ashwin Agrawal writes: >> > >

Re: Fast default stuff versus pg_upgrade

2018-06-20 Thread Andrew Dunstan
On 06/20/2018 12:58 PM, Andres Freund wrote: Hi, Just a quick scan-through review: On 2018-06-20 12:51:07 -0400, Andrew Dunstan wrote: diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index 0c54b02..2666ab2 100644 ---

Re: PATCH: backtraces for error messages

2018-06-20 Thread Daniel Gustafsson
> On 20 Jun 2018, at 17:10, Craig Ringer wrote: > BTW, Álvaro posted a simpler patch at > https://www.postgresql.org/message-id/20180410213203.nl645o5vj5ap66sl@alvherre.pgsql. > It uses backtrace() from glibc, and requires each site you want to bt to be > annotated explicitly. I forgot about

Re: Avoiding Tablespace path collision for primary and standby

2018-06-20 Thread Ashwin Agrawal
On Wed, Jun 20, 2018 at 9:39 AM Bruce Momjian wrote: > On Fri, May 25, 2018 at 02:17:23PM -0700, Ashwin Agrawal wrote: > > > > On Fri, May 25, 2018 at 7:33 AM, Tom Lane wrote: > > > > Ashwin Agrawal writes: > > > Proposing to create directory with timestamp at time of creating > >

Re: line numbers in error messages are off wrt debuggers

2018-06-20 Thread Tom Lane
Andres Freund writes: > Seems it can be hacked around: https://stackoverflow.com/a/13497879 > But yikes, that's ugly. I bet it's not very portable either. I'm not sure that all compilers will think that HANDLEARGS should be expanded in that situation. regards, tom lane

Re: line numbers in error messages are off wrt debuggers

2018-06-20 Thread Tom Lane
Andres Freund writes: > On 2018-06-20 12:12:26 -0400, Tom Lane wrote: >> Personally, my habit is to set the breakpoint at errfinish, which works >> independently of just where the call is. > I do that too, occasionally. Doesn't really if there's a lot of error > messages before the bug you're

Re: PATCH: backtraces for error messages

2018-06-20 Thread Andres Freund
On 2018-06-20 13:10:57 -0400, Robert Haas wrote: > On Wed, Jun 20, 2018 at 12:10 PM, Andres Freund wrote: > > If we instead had a backtrace enabled for all PANICs and some FATALs by > > default (and perhaps a SIGSEGV handler too), we'd be in a better > > place. That'd obviously only work when

Re: PATCH: backtraces for error messages

2018-06-20 Thread Robert Haas
On Wed, Jun 20, 2018 at 12:10 PM, Andres Freund wrote: > I think the problem is that this most frequently is an issue when > investigating an issue for customers. Often enough it'll legally not be > possible to gain direct access to the system, and remotely instructing > people to install an

Re: XML/XPath issues: text/CDATA in XMLTABLE, XPath evaluated with wrong context

2018-06-20 Thread Alvaro Herrera
Hi On 2018-Mar-27, Markus Winand wrote: > I’ve found two issues with XML/XPath integration in PostgreSQL. Two > patches are attached. I’ve just separated them because the second one > is an incompatible change. > > * Patch 0001: Accept TEXT and CDATA nodes in XMLTABLE’s column_expression. I

Re: Fast default stuff versus pg_upgrade

2018-06-20 Thread Andres Freund
Hi, Just a quick scan-through review: On 2018-06-20 12:51:07 -0400, Andrew Dunstan wrote: > diff --git a/src/backend/utils/adt/pg_upgrade_support.c > b/src/backend/utils/adt/pg_upgrade_support.c > index 0c54b02..2666ab2 100644 > --- a/src/backend/utils/adt/pg_upgrade_support.c > +++

Re: AtEOXact_ApplyLauncher() and subtransactions

2018-06-20 Thread Amit Khandekar
On 18 June 2018 at 15:02, Amit Khandekar wrote: > On 16 June 2018 at 00:03, Amit Khandekar wrote: >> The way I am implementing this can be seen in attached >> apply_launcher_subtrans_WIP.patch. (check launcher.c changes). I >> haven't started testing it yet though. > > Attached patch passes some

Re: Fast default stuff versus pg_upgrade

2018-06-20 Thread Andrew Dunstan
On 06/19/2018 10:46 PM, Andres Freund wrote: On 2018-06-19 22:41:26 -0400, Andrew Dunstan wrote: This unfortunately crashes and burns if we use DirectFunctionCall3 to call array_in, because it uses fn_extra. There is the CallerFInfoFunctionCall stuff, but it only has 1 and 2 arg variants, and

Re: Avoiding Tablespace path collision for primary and standby

2018-06-20 Thread Andres Freund
Hi, Hi, On 2018-05-26 10:08:57 -0400, Tom Lane wrote: > Not sure about the relative-path idea. Seems like that would create > a huge temptation to put tablespaces inside the data directory, which > would force us to deal with that can of worms. It doesn't seem impossible to normalize the path,

Re: line numbers in error messages are off wrt debuggers

2018-06-20 Thread Andres Freund
On 2018-06-20 12:12:26 -0400, Tom Lane wrote: > Andres Freund writes: > > I'm fairly frequently annoyed that when I see an error message in the > > log, I can't just generally set a breakpoint on the included line > > number. That's because the line number in the error message is from the > >

Re: Avoiding Tablespace path collision for primary and standby

2018-06-20 Thread Bruce Momjian
On Fri, May 25, 2018 at 02:17:23PM -0700, Ashwin Agrawal wrote: > > On Fri, May 25, 2018 at 7:33 AM, Tom Lane wrote: > > Ashwin Agrawal writes: > > Proposing to create directory with timestamp at time of creating > tablespace > > and create symbolic link to it instead. > >

Re: line numbers in error messages are off wrt debuggers

2018-06-20 Thread Andres Freund
Hi, On 2018-06-20 12:12:26 -0400, Tom Lane wrote: > Andres Freund writes: > > I'm fairly frequently annoyed that when I see an error message in the > > log, I can't just generally set a breakpoint on the included line > > number. That's because the line number in the error message is from the >

Re: pgsql: Store 2PC GID in commit/abort WAL recs for logical decoding

2018-06-20 Thread Alvaro Herrera
On 2018-Jun-15, Alvaro Herrera wrote: > By the way, why do we need to strlen() the target buffer when strlcpy > already reports the length? (You could argue that there is a difference > if the string is truncated ... but surely we don't care about that case) > I propose the attached. I decided

Re: line numbers in error messages are off wrt debuggers

2018-06-20 Thread Tom Lane
Andres Freund writes: > I'm fairly frequently annoyed that when I see an error message in the > log, I can't just generally set a breakpoint on the included line > number. That's because the line number in the error message is from the > *end* of the message: IME it varies depending on which

Re: PATCH: backtraces for error messages

2018-06-20 Thread Andres Freund
On 2018-06-20 12:04:51 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2018-06-20 11:20:49 -0400, Alvaro Herrera wrote: > >> I have no idea how expensive backtrace() and libunwind are, but I doubt > >> we want to pay the cost for every message before we know that error > >> requires the

Re: PATCH: backtraces for error messages

2018-06-20 Thread Tom Lane
Andres Freund writes: > On 2018-06-20 11:20:49 -0400, Alvaro Herrera wrote: >> I have no idea how expensive backtrace() and libunwind are, but I doubt >> we want to pay the cost for every message before we know that error >> requires the backtrace to be collected. Something like PGC_USERSET >>

line numbers in error messages are off wrt debuggers

2018-06-20 Thread Andres Freund
Hi, I'm fairly frequently annoyed that when I see an error message in the log, I can't just generally set a breakpoint on the included line number. That's because the line number in the error message is from the *end* of the message: 2018-06-20 09:02:39.226 PDT [21145][3/2] LOG: 0:

Re: ERROR: ORDER/GROUP BY expression not found in targetlist

2018-06-20 Thread Tom Lane
Rajkumar Raghuwanshi writes: > Below test case is failing with ERROR: ORDER/GROUP BY expression not found > in targetlist. this issue is reproducible from PGv10. Thanks for the report. I traced through this, and the problem seems to be that split_pathtarget_at_srfs() is neglecting to attach

Re: PATCH: backtraces for error messages

2018-06-20 Thread Andres Freund
Hi, On 2018-06-20 11:20:49 -0400, Alvaro Herrera wrote: > > I recently needed a way to get backtraces from errors in a convenient, > > non-interactive and indescriminate way. The attached patch is the result. > > It teaches Pg to use libunwind to self-backtrace in elog/ereport. > > > > Anyone

Re: found xmin from before relfrozenxid on pg_catalog.pg_authid

2018-06-20 Thread Andres Freund
Hi, On 2018-06-20 15:05:59 +0300, Sergey Burladyan wrote: > Andres Freund writes: > > > You should first make sure it's actually this problem - which tables are > > holding back the xmin horizon? After that, yes, deleting the > > global/pg_internal.init file is the way to go, and I can't think

Re: PATCH: backtraces for error messages

2018-06-20 Thread Alexander Kuzmenkov
On 06/20/2018 06:10 PM, Craig Ringer wrote: I recently needed a way to get backtraces from errors in a convenient, non-interactive and indescriminate way. The attached patch is the result. It teaches Pg to use libunwind to self-backtrace in elog/ereport. Anyone think this is

Re: partition -> partitioned

2018-06-20 Thread Alvaro Herrera
On 2018-Jun-19, Amit Langote wrote: > Noticed that the relevant code changed, so I rebased the patch. Also, > made a minor update to a nearby comment. Pushed, thanks. I made a couple of comments one or two words shorter while (IMO) not losing clarity. -- Álvaro Herrera

Re: PATCH: backtraces for error messages

2018-06-20 Thread Alvaro Herrera
On 2018-Jun-20, Craig Ringer wrote: > Hi folks > > I recently needed a way to get backtraces from errors in a convenient, > non-interactive and indescriminate way. The attached patch is the result. > It teaches Pg to use libunwind to self-backtrace in elog/ereport. > > Anyone think this is

Re: Keeping temporary tables in shared buffers

2018-06-20 Thread Bruce Momjian
On Sat, Jun 2, 2018 at 05:18:17PM -0400, Asim Praveen wrote: > Hi Amit > > On Mon, May 28, 2018 at 4:25 AM, Amit Kapila wrote: > > > > This is one way, but I think there are other choices as well. We can > > identify and flush all the dirty (local) buffers for the relation > > being accessed

Re: add default parallel query to v10 release notes? (Re: [PERFORM] performance drop after upgrade (9.6 > 10))

2018-06-20 Thread Bruce Momjian
On Thu, May 24, 2018 at 08:00:25PM -0500, Justin Pryzby wrote: > Moving to -hackers; > > On Sun, Jan 28, 2018 at 06:53:10PM -0500, Bruce Momjian wrote: > > On Thu, Oct 26, 2017 at 02:45:15PM -0500, Justin Pryzby wrote: > > > Is it because max_parallel_workers_per_gather now defaults to 2 ? > > >

PATCH: backtraces for error messages

2018-06-20 Thread Craig Ringer
Hi folks I recently needed a way to get backtraces from errors in a convenient, non-interactive and indescriminate way. The attached patch is the result. It teaches Pg to use libunwind to self-backtrace in elog/ereport. Anyone think this is useful/interesting/worth pursuing? (Patch is currently

Re: New function pg_stat_statements_reset_query() to reset statistics of a specific query

2018-06-20 Thread Euler Taveira
2018-06-20 4:30 GMT-03:00 Haribabu Kommi : > Attached is a simple patch with implementation. Comments? > Why don't you extend the existing function pg_stat_statements_reset()? -- Euler Taveira Timbira - http://www.timbira.com.br/ PostgreSQL: Consultoria,

Re: Loaded footgun open_datasync on Windows

2018-06-20 Thread Laurenz Albe
Kuntal Ghosh wrote: [pg_test_fsync doesn't use pgwin32_open and hence doesn't respect O_DSYNC] > On Wed, Jun 6, 2018 at 2:39 PM, Amit Kapila wrote: > > On Wed, Jun 6, 2018 at 10:18 AM, Michael Paquier > > wrote: > > > On Wed, Jun 06, 2018 at 09:58:34AM +0530, Amit Kapila wrote: > > > > One

Re: documentation is now XML

2018-06-20 Thread Tatsuo Ishii
>> It's unfortunate that we'll have to deal with different formats in the >> supported branches for several more years, but we at Postgres Professional >> are >> ready to accept any your decision on this matter for now. > > I had not thought of translations, but wouldn't all translations also >

Re: documentation is now XML

2018-06-20 Thread Bruce Momjian
On Wed, Jun 20, 2018 at 04:46:24PM +0300, Alexander Lakhin wrote: > 20.06.2018 15:03, Bruce Momjian wrote: > > On Wed, Jun 20, 2018 at 10:48:47AM +0300, Liudmila Mantrova wrote: > > It's unfortunate that we'll have to deal with different formats in the > supported branches

Re: documentation is now XML

2018-06-20 Thread Alexander Lakhin
20.06.2018 15:03, Bruce Momjian wrote: > On Wed, Jun 20, 2018 at 10:48:47AM +0300, Liudmila Mantrova wrote: >> It's unfortunate that we'll have to deal with different formats in the >> supported branches for several more years, but we at Postgres Professional >> are >> ready to accept any your

Re: Server crashed with TRAP: FailedAssertion("!(parallel_workers > 0)" when partitionwise_aggregate true.

2018-06-20 Thread Ashutosh Bapat
On Tue, Jun 19, 2018 at 2:13 PM, Jeevan Chalke wrote: > > > In the reported testcase, parallel_workers is set to 0 for all partition > (child) relations. Which means partial parallel paths are not possible for > child rels. However, the parent can have partial parallel paths. Thus, when > we have

Re: Server crashed with TRAP: FailedAssertion("!(parallel_workers > 0)" when partitionwise_aggregate true.

2018-06-20 Thread Jeevan Chalke
On Tue, Jun 19, 2018 at 7:14 PM, Jeevan Chalke < jeevan.cha...@enterprisedb.com> wrote: > > In the reported testcase, parallel_workers is set to 0 for all partition >> (child) relations. Which means partial parallel paths are not possible for >> child rels. However, the parent can have partial

Re: Concurrency bug in UPDATE of partition-key

2018-06-20 Thread Amit Kapila
On Tue, Jun 19, 2018 at 9:33 PM, Amit Khandekar wrote: > > Could not add RAISE statement, because the isolation test does not > seem to print those statements in the output. Instead, I have dumped > some rows in a new table through the trigger function, and did a > select from that table.

Re: Adding tests for inheritance trees with temporary tables

2018-06-20 Thread Ashutosh Bapat
On Wed, Jun 20, 2018 at 11:39 AM, Michael Paquier wrote: > > Good point, I have added those. Except that you meant > tableoid::regclass. Thanks. I have checked that make check passes with this patch. I have marked this entry as ready for committer. -- Best Wishes, Ashutosh Bapat

Re: Fix some error handling for read() and errno

2018-06-20 Thread Magnus Hagander
On Mon, Jun 18, 2018 at 6:42 AM, Michael Paquier wrote: > On Thu, Jun 14, 2018 at 09:50:33AM -0400, Robert Haas wrote: > > On Mon, Jun 11, 2018 at 6:11 PM, Alvaro Herrera > > wrote: > >> I would go as far as suggesting to remove qualifiers that indicate what > >> the file is for (such as

Re: pg_verify_checksums review

2018-06-20 Thread Magnus Hagander
On Tue, Jun 19, 2018 at 10:25 AM, Daniel Gustafsson wrote: > In looking over pg_verify_checksums I found a few small things that I think > would improve on it: > > * pg_verify_checksums was placed in the Client Utils section in the docs. > Since it requries physical access to the cluster

Re: Excessive CPU usage in StandbyReleaseLocks()

2018-06-20 Thread David Rowley
On 19 June 2018 at 17:43, Thomas Munro wrote: > The problem is that StandbyReleaseLocks() does a linear search of all > known AccessExclusiveLocks when a transaction ends. Luckily, since > v10 (commit 9b013dc2) that is skipped for transactions that haven't > taken any AELs and aren't using 2PC,

Re: documentation is now XML

2018-06-20 Thread Bruce Momjian
On Wed, Jun 20, 2018 at 10:48:47AM +0300, Liudmila Mantrova wrote: > To complete the picture of possible issues with older branches in XML, we > posted a question in packager lists some time ago and didn't receive any > objections. Just to keep record of all related questions in one place, here's

Re: Excessive CPU usage in StandbyReleaseLocks()

2018-06-20 Thread Thomas Munro
Adding Simon and David R. On Wed, Jun 20, 2018 at 5:44 AM, Tom Lane wrote: > Hence, I have a modest proposal: use elog(LOG) followed by Assert(false), > so that debug builds report such problems loudly but production builds > do not. We've done that before, see e.g. load_relcache_init_file

Re: [HACKERS] GUC for cleanup indexes threshold.

2018-06-20 Thread Alexander Korotkov
On Wed, Jun 20, 2018 at 8:32 AM Masahiko Sawada wrote: > On Wed, Jun 20, 2018 at 1:00 AM, Alexander Korotkov > > Ok. I've rephrased comment a bit. Also, you created "index vacuum" > > subsection in the "resource usage" section. I think it's not > > appropriate for this option to be in

Re: server crashed with TRAP: FailedAssertion("!(!parallel_aware || pathnode->path.parallel_safe)"

2018-06-20 Thread Amit Khandekar
On 20 June 2018 at 14:24, Amit Kapila wrote: > On Mon, Jun 18, 2018 at 3:11 PM, Amit Khandekar > wrote: >> On 16 June 2018 at 10:44, Amit Kapila wrote: >>> On Thu, Jun 14, 2018 at 10:05 PM, Tom Lane wrote: It looks to me like traversal of the partial subpaths is the right thing

Re: server crashed with TRAP: FailedAssertion("!(!parallel_aware || pathnode->path.parallel_safe)"

2018-06-20 Thread Amit Kapila
On Mon, Jun 18, 2018 at 3:11 PM, Amit Khandekar wrote: > On 16 June 2018 at 10:44, Amit Kapila wrote: >> On Thu, Jun 14, 2018 at 10:05 PM, Tom Lane wrote: >>> >>> It looks to me like traversal of the partial subpaths is the right >>> thing here, in which case we should do >>> >>> -

Re: server crashed with TRAP: FailedAssertion("!(!parallel_aware || pathnode->path.parallel_safe)"

2018-06-20 Thread Amit Kapila
On Tue, Jun 19, 2018 at 7:45 PM, Robert Haas wrote: > On Mon, Jun 18, 2018 at 11:36 PM, Amit Kapila wrote: >> Fixed in the attached patch. I will wait for a day or two to see if >> Tom or Robert wants to say something and then commit. > > The patch LGTM but the commit message could perhaps use

Re: ToDo: show size of partitioned table

2018-06-20 Thread Pavel Stehule
2018-06-20 10:03 GMT+02:00 Amit Langote : > On 2018/06/20 16:50, Pavel Stehule wrote: > > 2018-06-20 9:44 GMT+02:00 Amit Langote : > >> Do you mean \dt continues to show size 0 for partitioned tables, but > with > >> the new option (\dtP+) shows the actual size by aggregating across > >>

Re: ToDo: show size of partitioned table

2018-06-20 Thread Amit Langote
On 2018/06/20 16:50, Pavel Stehule wrote: > 2018-06-20 9:44 GMT+02:00 Amit Langote : >> Do you mean \dt continues to show size 0 for partitioned tables, but with >> the new option (\dtP+) shows the actual size by aggregating across >> partitions? +1 to such a feature, but we need to agree on an

Re: New function pg_stat_statements_reset_query() to reset statistics of a specific query

2018-06-20 Thread Sergei Kornilov
Hello > key.userid = GetUserId(); We can remove query id only from current user, right? Maybe better provide optional argument for userid? Typically user with pg_read_all_stats and user for application queries are different users. At least it should be documented. regards, Sergei

Re: Possible bug in logical replication.

2018-06-20 Thread Arseny Sher
Michael Paquier writes: > On Mon, Jun 18, 2018 at 09:42:36PM +0900, Michael Paquier wrote: >> On Fri, Jun 15, 2018 at 06:27:56PM +0300, Arseny Sher wrote: >> It seems to me that we still want to have the slot forwarding finish in >> this case even if this is interrupted. Petr, isn't that the

ERROR: ORDER/GROUP BY expression not found in targetlist

2018-06-20 Thread Rajkumar Raghuwanshi
Hi, Below test case is failing with ERROR: ORDER/GROUP BY expression not found in targetlist. this issue is reproducible from PGv10. postgres=# CREATE TABLE test(c1 int, c2 text, c3 text); CREATE TABLE postgres=# INSERT INTO test SELECT i % 10, i % 250, to_char(i % 4, 'FM000') FROM

  1   2   >