Re: [HACKERS] Add regression tests for autocommit-off mode for psql and fix some omissions

2014-10-07 Thread Feike Steenbergen
Apologies for the previous message, I didn't send the full version. On 6 October 2014 16:01, Tom Lane t...@sss.pgh.pa.us wrote: What class of bug would that prevent exactly? ERROR: [...] cannot run inside a transaction block when: - running psql in AUTOCOMMIT off - not having started a

Re: [HACKERS] Add regression tests for autocommit-off mode for psql and fix some omissions

2014-10-07 Thread Marko Tiikkaja
On 10/7/14, 9:11 AM, Feike Steenbergen wrote: Perhaps I am the only one using autocommit-off mode You most definitely aren't. and we shouldn't put effort into fixing this? It's not clear to me that this is fixing a problem, to be honest. If you're running autocommit=off, you have an

Re: [HACKERS] Feasibility of supporting bind params for all command types

2014-10-07 Thread Craig Ringer
On 10/06/2014 10:13 AM, Tom Lane wrote: I think it might be desirable but it'd be a mess, both as to the concept/definition and as to the implementation. Thanks Tom. The issues around ALTER etc pretty much put it in the not-worth-caring-about bucket. The issues around parameter typing alone...

[HACKERS] RLS - permissive vs restrictive

2014-10-07 Thread Thom Brown
Hi, It appears that I'm not the only person who finds it somewhat unintuitive for overlapping RLS policies to be permissive rather than restrictive (OR vs AND) (at least 3 others seem to expect AND behaviour), although I understand the reasoning behind it. And I've since discovered that the same

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-07 Thread Marti Raudsepp
On Thu, Sep 4, 2014 at 12:13 AM, Peter Geoghegan p...@heroku.com wrote: On Wed, Sep 3, 2014 at 9:51 AM, Robert Haas robertmh...@gmail.com wrote: INSERT INTO upsert(key, val) VALUES(1, 'insert') ON CONFLICT WITHIN upsert_pkey UPDATE SET val = 'update'; It seems to me that it would be better to

[HACKERS] Lets delete src/test/performance

2014-10-07 Thread Andres Freund
Hi, The code in there doesn't look very interesting - and very unlikely to run these days. Notably it relies on a binary called 'postmaster' et al... The last realy changes are from a long time ago: commit 142d42f9386ed81a4f0779ec8a0cad1254173b5e Author: Vadim B. Mikheev vadi...@yahoo.com Date:

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-07 Thread Bruce Momjian
On Fri, Oct 3, 2014 at 06:06:24PM -0400, Bruce Momjian wrote: I actually don't think that's true. Every lock acquiration implies a number of atomic locks. Those are expensive. And if you see individual locks acquired a high number of times in multiple proceses that's something important.

Re: [HACKERS] RLS - permissive vs restrictive

2014-10-07 Thread Robert Haas
On Tue, Oct 7, 2014 at 6:44 AM, Thom Brown t...@linux.com wrote: It appears that I'm not the only person who finds it somewhat unintuitive for overlapping RLS policies to be permissive rather than restrictive (OR vs AND) (at least 3 others seem to expect AND behaviour), although I understand

Re: [HACKERS] Promise index tuples for UPSERT

2014-10-07 Thread Simon Riggs
On 7 October 2014 03:31, Peter Geoghegan p...@heroku.com wrote: It may be that people on reading this now believe Peter's HW locking approach is the best. I'm happy to go with consensus. I bet you didn't think that you'd say that a week ago. :-) You're right, because last week I thought

[HACKERS] pg_upgrade, locale and encoding

2014-10-07 Thread Heikki Linnakangas
While looking at bug #11431, I noticed that pg_upgrade still seems to think that encoding and locale are cluster-wide properties. We got per-database locale support in 8.4, and encoding has been per-database much longer than that. pg_upgrade checks the encoding and locale of template0 in both

Re: [HACKERS] Promise index tuples for UPSERT

2014-10-07 Thread Robert Haas
On Tue, Oct 7, 2014 at 8:33 AM, Simon Riggs si...@2ndquadrant.com wrote: On 7 October 2014 03:31, Peter Geoghegan p...@heroku.com wrote: It may be that people on reading this now believe Peter's HW locking approach is the best. I'm happy to go with consensus. I bet you didn't think that you'd

Re: [HACKERS] Add regression tests for autocommit-off mode for psql and fix some omissions

2014-10-07 Thread Feike Steenbergen
On 7 October 2014 09:55, Marko Tiikkaja ma...@joh.to wrote: It's not clear to me that this is fixing a problem, to be honest. If you're running autocommit=off, you have an expectation that you can roll back commands at will. It's fine if I can't roll back a VACUUM, for example, since I would

Re: [HACKERS] TAP test breakage on MacOS X

2014-10-07 Thread Andrew Dunstan
On 10/07/2014 12:15 AM, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Oct 6, 2014 at 8:15 PM, Peter Eisentraut pete...@gmx.net wrote: The TAP tests are arguably already much easier to debug than pg_regress ever was. Well, maybe. I wasn't able, after about 5 minutes of

Re: [HACKERS] RLS - permissive vs restrictive

2014-10-07 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: On Tue, Oct 7, 2014 at 6:44 AM, Thom Brown t...@linux.com wrote: It appears that I'm not the only person who finds it somewhat unintuitive for overlapping RLS policies to be permissive rather than restrictive (OR vs AND) (at least 3 others seem

Re: [HACKERS] BRIN indexes - TRAP: BadArgument

2014-10-07 Thread Heikki Linnakangas
On 10/07/2014 01:33 AM, Alvaro Herrera wrote: Heikki Linnakangas wrote: On 09/23/2014 10:04 PM, Alvaro Herrera wrote: + Open questions + -- + + * Same-size page ranges? + Current related literature seems to consider that each index entry in a + BRIN index must cover the same

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-07 Thread Robert Haas
On Tue, Oct 7, 2014 at 8:03 AM, Bruce Momjian br...@momjian.us wrote: On Fri, Oct 3, 2014 at 06:06:24PM -0400, Bruce Momjian wrote: I actually don't think that's true. Every lock acquiration implies a number of atomic locks. Those are expensive. And if you see individual locks acquired a

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-07 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I think the easiest way to measure lwlock contention would be to put some counters in the lwlock itself. My guess, based on a lot of fiddling with LWLOCK_STATS over the years, is that there's no way to count lock acquisitions and releases without

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-07 Thread Andres Freund
On 2014-10-07 10:04:38 -0400, Robert Haas wrote: On Tue, Oct 7, 2014 at 8:03 AM, Bruce Momjian br...@momjian.us wrote: On Fri, Oct 3, 2014 at 06:06:24PM -0400, Bruce Momjian wrote: I actually don't think that's true. Every lock acquiration implies a number of atomic locks. Those are

Re: [HACKERS] Promise index tuples for UPSERT

2014-10-07 Thread Simon Riggs
On 7 October 2014 14:06, Robert Haas robertmh...@gmail.com wrote: Is there a way of detecting that we are updating a unique constraint column and then applying the HW locking only in that case? Or can we only apply locking when we have multiple unique constraints on a table? If so, I would

Re: [HACKERS] RLS - permissive vs restrictive

2014-10-07 Thread Robert Haas
On Tue, Oct 7, 2014 at 9:55 AM, Stephen Frost sfr...@snowman.net wrote: ... or you could just only have one policy on the table and do whatever you'd like with it (which was the original idea, actually, though I've found myself very much liking the ability to have multiple policies, and to

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-07 Thread Heikki Linnakangas
On 10/07/2014 05:04 PM, Robert Haas wrote: On Tue, Oct 7, 2014 at 8:03 AM, Bruce Momjian br...@momjian.us wrote: On Fri, Oct 3, 2014 at 06:06:24PM -0400, Bruce Momjian wrote: I actually don't think that's true. Every lock acquiration implies a number of atomic locks. Those are expensive. And

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-07 Thread Andres Freund
On 2014-10-07 17:22:18 +0300, Heikki Linnakangas wrote: FWIW, I liked Ilya's design. Before going to sleep, store the lock ID in shared memory. When you wake up, clear it. That should be cheap enough to have it always enabled. And it can easily be extended to other waits, e.g. when you're

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-07 Thread Robert Haas
On Tue, Oct 7, 2014 at 10:12 AM, Andres Freund and...@2ndquadrant.com wrote: Have you tried/considered putting the counters into a per-backend array somewhere in shared memory? That way they don't blow up the size of frequently ping-ponged cachelines. Then you can summarize those values

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-07 Thread Andres Freund
On 2014-10-07 10:30:54 -0400, Robert Haas wrote: On Tue, Oct 7, 2014 at 10:12 AM, Andres Freund and...@2ndquadrant.com wrote: Have you tried/considered putting the counters into a per-backend array somewhere in shared memory? That way they don't blow up the size of frequently ping-ponged

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-07 Thread Ilya Kosmodemiansky
On Tue, Oct 7, 2014 at 4:12 PM, Andres Freund and...@2ndquadrant.com wrote: I think the easiest way to measure lwlock contention would be to put some counters in the lwlock itself. My guess, based on a lot of fiddling with LWLOCK_STATS over the years, is that there's no way to count lock

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-07 Thread Robert Haas
On Tue, Oct 7, 2014 at 10:36 AM, Andres Freund and...@2ndquadrant.com wrote: That gets painful in a hurry. We just got rid of something like that with your patch to get rid of all the backend-local buffer pin arrays; I'm not keen to add another such thing right back. I think it might be ok

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-07 Thread Ilya Kosmodemiansky
On Tue, Oct 7, 2014 at 4:30 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-07 17:22:18 +0300, Heikki Linnakangas wrote: FWIW, I liked Ilya's design. Before going to sleep, store the lock ID in shared memory. When you wake up, clear it. That should be cheap enough to have it always

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-07 Thread Andres Freund
On 2014-10-07 10:45:24 -0400, Robert Haas wrote: It's not like it'd be significantly different today - in a read mostly workload that's bottlenecked on ProcArrayLock you'll not see many waits. There you'd have to count the total number of spinlocks cycles to measure anything interesting.

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2014-10-07 Thread Simon Riggs
On 31 July 2014 22:34, Stephen Frost sfr...@snowman.net wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: Stephen Frost sfr...@snowman.net writes: * Bruce Momjian (br...@momjian.us) wrote: Actually, thinking more, Stephen Frost mentioned that the auditing system has to modify database _state_,

Re: [HACKERS] RLS - permissive vs restrictive

2014-10-07 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: The key point from my angle is that if you grant user alice the right to see records where a = 1 and user bob the right to see records where a = 2, the multiple-policy approach allows those quals to be implemented as index-scans. If you had a

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2014-10-07 Thread Stephen Frost
Simon, * Simon Riggs (si...@2ndquadrant.com) wrote: On 31 July 2014 22:34, Stephen Frost sfr...@snowman.net wrote: There was a pretty good thread regarding reloptions and making it so extensions could use them which seemed to end up with a proposal to turn 'security labels' into a more

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2014-10-07 Thread Robert Haas
On Tue, Oct 7, 2014 at 12:24 PM, Simon Riggs si...@2ndquadrant.com wrote: I spoke with Robert about a year ago that the patch he was most proud of was the reloptions abstraction. Whatever we do in the future, keeping metadata in a slightly more abstract form is very useful. To slightly correct

Re: [HACKERS] Proposal for better support of time-varying timezone abbreviations

2014-10-07 Thread Jim Nasby
On 10/6/14, 6:19 PM, Jim Nasby wrote: FWIW, I agree for timestamptz, but I do wish we had a timestamp datatype that stored the exact timezone in effect when the data was entered. That can really, REALLY save your rear if you screw up either timezone in postgresql.conf, or the server's

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2014-10-07 Thread Fabrízio de Royes Mello
On Tue, Oct 7, 2014 at 1:24 PM, Simon Riggs si...@2ndquadrant.com wrote: On 31 July 2014 22:34, Stephen Frost sfr...@snowman.net wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: Stephen Frost sfr...@snowman.net writes: * Bruce Momjian (br...@momjian.us) wrote: Actually, thinking more,

Re: [HACKERS] OCLASS_ROWSECURITY oversights, and other kvetching

2014-10-07 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: The RLS patch added OCLASS_ROWSECURITY but it seems that not enough effort was made to grep for places that might require adjustment as a result. In objectaddress.c, getObjectDescription() was updated, but getObjectTypeDescription() and

Re: [HACKERS] TAP test breakage on MacOS X

2014-10-07 Thread Andrew Dunstan
On 10/07/2014 09:53 AM, Andrew Dunstan wrote: On 10/07/2014 12:15 AM, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Oct 6, 2014 at 8:15 PM, Peter Eisentraut pete...@gmx.net wrote: The TAP tests are arguably already much easier to debug than pg_regress ever was. Well,

[HACKERS] GIN pageinspect functions

2014-10-07 Thread Heikki Linnakangas
Some time ago, when debugging a GIN bug, I wrote these pageinspect functions to inspect GIN indexes. They were very useful; we should add them. - Heikki From 91ef58aab11e9077ab6a38268a1120806e42f2dd Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas heikki.linnakan...@iki.fi Date: Fri, 12 Sep

Re: [HACKERS] GIN pageinspect functions

2014-10-07 Thread Oleg Bartunov
On Tue, Oct 7, 2014 at 9:03 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Some time ago, when debugging a GIN bug, I wrote these pageinspect functions to inspect GIN indexes. They were very useful; we should add them. May be we can merge it with contrib/gevel, which we use many years

Re: [HACKERS] PL/pgSQL 2

2014-10-07 Thread Steven Lembark
On Mon, 1 Sep 2014 15:19:41 +0200 Joel Jacobson j...@trustly.com wrote: The fatal problems with Python3 and Perl6 was the inability to mix code between Python2/3 and Perl5/6. We don't have that problem with pl-languages in postgres, so please don't make that comparison, as it's incorrect.

Re: [HACKERS] GIN pageinspect functions

2014-10-07 Thread Heikki Linnakangas
On 10/07/2014 08:36 PM, Oleg Bartunov wrote: On Tue, Oct 7, 2014 at 9:03 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Some time ago, when debugging a GIN bug, I wrote these pageinspect functions to inspect GIN indexes. They were very useful; we should add them. May be we can merge

Re: [HACKERS] TAP test breakage on MacOS X

2014-10-07 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: I don't much like the idea of doing an install/initdb/start for every directory in src/bin, though. Can't we at least manage a single installation directory for all these? Peter had a patch to eliminate the overhead of multiple subinstalls; not sure

Re: [HACKERS] PL/pgSQL 2

2014-10-07 Thread Rodolfo Campero
2014-09-04 18:29 GMT-03:00 Robert Haas robertmh...@gmail.com: On Thu, Sep 4, 2014 at 2:31 PM, Josh Berkus j...@agliodbs.com wrote: Sadly, what's prevented us from having packages already has been the insistence of potential feature sponsors that they work *exactly* like PL/SQL's packages,

Re: [HACKERS] PL/pgSQL 2

2014-10-07 Thread Merlin Moncure
On Tue, Oct 7, 2014 at 12:42 PM, Steven Lembark lemb...@wrkhors.com wrote: On Mon, 1 Sep 2014 15:19:41 +0200 Joel Jacobson j...@trustly.com wrote: The fatal problems with Python3 and Perl6 was the inability to mix code between Python2/3 and Perl5/6. We don't have that problem with

Re: [HACKERS] Promise index tuples for UPSERT

2014-10-07 Thread Peter Geoghegan
On Tue, Oct 7, 2014 at 6:06 AM, Robert Haas robertmh...@gmail.com wrote: I'm not up on the details of what Peter's patch does with heavyweight locking, but I'd say it this way: if the patch uses heavyweight locking routinely, that's probably not going to scale well[1]. If the patch detects

Re: [HACKERS] lwlock contention with SSI

2014-10-07 Thread Robert Haas
On Tue, Oct 7, 2014 at 2:40 PM, Kevin Grittner kgri...@ymail.com wrote: Robert Haas robertmh...@gmail.com wrote: About a month ago, I told Kevin Grittner in an off-list conversation that I'd work on providing him with some statistics about lwlock contention under SSI. I then ran a benchmark

Re: [HACKERS] Add regression tests for autocommit-off mode for psql and fix some omissions

2014-10-07 Thread Jim Nasby
On 10/7/14, 2:11 AM, Feike Steenbergen wrote: On 7 October 2014 01:41, Jim Nasbyjim.na...@bluetreble.com wrote: The options I see... 1) If there's a definitive way to tell from backend source code what commands disallow transactions then we can just use that information to generate the list

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-07 Thread Peter Geoghegan
On Tue, Oct 7, 2014 at 5:23 AM, Simon Riggs si...@2ndquadrant.com wrote: IIRC it wasn't agreed that we needed to identify which indexes in the upsert SQL statement itself, since this would be possible in other ways and would require programmers to know which unique constraints are declared.

Re: [HACKERS] PL/pgSQL 2

2014-10-07 Thread Jim Nasby
On 10/7/14, 1:08 PM, Rodolfo Campero wrote: If it were possible to mark a function as private for its extension that would be awesome (the opposite would work too, i.e. a way to specify a public API, meaning the rest is private). For big extensions it's not clear which functions can be used

Re: [HACKERS] pg_dump refactor patch to remove global variables

2014-10-07 Thread Alvaro Herrera
Here's a rebased patch for this (I also pgindented it). One thing I find a bit odd is the fact that we copy RestoreOptions-superuser into DumpOptions-outputSuperuser (a char * pointer) without pstrdup or similar. We're already doing the inverse elsewhere, and the uses of the routine where this

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2014-10-07 Thread Josh Berkus
On 10/07/2014 09:44 AM, Fabrízio de Royes Mello wrote: We can think in a mechanism to create properties / options and assign it to objects (table, index, column, schema, ...) like COMMENT does. A quickly thought: CREATE OPTION [ IF NOT EXISTS ] name VALIDATOR valfunction [

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2014-10-07 Thread Fabrízio de Royes Mello
On Wed, Oct 8, 2014 at 12:36 AM, Josh Berkus j...@agliodbs.com wrote: On 10/07/2014 09:44 AM, Fabrízio de Royes Mello wrote: We can think in a mechanism to create properties / options and assign it to objects (table, index, column, schema, ...) like COMMENT does. A quickly thought:

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-10-07 Thread furuyao
On 09/29/2014 01:13 PM, furu...@pm.nttdata.co.jp wrote: I don't understand what this patch does. When would you want to use the new --reply-fsync option? Is there any reason *not* to use it? In other words, do we need an option for this, couldn't you just always send the feedback message

Re: [HACKERS] pg_receivexlog always handles -d option argument as connstr

2014-10-07 Thread Amit Kapila
On Tue, Oct 7, 2014 at 8:13 PM, Sawada Masahiko sawada.m...@gmail.com wrote: On Tue, Oct 7, 2014 at 12:58 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Mon, Oct 6, 2014 at 10:23 PM, Sawada Masahiko sawada.m...@gmail.com wrote: Hi all, pg_receivexlog always handles argument of -d

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2014-10-07 Thread David Fetter
On Wed, Oct 08, 2014 at 12:41:46AM -0300, Fabrízio de Royes Mello wrote: On Wed, Oct 8, 2014 at 12:36 AM, Josh Berkus j...@agliodbs.com wrote: On 10/07/2014 09:44 AM, Fabrízio de Royes Mello wrote: We can think in a mechanism to create properties / options and assign it to objects