Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Michael Paquier
On Sat, Oct 4, 2014 at 12:31 AM, Marco Nenciarini wrote: > Compared to first version, we switched from a timestamp+checksum based > approach to one based on LSN. Cool. > This patch adds an option to pg_basebackup and to replication protocol > BASE_BACKUP command to generate a backup_profile file.

Re: [HACKERS] How to make ResourceOwnerForgetBuffer() O(1), instead of O(N^2) scale

2014-10-03 Thread Tom Lane
Heikki Linnakangas writes: > On 10/03/2014 07:08 AM, Kouhei Kaigai wrote: >> What is the best way to solve the problem? > How about creating a separate ResourceOwner for these buffer pins, and > doing a wholesale ResourceOwnerRelease() on it when you're done? That's a thought. Another point is

Re: [HACKERS] pg_receivexlog and replication slots

2014-10-03 Thread Michael Paquier
On Fri, Oct 3, 2014 at 8:57 PM, Andres Freund wrote: > > > > +pg_receivexlog can run in one of two > > following > > +modes, which control physical replication slot: > > I don't think that's good enough. There's also the important mode where > it's not doing --create/--drop at all. W

Re: [HACKERS] How to make ResourceOwnerForgetBuffer() O(1), instead of O(N^2) scale

2014-10-03 Thread james
On 03/10/2014 05:53, Kouhei Kaigai wrote: > Yep, that's my pain. Even though usual query does not take many buffers > pinned, > my use case needs to fetch megabytes scale data at once because of performance > reason; page-by-page synchronous scan makes GPU being idle. Doesn't your GPU have an asyn

Re: [HACKERS] strip nulls functions for json and jsonb

2014-10-03 Thread Pavel Stehule
Hi 2014-10-04 1:23 GMT+02:00 Andrew Dunstan : > As discussed recently, here is an undocumented patch for json_strip_nulls > and jsonb_strip_nulls. > It is looking well Regards Pavel > > cheers > > andrew > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make

Re: [HACKERS] pg_receivexlog and replication slots

2014-10-03 Thread Michael Paquier
On Sat, Oct 4, 2014 at 9:24 AM, Michael Paquier wrote: > > > On Sat, Oct 4, 2014 at 6:48 AM, Andres Freund > wrote: > >> On 2014-10-03 14:02:08 -0400, Robert Haas wrote: >> > On Fri, Oct 3, 2014 at 7:57 AM, Andres Freund >> wrote: >> > > I do wonder whether --create/--drop aren't somewhat wierd

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Mark Kirkwood
On 04/10/14 12:10, Bruce Momjian wrote: On Sat, Oct 4, 2014 at 12:00:36PM +1300, Mark Kirkwood wrote: I don't think we can offer absolutely accurate tuning advice, but I'm sure we can give some guidance. Let me try. +1 I think it is ok to document our reason for providing the new GUC - alon

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Gregory Smith
On 10/3/14, 5:23 PM, Andres Freund wrote: How are we ever going to be able to tune it further without feedback from actual production usage? With improved targeted synthetic test cases that isolate the bottleneck to where it's really obvious, way more obvious than it will ever be in productio

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Tom Lane
Andres Freund writes: > On 2014-10-03 19:54:36 -0400, Tom Lane wrote: >> Good point: independently of all else, it's a bit late to be adding new >> features to 9.4. > This is getting absurd. The feature was there three days ago. Well, an undocumented feature isn't a feature.

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Josh Berkus
On 10/03/2014 05:04 PM, Bruce Momjian wrote: > On Sat, Oct 4, 2014 at 01:57:01AM +0200, Andres Freund wrote: >> On 2014-10-03 19:54:36 -0400, Tom Lane wrote: >>> Bruce Momjian writes: Agreeed. Also, reality check --- we can't change postgresql.conf easily without an initdb, and I think

Re: [HACKERS] pg_receivexlog and replication slots

2014-10-03 Thread Michael Paquier
On Sat, Oct 4, 2014 at 6:48 AM, Andres Freund wrote: > On 2014-10-03 14:02:08 -0400, Robert Haas wrote: > > On Fri, Oct 3, 2014 at 7:57 AM, Andres Freund > wrote: > > > I do wonder whether --create/--drop aren't somewhat wierd for > > > pg_receivexlog. It's not that clear what it means. It'd be

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Bruce Momjian
On Sat, Oct 4, 2014 at 01:57:01AM +0200, Andres Freund wrote: > On 2014-10-03 19:54:36 -0400, Tom Lane wrote: > > Bruce Momjian writes: > > > Agreeed. Also, reality check --- we can't change postgresql.conf easily > > > without an initdb, and I think our last 9.4 initdb is going to be > > > 9.4b

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Andres Freund
On 2014-10-03 19:54:36 -0400, Tom Lane wrote: > Bruce Momjian writes: > > Agreeed. Also, reality check --- we can't change postgresql.conf easily > > without an initdb, and I think our last 9.4 initdb is going to be > > 9.4beta3, which is going to be packaged on Monday. > > Good point: independe

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Tom Lane
Bruce Momjian writes: > Agreeed. Also, reality check --- we can't change postgresql.conf easily > without an initdb, and I think our last 9.4 initdb is going to be > 9.4beta3, which is going to be packaged on Monday. Good point: independently of all else, it's a bit late to be adding new feature

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Bruce Momjian
On Fri, Oct 3, 2014 at 07:39:25PM -0400, Greg Smith wrote: > I do not disagree with you fundamentally here: this *is* worth > refining further, for sure, and the gains might be even greater if > that keeps going. My guess is just that the Postgres community > would not get a net benefit chasing t

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Gregory Smith
On 10/3/14, 10:11 AM, Andres Freund wrote: So 25% performance on a relatively small machine improvements aren't worth a GUC? That are likely to be larger on a bigger machine? I utterly fail to see why that's a service to our users. I didn't say that. I said I don't think they're worth a GUC t

[HACKERS] strip nulls functions for json and jsonb

2014-10-03 Thread Andrew Dunstan
As discussed recently, here is an undocumented patch for json_strip_nulls and jsonb_strip_nulls. cheers andrew diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c index 2d00dbe..e9636d8 100644 --- a/src/backend/utils/adt/jsonfuncs.c +++ b/src/backend/utils/adt/jso

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Bruce Momjian
On Sat, Oct 4, 2014 at 12:00:36PM +1300, Mark Kirkwood wrote: > >I don't think we can offer absolutely accurate tuning advice, but I'm > >sure we can give some guidance. Let me try. > > > > +1 > > I think it is ok to document our reason for providing the new GUC - > along with that fact that it

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Mark Kirkwood
On 04/10/14 11:21, Andres Freund wrote: On 2014-10-03 18:16:28 -0400, Bruce Momjian wrote: On Sat, Oct 4, 2014 at 12:13:00AM +0200, Andres Freund wrote: Do we really want to expose a setting a few of us _might_ ask customers to change? They also will try that themselves. Our customers aren't

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-03 Thread Peter Geoghegan
On Fri, Oct 3, 2014 at 3:42 PM, Peter Geoghegan wrote: >> We routinely pick very >> different plans based on the presence or absence of an index, and >> we use special snapshots in the course of executing many DML >> statements (if FK triggers are fired) Apart from FK snapshots, we also use dirty

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-03 Thread Peter Geoghegan
On Fri, Oct 3, 2014 at 3:42 PM, Peter Geoghegan wrote: > Yes, it is. I am opposed to using the MERGE syntax for this *because* > MERGE is useful (independently useful, when done properly), not > because it is not useful. As I've mentioned, there is also the practical argument: No one else does t

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-03 Thread Peter Geoghegan
On Fri, Oct 3, 2014 at 2:44 PM, Kevin Grittner wrote: > I've never seen "atomic" used to mean "you never get an error" > before. > When you are saying "atomic" you mean something quite different. Perhaps I should have been more careful on that point. Just to be crystal clear: I don't intend that

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Andres Freund
On 2014-10-03 18:16:28 -0400, Bruce Momjian wrote: > On Sat, Oct 4, 2014 at 12:13:00AM +0200, Andres Freund wrote: > > > Do we really want to expose a setting a few of us _might_ ask customers > > > to change? > > > > They also will try that themselves. Our customers aren't a horde of dumb > > pe

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Bruce Momjian
On Sat, Oct 4, 2014 at 12:13:00AM +0200, Andres Freund wrote: > > Do we really want to expose a setting a few of us _might_ ask customers > > to change? > > They also will try that themselves. Our customers aren't a horde of dumb > people. Some of them are willing to try things if they hit scalab

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Andres Freund
On 2014-10-03 18:08:56 -0400, Bruce Momjian wrote: > On Fri, Oct 3, 2014 at 11:58:14PM +0200, Andres Freund wrote: > > On 2014-10-03 17:55:19 -0400, Tom Lane wrote: > > > Andres Freund writes: > > > > On 2014-10-03 12:40:21 -0400, Bruce Momjian wrote: > > > >> Well, I think the issue is that havi

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Bruce Momjian
On Fri, Oct 3, 2014 at 11:58:14PM +0200, Andres Freund wrote: > On 2014-10-03 17:55:19 -0400, Tom Lane wrote: > > Andres Freund writes: > > > On 2014-10-03 12:40:21 -0400, Bruce Momjian wrote: > > >> Well, I think the issue is that having a GUC that can't reasonably be > > >> tuned by 95% of our

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-03 Thread Bruce Momjian
On Fri, Oct 3, 2014 at 11:15:13PM +0200, Andres Freund wrote: > > As far as gathering data, I don't think we are going to do any better in > > terms of performance/simplicity/reliability than to have a single PGPROC > > entry to record when we enter/exit a lock, and having a secondary > > process

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Tom Lane
Andres Freund writes: > On 2014-10-03 12:40:21 -0400, Bruce Momjian wrote: >> Well, I think the issue is that having a GUC that can't reasonably be >> tuned by 95% of our users is nearly useless. Few users are going to run >> benchmarks to see what the optimal value is. > It's possible to convin

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Andres Freund
On 2014-10-03 17:55:19 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2014-10-03 12:40:21 -0400, Bruce Momjian wrote: > >> Well, I think the issue is that having a GUC that can't reasonably be > >> tuned by 95% of our users is nearly useless. Few users are going to run > >> benchmarks to s

Re: [HACKERS] pg_receivexlog and replication slots

2014-10-03 Thread Andres Freund
On 2014-10-03 14:02:08 -0400, Robert Haas wrote: > On Fri, Oct 3, 2014 at 7:57 AM, Andres Freund wrote: > > I do wonder whether --create/--drop aren't somewhat wierd for > > pg_receivexlog. It's not that clear what it means. It'd be ugly, but we > > could rename them --create-slot/drop-slot. > >

Re: [HACKERS] test_shm_mq failing on anole (was: Sending out a request for more buildfarm animals?)

2014-10-03 Thread Andres Freund
On 2014-10-03 14:38:10 -0400, Robert Haas wrote: > On Fri, Oct 3, 2014 at 1:09 PM, Robert Haas wrote: > > Further debugging reveals that sigusr1_handler() gets called > > repeatedly, to start autovacuum workers, and it keeps waking up and > > starting them. But that doesn't cause the background w

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-03 Thread Kevin Grittner
Peter Geoghegan wrote: > On Fri, Oct 3, 2014 at 1:16 PM, Kevin Grittner wrote: >> I'm having trouble parsing what this statement means: >> >>> ... the SQL standard does not require that MERGE be atomic in >>> the sense of atomically providing either an INSERT or UPDATE, ... > ... always getting

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Andres Freund
On 2014-10-03 12:40:21 -0400, Bruce Momjian wrote: > On Fri, Oct 3, 2014 at 04:11:30PM +0200, Andres Freund wrote: > > On 2014-10-03 10:07:39 -0400, Gregory Smith wrote: > > > On 10/3/14, 8:26 AM, Andres Freund wrote: > > > >#define NUM_XLOGINSERT_LOCKS 1 > > > >tps = 52.711939 (including connect

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Andres Freund
On 2014-10-03 14:02:09 -0300, Alvaro Herrera wrote: > Since the patch has had good feedback and no further comments arise, I > can just implement support for those two missing object types and push, > and everybody will be happy. Right? I'd like to see a new version before that out here... I don'

Re: [HACKERS] Last Commitfest patches waiting review

2014-10-03 Thread Andres Freund
On 2014-10-03 19:14:14 +0300, Heikki Linnakangas wrote: > Thanks to everyone's who's reviewed a patch so far. One last crunch, and > we'll be through. > > We have 7 patches left in Needs Review state: > > pg_receivexlog: addition of --create/--drop to create/drop repslots > > Waiting for Magn

Re: [HACKERS] Promise index tuples for UPSERT

2014-10-03 Thread Peter Geoghegan
On Fri, Oct 3, 2014 at 3:54 AM, Heikki Linnakangas wrote: > Simon's approach would actually pass that test case just fine. It inserts > the (promise) index tuple first, and heap tuple only after that. It will > fail the test case with more than one unique index, however. Oh, I see. Still, I don't

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-03 Thread Andres Freund
On 2014-10-03 11:51:46 -0400, Robert Haas wrote: > On Fri, Oct 3, 2014 at 11:33 AM, Bruce Momjian wrote: > > I am assuming almost no one cares about the number of locks, but rather > > they care about cummulative lock durations. > > > > I am having trouble seeing any other option that has such a g

Re: [HACKERS] Trailing comma support in SELECT statements

2014-10-03 Thread Pavel Stehule
2014-09-28 13:42 GMT+02:00 Bogdan Pilch : > Hi, > I have created a small patch to postgres source (in particular the > psql part of it) that accepts trailing comma at the end of list in > SELECT statement. > It is ANSI/SQL ? Why we should to enable? We can be tolerant to this bug, but then devel

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-03 Thread Andres Freund
On 2014-10-03 11:33:18 -0400, Bruce Momjian wrote: > On Thu, Oct 2, 2014 at 11:50:14AM +0200, Andres Freund wrote: > > The first problem that comes to my mind about collecting enough data is > > that we have a very large number of lwlocks (fixed_number + 2 * > > shared_buffers). One 'trivial' way

Re: [HACKERS] superuser() shortcuts

2014-10-03 Thread Brightwell, Adam
All, > Thanks! Please add it to the next commitfest. > > > Sounds good. I'll update the patch and add accordingly. > Attached is an updated patch. -Adam -- Adam Brightwell - adam.brightw...@crunchydatasolutions.com Database Engineer - www.crunchydatasolutions.com diff --git a/src/backend/ac

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Andres Freund
On 2014-10-03 17:31:45 +0200, Marco Nenciarini wrote: > I've updated the wiki page > https://wiki.postgresql.org/wiki/Incremental_backup following the result > of discussion on hackers. > > Compared to first version, we switched from a timestamp+checksum based > approach to one based on LSN. > >

Re: [HACKERS] Trailing comma support in SELECT statements

2014-10-03 Thread David G Johnston
Andrew Dunstan wrote > On 10/03/2014 12:20 PM, Bruce Momjian wrote: >> On Sun, Sep 28, 2014 at 01:42:46PM +0200, Bogdan Pilch wrote: >>> Hi, >>> I have created a small patch to postgres source (in particular the >>> psql part of it) that accepts trailing comma at the end of list in >>> SELECT state

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-03 Thread Peter Geoghegan
On Fri, Oct 3, 2014 at 1:16 PM, Kevin Grittner wrote: > I have two questions I hope you can clarify. I'm having trouble > parsing what this statement means: > >> ... the SQL standard does not require that MERGE be atomic in the >> sense of atomically providing either an INSERT or UPDATE, ... > >

Re: [HACKERS] DDL Damage Assessment

2014-10-03 Thread José Luis Tallón
On 10/03/2014 11:02 AM, Dimitri Fontaine wrote: Jim Nasby writes: EXPLAIN ALTER TABLE I'm thinking it would be better to have something you could set at a session level, so you don't have to stick EXPLAIN in front of all your DDL. We were considering the potential needs of "accidental D

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Alvaro Herrera
Robert Haas wrote: > I'm not really very convinced that it's a good idea to expose this > instead of just figuring out a way to parse the object identity. That's the first thing I tried. But it's not pretty: you have to extract schema names by splitting at a period (and what if a schema name con

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Robert Haas
On Fri, Oct 3, 2014 at 2:33 PM, Alvaro Herrera wrote: > Heikki Linnakangas wrote: >> On 10/03/2014 09:06 PM, Alvaro Herrera wrote: > >> >Well, the return value from get_object_address is an ObjectAddress. >> >It's simple enough to create an SQL wrapper that takes the >> >address_names/address_args

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Stephen Frost
* Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > Stephen Frost wrote: > > ahh, ok, that makes a bit more sense, sorry for missing it. Still makes > > me wonder why objargs gets special treatment at the top of the function > > and objnames doesn't- seems like both should be initialized either >

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Alvaro Herrera
Stephen Frost wrote: > * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > > Right. In the "add to objname" cases, there is already some other > > routine that initialized it previously by filling in some stuff; in the > > case above, this happens in the getRelationIdentity() immediately > > prec

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Robert Haas
On Fri, Oct 3, 2014 at 12:08 PM, Marco Nenciarini wrote: > Il 03/10/14 17:53, Heikki Linnakangas ha scritto: >> If we're going to need a profile file - and I'm not convinced of that - >> is there any reason to not always include it in the backup? > > The main reason is to have a centralized list o

Re: [HACKERS] Trailing comma support in SELECT statements

2014-10-03 Thread Andrew Dunstan
On 10/03/2014 12:20 PM, Bruce Momjian wrote: On Sun, Sep 28, 2014 at 01:42:46PM +0200, Bogdan Pilch wrote: Hi, I have created a small patch to postgres source (in particular the psql part of it) that accepts trailing comma at the end of list in SELECT statement. The idea is to be able to say b

Re: [HACKERS] Missing newlines in verbose logs of pg_dump, introduced by RLS patch

2014-10-03 Thread Stephen Frost
* Stephen Frost (sfr...@snowman.net) wrote: > * Michael Paquier (michael.paqu...@gmail.com) wrote: > > On Mon, Sep 29, 2014 at 10:07 AM, Fabrízio de Royes Mello < > > fabriziome...@gmail.com> wrote: > > > > > The schema name is missing... attached patch add it. > > > > > Ah, right, thanks. It didn

[HACKERS] Re: Valgrind warnings in master branch ("Invalid read of size 8") originating within CreatePolicy()

2014-10-03 Thread Stephen Frost
Peter, * Peter Geoghegan (p...@heroku.com) wrote: > I see the following Valgrind warnings in a recent build of the master branch: Fix pushed, thanks! Stephen signature.asc Description: Digital signature

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Stephen Frost
* Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > Right. In the "add to objname" cases, there is already some other > routine that initialized it previously by filling in some stuff; in the > case above, this happens in the getRelationIdentity() immediately > preceding this. > > In the other c

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Alvaro Herrera
Stephen Frost wrote: > Alvaro, > > * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > > + /* > > +* Make sure that both objname and objargs were passed, or none was. > > +* Initialize objargs to empty list, which is the most common case. > > +*/ > > + Assert(PointerIsValid(objnam

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-03 Thread Kevin Grittner
Peter Geoghegan wrote: > The page is: https://wiki.postgresql.org/wiki/UPSERT Thanks! I have two questions I hope you can clarify. I'm having trouble parsing what this statement means: > ... the SQL standard does not require that MERGE be atomic in the > sense of atomically providing eithe

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Stephen Frost
Alvaro, * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > + /* > + * Make sure that both objname and objargs were passed, or none was. > + * Initialize objargs to empty list, which is the most common case. > + */ > + Assert(PointerIsValid(objname) == PointerIsValid(objarg

Re: [HACKERS] superuser() shortcuts

2014-10-03 Thread Brightwell, Adam
Stephen, Thanks! Please add it to the next commitfest. Sounds good. I'll update the patch and add accordingly. > I don't think has_rolinherit or has_rolcatupdate really need to move and > it seems unlikely that they'd be needed from elsewhere.. Is there a > reason you think they'd need to b

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Alvaro Herrera
Alvaro Herrera wrote: > Precisely the point is not returning those values, because they are > useless to identify the equivalent object in a remote database. What we > need is the object names and other stuff used to uniquely identify it > "by user-visible name". We transmit those name arrays to

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Merlin Moncure
On Fri, Oct 3, 2014 at 2:20 PM, Robert Haas wrote: > On Fri, Oct 3, 2014 at 2:49 PM, Heikki Linnakangas > wrote: >> I stand by my decision to make it a #define, at least until someone voices >> their objection in the form of a documentation patch. > > I think that's exactly right. If we knew use

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Robert Haas
On Fri, Oct 3, 2014 at 2:49 PM, Heikki Linnakangas wrote: > I stand by my decision to make it a #define, at least until someone voices > their objection in the form of a documentation patch. I think that's exactly right. If we knew users should tune this, then we might be able to make it self-tu

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Heikki Linnakangas
On 10/03/2014 09:42 PM, Bruce Momjian wrote: On Fri, Oct 3, 2014 at 03:30:35PM -0300, Arthur Silva wrote: > Every GUC add complexity to the system because people have to understand > it to know if they should tune it. No GUC is zero-cost. Please see my blog post about the cost

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Bruce Momjian
On Fri, Oct 3, 2014 at 03:30:35PM -0300, Arthur Silva wrote: > > Every GUC add complexity to the system because people have to understand > > it to know if they should tune it. No GUC is zero-cost. > > Please see my blog post about the cost of adding GUCs: > > http://mom

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Tom Lane
Bruce Momjian writes: > On Fri, Oct 3, 2014 at 03:00:56PM -0300, Arthur Silva wrote: >> Not all GUC need to be straight forward to tune. >> If the gains are worthy I don't see any reason not to have it. > Every GUC add complexity to the system because people have to understand > it to know if th

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-03 Thread Bruce Momjian
On Fri, Oct 3, 2014 at 05:53:59PM +0200, Ilya Kosmodemiansky wrote: > > What that gives us is almost zero overhead on backends, high > > reliability, and the ability of the scan daemon to give higher weights > > to locks that are held longer. Basically, if you just stored the locks > > you held a

Re: [HACKERS] test_shm_mq failing on anole (was: Sending out a request for more buildfarm animals?)

2014-10-03 Thread Robert Haas
On Fri, Oct 3, 2014 at 1:09 PM, Robert Haas wrote: > Further debugging reveals that sigusr1_handler() gets called > repeatedly, to start autovacuum workers, and it keeps waking up and > starting them. But that doesn't cause the background workers to get > started either, because although sigusr1_

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Alvaro Herrera
Heikki Linnakangas wrote: > On 10/03/2014 09:06 PM, Alvaro Herrera wrote: > >Well, the return value from get_object_address is an ObjectAddress. > >It's simple enough to create an SQL wrapper that takes the > >address_names/address_args arrays and return an ObjectAddress; would > >this be useful?

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Arthur Silva
On Fri, Oct 3, 2014 at 3:10 PM, Bruce Momjian wrote: > On Fri, Oct 3, 2014 at 02:07:45PM -0400, Bruce Momjian wrote: > > On Fri, Oct 3, 2014 at 03:00:56PM -0300, Arthur Silva wrote: > > > I remember Informix had a setting that had no description except > "try > > > different values to s

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Heikki Linnakangas
On 10/03/2014 09:06 PM, Alvaro Herrera wrote: Heikki Linnakangas wrote: I had a very brief look at the docs, and these extra outputs from pg_event_trigger_dropped_objects caught my eye: + + address_names + text[] + + An array that, together with addre

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Bruce Momjian
On Fri, Oct 3, 2014 at 06:08:47PM +0200, Marco Nenciarini wrote: > >> Any comment will be appreciated. In particular I'd appreciate comments > >> on correctness of relnode files detection and LSN extraction code. > > > > I didn't look at it in detail, but one future problem comes to mind: > > Onc

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Bruce Momjian
On Fri, Oct 3, 2014 at 02:07:45PM -0400, Bruce Momjian wrote: > On Fri, Oct 3, 2014 at 03:00:56PM -0300, Arthur Silva wrote: > > I remember Informix had a setting that had no description except "try > > different values to see if it helps performance" --- we don't want to do > > that.

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Bruce Momjian
On Fri, Oct 3, 2014 at 03:00:56PM -0300, Arthur Silva wrote: > I remember Informix had a setting that had no description except "try > different values to see if it helps performance" --- we don't want to do > that. > > What if we emit a server message if the setting is too low?

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Alvaro Herrera
Heikki Linnakangas wrote: > I had a very brief look at the docs, and these extra outputs from > pg_event_trigger_dropped_objects caught my eye: > > >+ > >+ address_names > >+ text[] > >+ > >+ An array that, together with address_args, > >+ can be

Re: [HACKERS] pg_receivexlog and replication slots

2014-10-03 Thread Robert Haas
On Fri, Oct 3, 2014 at 7:57 AM, Andres Freund wrote: > I do wonder whether --create/--drop aren't somewhat wierd for > pg_receivexlog. It's not that clear what it means. It'd be ugly, but we > could rename them --create-slot/drop-slot. +1 on doing it, -1 on it being ugly. -- Robert Haas Enterpr

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Arthur Silva
On Fri, Oct 3, 2014 at 1:40 PM, Bruce Momjian wrote: > On Fri, Oct 3, 2014 at 04:11:30PM +0200, Andres Freund wrote: > > On 2014-10-03 10:07:39 -0400, Gregory Smith wrote: > > > On 10/3/14, 8:26 AM, Andres Freund wrote: > > > >#define NUM_XLOGINSERT_LOCKS 1 > > > >tps = 52.711939 (including con

Re: [HACKERS] Proposal for updating src/timezone

2014-10-03 Thread Tom Lane
I wrote: > The urgency of updating our timezone code has risen quite a bit for me, > because while testing an update of the data files to tzdata2014h I became > aware that the "-P" option is failing to print a noticeable number of > zone abbreviations that clearly exist in the data files. Since th

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Heikki Linnakangas
On 10/03/2014 08:08 PM, Stephen Frost wrote: Alvaro, * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: There are three fixmes in the code. One can be handled by just removing the line; we don't really care about duplicating 10 lines of boilerplate code. The other two mean missing support for

Re: [HACKERS] test_shm_mq failing on anole (was: Sending out a request for more buildfarm animals?)

2014-10-03 Thread Robert Haas
On Wed, Oct 1, 2014 at 11:10 AM, Robert Haas wrote: > As far as I can tell, it's configured to run everything. I just > checked, though, and found it wedged again. I'm not sure whether it > was the same problem, though; I ended up just killing all of the > postgres processes to fix it. We may b

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Stephen Frost
Alvaro, * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > There are three fixmes in the code. One can be handled by just removing > the line; we don't really care about duplicating 10 lines of boilerplate > code. The other two mean missing support for domain constraints and for > default ACLs

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Alvaro Herrera
Heikki Linnakangas wrote: > On 09/16/2014 09:09 PM, Brightwell, Adam wrote: > >I have given this patch the following review: > > > >- Apply to current master (77e65bf). -- success > >- check-world. --success > >- multiple FIXME statements still exist -- are there plans to fix these > >items? Can

Re: [HACKERS] Last Commitfest patches waiting review

2014-10-03 Thread Magnus Hagander
On Fri, Oct 3, 2014 at 6:14 PM, Heikki Linnakangas wrote: > Thanks to everyone's who's reviewed a patch so far. One last crunch, and > we'll be through. > > We have 7 patches left in Needs Review state: > > pg_receivexlog: addition of --create/--drop to create/drop repslots > > Waiting for Magn

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Bruce Momjian
On Fri, Oct 3, 2014 at 04:11:30PM +0200, Andres Freund wrote: > On 2014-10-03 10:07:39 -0400, Gregory Smith wrote: > > On 10/3/14, 8:26 AM, Andres Freund wrote: > > >#define NUM_XLOGINSERT_LOCKS 1 > > >tps = 52.711939 (including connections establishing) > > >#define NUM_XLOGINSERT_LOCKS 8 > > >

Re: [HACKERS] CREATE IF NOT EXISTS INDEX

2014-10-03 Thread Fabrízio de Royes Mello
On Fri, Oct 3, 2014 at 5:26 AM, Marti Raudsepp wrote: > > "On Fri, Oct 3, 2014 at 6:25 AM, Fabrízio de Royes Mello > wrote: > >> Documentation: I would prefer if the explanation were consistent with > > > "Do not throw an error if the index already exists. A notice is issued in > > this case." >

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Claudio Freire
On Fri, Oct 3, 2014 at 1:08 PM, Marco Nenciarini wrote: >>> Any comment will be appreciated. In particular I'd appreciate comments >>> on correctness of relnode files detection and LSN extraction code. >> >> I didn't look at it in detail, but one future problem comes to mind: >> Once you implement

Re: [HACKERS] Trailing comma support in SELECT statements

2014-10-03 Thread Bruce Momjian
On Sun, Sep 28, 2014 at 01:42:46PM +0200, Bogdan Pilch wrote: > Hi, > I have created a small patch to postgres source (in particular the > psql part of it) that accepts trailing comma at the end of list in > SELECT statement. > > The idea is to be able to say both (with the same result): > SELECT

Re: [HACKERS] Trailing comma support in SELECT statements

2014-10-03 Thread Tom Lane
Bogdan Pilch writes: > I have created a small patch to postgres source (in particular the > psql part of it) that accepts trailing comma at the end of list in > SELECT statement. This doesn't seem to me to be a remarkably good idea. What's the difference between this and accepting random misspel

Re: [HACKERS] Last Commitfest patches waiting review

2014-10-03 Thread Heikki Linnakangas
Thanks to everyone's who's reviewed a patch so far. One last crunch, and we'll be through. We have 7 patches left in Needs Review state: pg_receivexlog: addition of --create/--drop to create/drop repslots Waiting for Magnus. Amit promised to take a look if Magnus continues to be busy. So

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Marco Nenciarini
Il 03/10/14 17:53, Heikki Linnakangas ha scritto: > If we're going to need a profile file - and I'm not convinced of that - > is there any reason to not always include it in the backup? > The main reason is to have a centralized list of files that need to be present. Without a profile, you have t

Re: [HACKERS] replicating DROP commands across servers

2014-10-03 Thread Heikki Linnakangas
On 09/16/2014 09:09 PM, Brightwell, Adam wrote: I think there's been some changes to this patch since july, care to resend a new version? Sure, here it is. The only difference with the previous version is that it now also supports column defaults. This was found to be a problem when you dro

Re: [HACKERS] Per table autovacuum vacuum cost limit behaviour strange

2014-10-03 Thread Alvaro Herrera
Stephen Frost wrote: > * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > > I am rather surprised that nobody has reported this problem before. I > > am now of the mind that this is clearly a bug that should be fixed all > > the way back. > > I'm coming around to that also, however, should we

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-03 Thread Ilya Kosmodemiansky
On Fri, Oct 3, 2014 at 5:51 PM, Robert Haas wrote: > I do think that the instrumentation data gathered by LWLOCK_STATS is > useful - very useful. Sure, quite useful. But how about this comment: /* * The LWLock stats will be updated within a critical section, which * requires alloc

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-03 Thread Ilya Kosmodemiansky
On Fri, Oct 3, 2014 at 5:33 PM, Bruce Momjian wrote: > As far as gathering data, I don't think we are going to do any better in > terms of performance/simplicity/reliability than to have a single PGPROC > entry to record when we enter/exit a lock, and having a secondary > process scan the PGPROC a

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Heikki Linnakangas
On 10/03/2014 06:31 PM, Marco Nenciarini wrote: Hi Hackers, I've updated the wiki page https://wiki.postgresql.org/wiki/Incremental_backup following the result of discussion on hackers. Compared to first version, we switched from a timestamp+checksum based approach to one based on LSN. This pa

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-03 Thread Robert Haas
On Fri, Oct 3, 2014 at 11:33 AM, Bruce Momjian wrote: > I am assuming almost no one cares about the number of locks, but rather > they care about cummulative lock durations. > > I am having trouble seeing any other option that has such a good > cost/benefit profile. I do think that the instrument

Re: [HACKERS] Typo fixes in src/backend/rewrite/rewriteHandler.c

2014-10-03 Thread Robert Haas
On Thu, Oct 2, 2014 at 2:55 AM, Etsuro Fujita wrote: > Here is the comments in process_matched_tle() in rewriteHandler.c. > > 883 * such nodes; consider > 884 * UPDATE tab SET col.fld1.subfld1 = x, col.fld2.subfld2 = y > 885 * The two expressions produced by the parser will loo

Re: [HACKERS] Dynamic LWLock tracing via pg_stat_lwlock (proof of concept)

2014-10-03 Thread Bruce Momjian
On Thu, Oct 2, 2014 at 11:50:14AM +0200, Andres Freund wrote: > The first problem that comes to my mind about collecting enough data is > that we have a very large number of lwlocks (fixed_number + 2 * > shared_buffers). One 'trivial' way of implementing this is to have a per > backend array colle

[HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Marco Nenciarini
Hi Hackers, I've updated the wiki page https://wiki.postgresql.org/wiki/Incremental_backup following the result of discussion on hackers. Compared to first version, we switched from a timestamp+checksum based approach to one based on LSN. This patch adds an option to pg_basebackup and to replica

Re: [HACKERS] Escaping from blocked send() reprised.

2014-10-03 Thread Heikki Linnakangas
On 10/03/2014 05:26 PM, Andres Freund wrote: On 2014-10-03 17:12:18 +0300, Heikki Linnakangas wrote: On 09/28/2014 01:54 AM, Andres Freund wrote: 0003 Sinval/notify processing got simplified further. There really isn't any need for DisableNotifyInterrupt/DisableCatchupInterrupt anym

Re: [HACKERS] Log notice that checkpoint is to be written on shutdown

2014-10-03 Thread Alvaro Herrera
Michael Banck wrote: > diff --git a/src/backend/access/transam/xlog.c > b/src/backend/access/transam/xlog.c > index 5a4dbb9..f2716ae 100644 > --- a/src/backend/access/transam/xlog.c > +++ b/src/backend/access/transam/xlog.c > @@ -8085,10 +8085,14 @@ CreateCheckPoint(int flags) > > /* >

[HACKERS] Trailing comma support in SELECT statements

2014-10-03 Thread Bogdan Pilch
Hi, I have created a small patch to postgres source (in particular the psql part of it) that accepts trailing comma at the end of list in SELECT statement. The idea is to be able to say both (with the same result): SELECT a, b, c from t; SELECT a, b, c, from t; Attached you can find a patch conta

Re: [HACKERS] Escaping from blocked send() reprised.

2014-10-03 Thread Andres Freund
Hi, On 2014-10-03 17:12:18 +0300, Heikki Linnakangas wrote: > On 09/28/2014 01:54 AM, Andres Freund wrote: > >I've invested some more time in this: > > Thanks! > > In 0001, the select() codepath will not return (WL_SOCKET_READABLE | > WL_SOCKET_WRITEABLE) on EOF or error, like the comment says a

  1   2   >