Re: [HACKERS] drop duplicate buffers in OS

2014-01-16 Thread KONDO Mitsumasa
(2014/01/16 3:34), Robert Haas wrote: On Wed, Jan 15, 2014 at 1:53 AM, KONDO Mitsumasa wrote: I create patch that can drop duplicate buffers in OS using usage_count alogorithm. I have developed this patch since last summer. This feature seems to be discussed in hot topic, so I submit it more fa

Re: [HACKERS] drop duplicate buffers in OS

2014-01-16 Thread KONDO Mitsumasa
(2014/01/16 21:38), Aidan Van Dyk wrote: Can we just get the backend that dirties the page to the posix_fadvice DONTNEED? No, it can remove clean page in OS file caches. Because if page is dirtied, it cause physical-disk-writing. However, it is experimental patch so it might be changed by futur

Re: [HACKERS] Backup throttling

2014-01-16 Thread Michael Paquier
On Fri, Jan 17, 2014 at 5:03 AM, Andres Freund wrote: > slightly related: we should start to reuse procLatch for walsenders > instead of having a separate latch someday. + 1 on that. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscr

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Amit Kapila
On Fri, Jan 17, 2014 at 12:16 AM, Tom Lane wrote: > PS: off topic, but isn't ParseConfigDirectory leaking the result > of AbsoluteConfigLocation? In both normal and error paths? Yes, I also think it leaks in both cases and similar leak is present in ParseConfigFile(). I have tried to fix b

Re: [HACKERS] Heavily modified big table bloat even in auto vacuum is running

2014-01-16 Thread Amit Kapila
On Thu, Jan 16, 2014 at 10:24 PM, tirtho wrote: > Is this now fixed? If so, where do I find the patch for postgres 9.2.2. This is still not fixed, the patch is in Ready For Committer stage. You can confirm the status here: https://commitfest.postgresql.org/action/patch_view?id=1258

Re: [HACKERS] [Lsf-pc] Linux kernel impact on PostgreSQL performance

2014-01-16 Thread Jeff Janes
On Thursday, January 16, 2014, Dave Chinner > wrote: > On Thu, Jan 16, 2014 at 03:58:56PM -0800, Jeff Janes wrote: > > On Thu, Jan 16, 2014 at 3:23 PM, Dave Chinner > wrote: > > > > > On Wed, Jan 15, 2014 at 06:14:18PM -0600, Jim Nasby wrote: > > > > On 1/15/14, 12:00 AM, Claudio Freire wrote: >

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Craig Ringer
On 01/16/2014 11:52 PM, Tom Lane wrote: > Heikki Linnakangas writes: >> On 01/16/2014 05:39 PM, Andres Freund wrote: >>> Do you see a reasonable way to implement this generically for all >>> commands? I don't. > >> In suitable safe places, check if you've written too much WAL, and sleep >> if so

Re: [HACKERS] Changeset Extraction v7.0 (was logical changeset generation)

2014-01-16 Thread Craig Ringer
On 01/16/2014 02:28 AM, Robert Haas wrote: > - If you address /* FIXME: apply sanity checking to slot name */, then > I think that also addresses /* XXX: do we want to use truncate > identifier instead? */. In other words, let's just error out if the > name is too long. I'm not sure what other san

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-16 Thread Peter Geoghegan
On Wed, Jan 15, 2014 at 11:02 PM, Peter Geoghegan wrote: > It might just be a matter of: > > @@ -186,6 +186,13 @@ ExecLockHeapTupleForUpdateSpec(EState *estate, > switch (test) > { > case HeapTupleInvisible: > + /* > +*

Re: [HACKERS] [Lsf-pc] Linux kernel impact on PostgreSQL performance

2014-01-16 Thread Greg Stark
On Wed, Jan 15, 2014 at 7:53 AM, Mel Gorman wrote: > The second is have > pages that are strictly kept dirty until the application syncs them. An > unbounded number of these pages would blow up but maybe bounds could be > placed on it. There are no solid conclusions on that part yet. I think the

Re: [HACKERS] new json funcs

2014-01-16 Thread Andrew Dunstan
On 01/16/2014 07:39 PM, Andrew Dunstan wrote: On 01/16/2014 01:57 PM, Peter Eisentraut wrote: On 1/3/14, 9:00 PM, Andrew Dunstan wrote: Here is a patch for the new json functions I mentioned a couple of months ago. These are: json_to_record json_to_recordset json_object j

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-16 Thread Florian Pflug
I had some more fun with this, the result is v2.5 of the patch (attached). Changes are explained below. On Jan16, 2014, at 19:10 , Florian Pflug wrote: > On Jan16, 2014, at 09:07 , David Rowley wrote: >> On Wed, Jan 15, 2014 at 5:39 AM, Florian Pflug wrote: >>> The notnullcount machinery seems

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-16 Thread Robert Haas
On Thu, Jan 16, 2014 at 7:31 PM, Dave Chinner wrote: > But there's something here that I'm not getting - you're talking > about a data set that you want ot keep cache resident that is at > least an order of magnitude larger than the cyclic 5-15 minute WAL > dataset that ongoing operations need to

Re: [HACKERS] new json funcs

2014-01-16 Thread Andrew Dunstan
On 01/16/2014 01:57 PM, Peter Eisentraut wrote: On 1/3/14, 9:00 PM, Andrew Dunstan wrote: Here is a patch for the new json functions I mentioned a couple of months ago. These are: json_to_record json_to_recordset json_object json_build_array json_build_object json

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-16 Thread Jeff Janes
On Wed, Jan 15, 2014 at 2:08 AM, Mel Gorman wrote: > On Tue, Jan 14, 2014 at 09:30:19AM -0800, Jeff Janes wrote: > > > > > > That could be something we look at. There are cases buried deep in the > > > VM where pages get shuffled to the end of the LRU and get tagged for > > > reclaim as soon as p

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-16 Thread Jeff Janes
On Thu, Jan 16, 2014 at 3:23 PM, Dave Chinner wrote: > On Wed, Jan 15, 2014 at 06:14:18PM -0600, Jim Nasby wrote: > > On 1/15/14, 12:00 AM, Claudio Freire wrote: > > >My completely unproven theory is that swapping is overwhelmed by > > >near-misses. Ie: a process touches a page, and before it's >

Re: [HACKERS] proposal, patch: allow multiple plpgsql plugins

2014-01-16 Thread Marko Tiikkaja
Hi Pavel, First of all, thanks for working on this! On 1/12/14, 8:58 PM, Pavel Stehule wrote: I still not happy with plugin_info - it is only per plugin now and should be per plugin and per function. I'm not sure I understand the point of plugin_info in the first place, but what would having

Re: [HACKERS] [PATCH] Filter error log statements by sqlstate

2014-01-16 Thread Oskari Saarenmaa
17.01.2014 00:13, Tom Lane kirjoitti: Oskari Saarenmaa writes: [ 0001-Filter-error-log-statements-by-sqlstate.patch ] I looked at this patch. It took me some time to understand that what it actually does has got approximately nothing to do with what one might first expect: rather than suppre

Re: [HACKERS] ECPG FETCH readahead, was: Re: ECPG fixes

2014-01-16 Thread Alvaro Herrera
Boszormenyi Zoltan escribió: > Rebased patches after the regression test and other details were fixed > in the infrastructure part. This thread started in 2010, and various pieces have been applied already and some others have changed in nature. Would you please post a new patchset, containing re

Re: [HACKERS] [PATCH] Filter error log statements by sqlstate

2014-01-16 Thread Peter Eisentraut
Please fix the compiler warning: elog.c: In function ‘check_log_sqlstate_error’: elog.c:3789:41: warning: ‘new_newval_end’ may be used uninitialized in this function [-Wuninitialized] -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: htt

Re: [HACKERS] [PATCH] Filter error log statements by sqlstate

2014-01-16 Thread Tom Lane
Oskari Saarenmaa writes: > [ 0001-Filter-error-log-statements-by-sqlstate.patch ] I looked at this patch. It took me some time to understand that what it actually does has got approximately nothing to do with what one might first expect: rather than suppressing the entire log message about some

[HACKERS] 9.3.2 Client-only installation per docs fails creating directory.

2014-01-16 Thread Mike Blackwell
Per the docs (15.4 Installation Procedure, Client-only installation), after running make, the following should work: gmake -C src/bin installgmake -C src/include installgmake -C src/interfaces installgmake -C doc install However, it fails on the first command: [postgresql-9.3.2]$ sudo gmake -C

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-16 Thread Theodore Ts'o
On Wed, Jan 15, 2014 at 10:35:44AM +0100, Jan Kara wrote: > Filesystems could in theory provide facility like atomic write (at least up > to a certain size say in MB range) but it's not so easy and when there are > no strong usecases fs people are reluctant to make their code more complex > unneces

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-16 Thread Jeff Layton
On Wed, 15 Jan 2014 21:37:16 -0500 Robert Haas wrote: > On Wed, Jan 15, 2014 at 8:41 PM, Jan Kara wrote: > > On Wed 15-01-14 10:12:38, Robert Haas wrote: > >> On Wed, Jan 15, 2014 at 4:35 AM, Jan Kara wrote: > >> > Filesystems could in theory provide facility like atomic write (at least > >> >

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-16 Thread Dave Chinner
On Wed, Jan 15, 2014 at 07:31:15PM -0500, Tom Lane wrote: > Dave Chinner writes: > > On Wed, Jan 15, 2014 at 07:08:18PM -0500, Tom Lane wrote: > >> No, we'd be happy to re-request it during each checkpoint cycle, as > >> long as that wasn't an unduly expensive call to make. I'm not quite > >> sur

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-16 Thread Jan Kara
On Wed 15-01-14 10:12:38, Robert Haas wrote: > On Wed, Jan 15, 2014 at 4:35 AM, Jan Kara wrote: > > Filesystems could in theory provide facility like atomic write (at least up > > to a certain size say in MB range) but it's not so easy and when there are > > no strong usecases fs people are reluct

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-16 Thread Jan Kara
On Wed 15-01-14 21:37:16, Robert Haas wrote: > On Wed, Jan 15, 2014 at 8:41 PM, Jan Kara wrote: > > On Wed 15-01-14 10:12:38, Robert Haas wrote: > >> On Wed, Jan 15, 2014 at 4:35 AM, Jan Kara wrote: > >> > Filesystems could in theory provide facility like atomic write (at least > >> > up > >> >

[HACKERS] ALTER TABLESPACE ... MOVE ALL TO ...

2014-01-16 Thread Stephen Frost
Greetings, It's a day late and I'm a dollar short, but attached is a (very) minor patch to allow users to more easily move their various objects from one tablespace to another. Included are docs and a regression test; I'm happy to improve on both should folks send me suggestions. As we

[HACKERS] ALTER TABLE ... SET TABLESPACE pg_default

2014-01-16 Thread Stephen Frost
Greetings, Harking back to 10 years ago when tablespaces were added, it looks like we originally figured that users didn't need permissions to create tables in the database default, per 2467394e. That strikes me as perfectly fair. Unfortunately, the later addition of ALTER TABLE ... SE

Re: [HACKERS] Backup throttling

2014-01-16 Thread Peter Geoghegan
On Thu, Jan 16, 2014 at 12:03 PM, Andres Freund wrote: > slightly related: we should start to reuse procLatch for walsenders > instead of having a separate latch someday. +1. The potential for bugs from failing to account for this within signal handlers seems like a concern. I think that every pr

Re: Review: [HACKERS] ECPG infrastructure changes part 1, was: Re: ECPG fixes

2014-01-16 Thread Alvaro Herrera
Alvaro Herrera escribió: > Boszormenyi Zoltan escribió: > > > All patches are attached again for completeness. > > Thanks. I pushed a commit comprising patches 09 through 14. Now also pushed 15 to 17. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Su

Re: [HACKERS] dblink performance regression

2014-01-16 Thread Joe Conway
yes Joe Sent with AquaMail for Android http://www.aqua-mail.com On January 16, 2014 2:32:55 PM Josh Berkus wrote: On 12/07/2013 05:50 PM, Joe Conway wrote: > On 12/07/2013 05:41 PM, Fabrízio de Royes Mello wrote: > >> On Sat, Dec 7, 2013 at 11:20 PM, Michael Paquier >> mailto:michael.paqu.

Re: [HACKERS] Display oprcode and its volatility in \do+

2014-01-16 Thread Marko Tiikkaja
On 1/16/14, 9:32 PM, Tom Lane wrote: Given the lack of other votes, I pushed this without a volatility column, and with some other changes --- mostly, I kept the Description column last, because that's how all the other \d commands do it. Thanks! And looks like I missed the documentation as we

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-16 Thread Tom Lane
Andres Freund writes: > On 2014-01-16 14:23:47 -0500, Tom Lane wrote: >> Dunno, I think that a transition state containing both an int64 and >> a (presumably separately palloc'd) numeric will be a real PITA. > Yea, not sure myself. I just dislike the idea of having a good part of a > 128bit math

Re: [HACKERS] Display oprcode and its volatility in \do+

2014-01-16 Thread Tom Lane
I wrote: > Anybody else have an opinion? Given the lack of other votes, I pushed this without a volatility column, and with some other changes --- mostly, I kept the Description column last, because that's how all the other \d commands do it. regards, tom lane -- Sent v

Re: [HACKERS] Backup throttling

2014-01-16 Thread Andres Freund
Hi, On 2014-01-15 18:52:32 -0300, Alvaro Herrera wrote: > Another thing I found a bit strange was the use of the latch. What this > patch does is create a separate latch which is used for the throttling. > This means that if the walsender process receives a signal, it will not > wake up if it's s

Re: Review: [HACKERS] ECPG infrastructure changes part 1, was: Re: ECPG fixes

2014-01-16 Thread Alvaro Herrera
Boszormenyi Zoltan escribió: > All patches are attached again for completeness. Thanks. I pushed a commit comprising patches 09 through 14. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailin

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-16 Thread Andres Freund
On 2014-01-16 14:23:47 -0500, Tom Lane wrote: > > Maybe it's instead sufficient to just have flag indicating that you're > > working with a state that hasn't overflowed so far and just plain int8 > > math as long as that's the case, and entirely fall back to the current > > path once overflowed. Th

Re: [HACKERS] [PATCH] Relocation of tablespaces in pg_basebackup

2014-01-16 Thread Peter Eisentraut
You appear to be generating your patches with git diff --no-prefix. Don't do that, leave the a/ and b/ in. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dblink performance regression

2014-01-16 Thread Josh Berkus
On 12/07/2013 05:50 PM, Joe Conway wrote: > On 12/07/2013 05:41 PM, Fabrízio de Royes Mello wrote: > >> On Sat, Dec 7, 2013 at 11:20 PM, Michael Paquier >> mailto:michael.paqu...@gmail.com>> >> wrote: IMHO is more elegant create a procedure to encapsulate the code to avoid redundan

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-16 Thread Tom Lane
Andres Freund writes: > You'll have to handle adding negative values and underflow as > well. Right. > Maybe it's instead sufficient to just have flag indicating that you're > working with a state that hasn't overflowed so far and just plain int8 > math as long as that's the case, and entirely f

Re: [HACKERS] Patch for fail-back without fresh backup

2014-01-16 Thread Tom Lane
Jeff Janes writes: > I think the argument is that drawing the next value from a sequence can > generate xlog that needs to be flushed, but doesn't assign an xid. > I would think the sequence should flush that record before it hands out the > value, not before the commit, but... IIRC the argument

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-16 Thread Andres Freund
On 2014-01-16 21:07:33 +0200, Heikki Linnakangas wrote: > On 01/16/2014 08:59 PM, Tom Lane wrote: > >Heikki Linnakangas writes: > >>I propose that we reimplement sum(bigint) in a more efficient way: For > >>the internal state, let's use an int8 and a numeric overflow field. The > >>transition func

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-16 Thread Heikki Linnakangas
On 01/16/2014 08:59 PM, Tom Lane wrote: Heikki Linnakangas writes: I propose that we reimplement sum(bigint) in a more efficient way: For the internal state, let's use an int8 and a numeric overflow field. The transition function adds to the int8 variable, and checks for overflow. On overflow,

Re: [HACKERS] Patch for fail-back without fresh backup

2014-01-16 Thread Andres Freund
On 2014-01-16 11:01:29 -0800, Jeff Janes wrote: > I think the argument is that drawing the next value from a sequence can > generate xlog that needs to be flushed, but doesn't assign an xid. Then that should assign an xid. Which would yield correct behaviour with async commit where it's currently

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Peter Eisentraut
The documentation doesn't build: openjade:config.sgml:1416:11:E: end tag for "VARIABLELIST" omitted, but OMITTAG NO was specified openjade:config.sgml:1390:5: start tag was here -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://

Re: [HACKERS] Patch for fail-back without fresh backup

2014-01-16 Thread Jeff Janes
On Thu, Jan 16, 2014 at 9:37 AM, Andres Freund wrote: > On 2014-01-16 09:25:51 -0800, Jeff Janes wrote: > > On Thu, Nov 21, 2013 at 2:43 PM, Andres Freund >wrote: > > > > > On 2013-11-21 14:40:36 -0800, Jeff Janes wrote: > > > > But if the transaction would not have otherwise generated WAL (i.e.

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Josh Berkus
On 01/16/2014 07:32 AM, Christian Kruse wrote: > Hi Alvaro, > > On 16/01/14 10:21, Alvaro Herrera wrote: >> 1. it is to be read automatically by the server without need for an >>"include_dir conf.d" option in the main postgresql.conf. > > +1 > >> 4. there is no point in "disabling" it, and t

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-16 Thread Tom Lane
Heikki Linnakangas writes: > I propose that we reimplement sum(bigint) in a more efficient way: For > the internal state, let's use an int8 and a numeric overflow field. The > transition function adds to the int8 variable, and checks for overflow. > On overflow, increment the numeric field by o

Re: [HACKERS] new json funcs

2014-01-16 Thread Peter Eisentraut
On 1/3/14, 9:00 PM, Andrew Dunstan wrote: > > Here is a patch for the new json functions I mentioned a couple of > months ago. These are: > > json_to_record > json_to_recordset > json_object > json_build_array > json_build_object > json_object_agg > > So far there are no

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Stephen Frost
* Josh Berkus (j...@agliodbs.com) wrote: > On 01/16/2014 10:46 AM, Tom Lane wrote: > > I'm fine if the proposal is that postgresql.conf include "include_dir > > conf.d" by default (where that's read as relative to postgresql.conf's own > > directory). Even better if it's not terribly difficult for

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Josh Berkus
On 01/16/2014 10:46 AM, Tom Lane wrote: > I'm fine if the proposal is that postgresql.conf include "include_dir > conf.d" by default (where that's read as relative to postgresql.conf's own > directory). Even better if it's not terribly difficult for a packager to > change that, because I think som

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Tom Lane
Alvaro Herrera writes: > Tom Lane wrote: >> Another point here is that hard-wiring a config directory location >> into the executables completely breaks many scenarios for running >> multiple clusters with the same executables. > Therefore the proposal is not to hardwire the location in the > exe

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-16 Thread Heikki Linnakangas
On 01/16/2014 01:02 PM, David Rowley wrote: sum(bigint) became a bit weird as it uses numeric types internally, so I had to keep the do_numeric_discard() function to support it. It's pretty weird that we have implemented sum(bigint) that way. I understand that the result is a numeric so that i

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Stephen Frost
* Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > Stephen Frost wrote: > > I missed the part of this where you point out that Apache on Debian has > > some kind of problem because it's possible for an admin to remove the > > 'IncludeDir sites-enabled' line from apache2.conf. > > I don't see tha

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Alvaro Herrera
Tom Lane wrote: > Stephen Frost writes: > > * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > >> 1. it is to be read automatically by the server without need for an > >> "include_dir conf.d" option in the main postgresql.conf. > > > I am not thrilled with the idea that we're claiming ownership

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Jeff Janes
On Thu, Jan 16, 2014 at 8:19 AM, Tom Lane wrote: > > > I think the usecases that would want this for DML probably also wan this > > to work for unlogged, temp tables. > > Huh? Unlogged tables generate *zero* WAL, by definition. > Transactions that only change unlogged tables still generate comm

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-16 Thread Florian Pflug
On Jan16, 2014, at 09:07 , David Rowley wrote: > On Wed, Jan 15, 2014 at 5:39 AM, Florian Pflug wrote: >> The notnullcount machinery seems to apply to both STRICT and non-STRICT >> transfer function pairs. Shouldn't that be constrained to STRICT transfer >> function pairs? For non-STRICT pairs,

Re: [HACKERS] Patch for fail-back without fresh backup

2014-01-16 Thread Andres Freund
On 2014-01-16 09:25:51 -0800, Jeff Janes wrote: > On Thu, Nov 21, 2013 at 2:43 PM, Andres Freund wrote: > > > On 2013-11-21 14:40:36 -0800, Jeff Janes wrote: > > > But if the transaction would not have otherwise generated WAL (i.e. a > > > select that did not have to do any HOT pruning, or an upda

Re: [HACKERS] Patch for fail-back without fresh backup

2014-01-16 Thread Jeff Janes
On Thu, Nov 21, 2013 at 2:43 PM, Andres Freund wrote: > On 2013-11-21 14:40:36 -0800, Jeff Janes wrote: > > But if the transaction would not have otherwise generated WAL (i.e. a > > select that did not have to do any HOT pruning, or an update with zero > rows > > matching the where condition), doe

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Heikki Linnakangas
On 01/16/2014 05:39 PM, Andres Freund wrote: On 2014-01-16 10:35:20 -0500, Tom Lane wrote: I think possibly a more productive approach to this would be to treat it as a session-level GUC setting, rather than hard-wiring it to affect certain commands and not others. Do you see a reasonable way

Re: [HACKERS] [PATCH] Relocation of tablespaces in pg_basebackup

2014-01-16 Thread Steeve Lennmark
Alvaro, On Thu, Jan 16, 2014 at 3:25 PM, Alvaro Herrera wrote: > Eyeballing this patch, three thoughts: > > 1. I wonder whether ilist.c/h should be moved to src/common so that > frontend code could use it. > That would be nice, I guess lack of helpers is why a lot of stuff is using pgdumputils.h

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Simon Riggs
On 16 January 2014 17:49, Tom Lane wrote: > Simon Riggs writes: >> Agreed, but it won't happen in this release. I/O resource control to >> follow in later releases. > > "This release"? TBH, I think this patch has arrived too late to be > considered for 9.4. It's a fairly major feature and clear

Re: [HACKERS] Heavily modified big table bloat even in auto vacuum is running

2014-01-16 Thread tirtho
Is this now fixed? If so, where do I find the patch for postgres 9.2.2. Thanks - Tirthankar -- View this message in context: http://postgresql.1045698.n5.nabble.com/Heavily-modified-big-table-bloat-even-in-auto-vacuum-is-running-tp5774274p5787477.html Sent from the PostgreSQL - hackers mailin

Re: [HACKERS] [PATCH] Relocation of tablespaces in pg_basebackup

2014-01-16 Thread Steeve Lennmark
Alvaro, On Thu, Jan 16, 2014 at 3:20 PM, Alvaro Herrera wrote: > Please keep the --help and the options in the SGML table in alphabetical > order within their respective sections. Ah, I failed to see that there was sub groups and thought the options where not alphabetically ordered. This patch f

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Tom Lane
Simon Riggs writes: > Agreed, but it won't happen in this release. I/O resource control to > follow in later releases. "This release"? TBH, I think this patch has arrived too late to be considered for 9.4. It's a fairly major feature and clearly not without controversy, so I don't think that po

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Simon Riggs
On 16 January 2014 17:29, Andres Freund wrote: >> Please let me know if I missed something (rather than debating what is >> or is not a "maintenance" command). > > If we're going to do this for DML - which I am far from convinced of - > we also should do it for SELECT, since that can generate sig

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > On 2014-01-16 11:35:00 -0500, Stephen Frost wrote: > > * Andres Freund (and...@2ndquadrant.com) wrote: > > > So, if we want to support antything but a) relative to pgdata b) > > > relative to postgresql.conf it needs to be configurable at startup >

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Andres Freund
On 2014-01-16 11:35:00 -0500, Stephen Frost wrote: > * Andres Freund (and...@2ndquadrant.com) wrote: > > So, if we want to support antything but a) relative to pgdata b) > > relative to postgresql.conf it needs to be configurable at startup > > time. Possibly with an added initdb switch to set the

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > So, if we want to support antything but a) relative to pgdata b) > relative to postgresql.conf it needs to be configurable at startup > time. Possibly with an added initdb switch to set the location. How would an initdb switch be better than an opt

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Andres Freund
On 2014-01-16 17:20:19 +0100, Simon Riggs wrote: > I'm comfortable with a session level parameter. I was mulling over a > wal_rate_limit_scope parameter but I think that is too much. > I will follow Craig's proposal to define this in terms of MB/s, adding > that I would calc from beginning of state

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Stephen Frost
Avlaro, * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > Please explain. I don't see a reason not to. Are you saying you expect > to put, say, Apache configuration files in the same directory as > PostgreSQL's? That doesn't sound really tenable to me, and it doesn't > sounds like what any s

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Simon Riggs
On 16 January 2014 17:22, Andres Freund wrote: > On 2014-01-16 11:19:28 -0500, Tom Lane wrote: >> > I think the usecases that would want this for DML probably also wan this >> > to work for unlogged, temp tables. >> >> Huh? Unlogged tables generate *zero* WAL, by definition. > > Yes. That's my po

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Andres Freund
On 2014-01-16 11:12:55 -0500, Tom Lane wrote: > If we were to do this, I'd argue that the location of this hard-wired > config directory ought to be selected by a configure option. > in fact, I'd argue that the default behavior with no such option be > that there's no such hard-wired directory. Th

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Tom Lane
Andres Freund writes: > On 2014-01-16 17:47:51 +0200, Heikki Linnakangas wrote: >> In suitable safe places, check if you've written too much WAL, and sleep if >> so. > That'd be most places doing XLogInsert() if you want to be > consistent. See my other response. There's no need for "consistenc

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Andres Freund
On 2014-01-16 11:19:28 -0500, Tom Lane wrote: > > I think the usecases that would want this for DML probably also wan this > > to work for unlogged, temp tables. > > Huh? Unlogged tables generate *zero* WAL, by definition. Yes. That's my point. If we provide it as a generic resource control - wh

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Simon Riggs
On 16 January 2014 16:56, Andres Freund wrote: > On 2014-01-16 17:47:51 +0200, Heikki Linnakangas wrote: >> On 01/16/2014 05:39 PM, Andres Freund wrote: >> >On 2014-01-16 10:35:20 -0500, Tom Lane wrote: >> >>I think possibly a more productive approach to this would be to treat >> >>it as a session

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Andres Freund
On 2014-01-16 10:35:20 -0500, Tom Lane wrote: > Andres Freund writes: > > I don't really see much difficulty in determining what's a utility > > command and what not for the purpose of this? All utility commands which > > create WAL in O(table_size) or worse. > > By that definition, INSERT, UPDAT

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Alvaro Herrera
Stephen Frost wrote: > * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > > Peter Eisentraut wrote: > > > In my mind, a conf.d directory is an extension of a single-file > > > configuration, and so it should be handled that way. > > > > +1 on this. This means > > > > 1. it is to be read automa

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-16 Thread Jeremy Harris
On 14/01/14 22:23, Dave Chinner wrote: On Tue, Jan 14, 2014 at 11:40:38AM -0800, Kevin Grittner wrote: To quantify that, in a production setting we were seeing pauses of up to two minutes with shared_buffers set to 8GB and default dirty ^^

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Tom Lane
Stephen Frost writes: > * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: >> 1. it is to be read automatically by the server without need for an >> "include_dir conf.d" option in the main postgresql.conf. > I am not thrilled with the idea that we're claiming ownership of the > directory which po

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Andres Freund
On 2014-01-16 17:47:51 +0200, Heikki Linnakangas wrote: > On 01/16/2014 05:39 PM, Andres Freund wrote: > >On 2014-01-16 10:35:20 -0500, Tom Lane wrote: > >>I think possibly a more productive approach to this would be to treat > >>it as a session-level GUC setting, rather than hard-wiring it to affe

Re: [HACKERS] Add force option to dropdb

2014-01-16 Thread salah jubeh
>If the user owns objects, that will prevent this from working also.  I >have the feeling that adding DROP OWNED BY and/or REASSIGNED OWNED BY >calls to this utility would be a bit excessive, but who knows. Please find attached the first attempt to drop drop the client connections. I have added

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > Given that the majority didn't seem to be convinced by this and that the > feature was written differently this isn't a convincing argument for the > location of the file given the current feature, no? I'll start by pointing out that the only reaso

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Stephen Frost
* Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > Peter Eisentraut wrote: > > In my mind, a conf.d directory is an extension of a single-file > > configuration, and so it should be handled that way. > > +1 on this. This means > > 1. it is to be read automatically by the server without need fo

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Tom Lane
Heikki Linnakangas writes: > On 01/16/2014 05:39 PM, Andres Freund wrote: >> Do you see a reasonable way to implement this generically for all >> commands? I don't. > In suitable safe places, check if you've written too much WAL, and sleep > if so. Call it CHECK_FOR_WAL_BUDGET(), along the lines

Re: [HACKERS] Display oprcode and its volatility in \do+

2014-01-16 Thread Tom Lane
Marko Tiikkaja writes: > On 1/16/14 4:22 PM, Tom Lane wrote: >> FWIW, I'm on board with the idea of printing the oprcode, but adding >> volatility here seems like probably a waste of valuable terminal width. >> I think that the vast majority of operators have immutable or at worst >> stable underl

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-16 Thread Christian Kruse
Hi Alvaro, On 16/01/14 10:21, Alvaro Herrera wrote: > 1. it is to be read automatically by the server without need for an >"include_dir conf.d" option in the main postgresql.conf. +1 > 4. there is no point in "disabling" it, and thus we offer no mechanism >to do that. Not only there is

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Tom Lane
Andres Freund writes: > I don't really see much difficulty in determining what's a utility > command and what not for the purpose of this? All utility commands which > create WAL in O(table_size) or worse. By that definition, INSERT, UPDATE, and DELETE can all be "utility commands". So would a f

Re: [HACKERS] Display oprcode and its volatility in \do+

2014-01-16 Thread Marko Tiikkaja
On 1/16/14 4:22 PM, Tom Lane wrote: Marko Tiikkaja writes: On 1/16/14 9:53 AM, Rushabh Lathia wrote: Even I personally felt the Function and Volatility is nice to have info into \do+. FWIW, I'm on board with the idea of printing the oprcode, but adding volatility here seems like probably a w

Re: [HACKERS] Display oprcode and its volatility in \do+

2014-01-16 Thread Tom Lane
Marko Tiikkaja writes: > On 1/16/14 9:53 AM, Rushabh Lathia wrote: >> Even I personally felt the Function and Volatility is nice to have info >> into \do+. FWIW, I'm on board with the idea of printing the oprcode, but adding volatility here seems like probably a waste of valuable terminal width.

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2014-01-16 Thread Alvaro Herrera
Euler Taveira wrote: > On 08-11-2013 06:20, Dilip kumar wrote: > > On 08 November 2013 13:38, Jan Lentfer > > > > > >> For this use case, would it make sense to queue work (tables) in order of > >> their size, starting on the largest one? > > > >> For the case where you have tables of varying s

Re: [HACKERS] Changeset Extraction v7.0 (was logical changeset generation)

2014-01-16 Thread Andres Freund
Hi, On 2014-01-16 09:34:51 -0500, Robert Haas wrote: > >> - ReplicationSlotAcquire probably needs to ignore slots that are not > >> active. > > Not sure what you mean? If the slot isn't in_use we'll skip it in the loop. > > active != in_use. > > I suppose your point is that the slot can't be in

Re: [HACKERS] Performance Improvement by reducing WAL for Update Operation

2014-01-16 Thread Robert Haas
On Thu, Jan 16, 2014 at 12:07 AM, Amit Kapila wrote: > On Thu, Jan 16, 2014 at 12:49 AM, Robert Haas wrote: >> On Wed, Jan 15, 2014 at 7:28 AM, Amit Kapila wrote: >>> Unpatched >>> --- >>> testname | wal_generated | >>> duration >>>

Re: [HACKERS] Changeset Extraction v7.0 (was logical changeset generation)

2014-01-16 Thread Robert Haas
On Wed, Jan 15, 2014 at 3:39 PM, Andres Freund wrote: > On 2014-01-15 13:28:25 -0500, Robert Haas wrote: >> - I think you should just regard ReplicationSlotCtlLock as protecting >> the "name" and "active" flags of every slot. ReplicationSlotCreate() >> would then not need to mess with the spinloc

Re: [HACKERS] [PATCH] Relocation of tablespaces in pg_basebackup

2014-01-16 Thread Andres Freund
Hi, On 2014-01-16 11:25:59 -0300, Alvaro Herrera wrote: > Eyeballing this patch, three thoughts: > > 1. I wonder whether ilist.c/h should be moved to src/common so that > frontend code could use it. Sounds like a good idea. There's some debugging checks that elog, but that should be fixable easi

Re: [HACKERS] [PATCH] Relocation of tablespaces in pg_basebackup

2014-01-16 Thread Alvaro Herrera
Eyeballing this patch, three thoughts: 1. I wonder whether ilist.c/h should be moved to src/common so that frontend code could use it. 2. I wonder whether ilist.c should gain the ability to have singly-linked lists with a pointer to the tail node for appending to the end. This code would use it,

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2014-01-16 Thread Euler Taveira
On 08-11-2013 06:20, Dilip kumar wrote: > On 08 November 2013 13:38, Jan Lentfer > > >> For this use case, would it make sense to queue work (tables) in order of >> their size, starting on the largest one? > >> For the case where you have tables of varying size this would lead to a >> reduced

Re: [HACKERS] WAL Rate Limiting

2014-01-16 Thread Andres Freund
On 2014-01-16 09:06:30 -0500, Robert Haas wrote: > > Seems like a really bad name if we are only slowing down some commands - > > that seems to indicate we're slowing down all of them. I think it should be > > something that indicates that it only affects the maintenance commands. > > And why shou

Re: [HACKERS] [PATCH] Relocation of tablespaces in pg_basebackup

2014-01-16 Thread Alvaro Herrera
Please keep the --help and the options in the SGML table in alphabetical order within their respective sections. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgres

  1   2   >