Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-24 Thread Atri Sharma
Perhaps this isn't the help you were looking for, but I spent a long time looking into this a few years ago. Then I stopped and decided to work on other things. I would recommend you do so too. Agreed. It seems that my concerns were not valid, and since you have already done some testing

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-24 Thread Atri Sharma
I'll have to take a look. Removing *all spinning* from from page allocation though feels like it might be worthwhile to test (got to give some bonus points for being a very local change and simple to implement). I wonder if with more shared buffers you tend to sweep more buffers per

Re: [HACKERS] Request for Patch Feedback: Lag Lead Window Functions Can Ignore Nulls

2013-03-24 Thread Hitoshi Harada
On Sat, Mar 23, 2013 at 3:25 PM, Nicholas White n.j.wh...@gmail.com wrote: Thanks - I've added it here: https://commitfest.postgresql.org/action/patch_view?id=1096 . I've also attached a revised version that makes IGNORE and RESPECT UNRESERVED keywords (following the pattern of NULLS_FIRST

Re: [HACKERS] Problem with background worker

2013-03-24 Thread Guillaume Lelarge
On Wed, 2013-03-20 at 13:13 -0300, Alvaro Herrera wrote: Marc Cousin escribió: On 20/03/2013 16:33, Alvaro Herrera wrote: Ah. The reason for this problem is that the statement start time (which also sets the transaction start time, when it's the first statement) is set by postgres.c,

[HACKERS] Limiting setting of hint bits by read-only queries; vacuum_delay

2013-03-24 Thread Simon Riggs
vacuum_delay is designed to slow down VACUUMs from writing too many blocks. However, SELECTs also dirty data blocks but are NOT slowed down by vacuum_delay. So the current situation is that a large SELECT operates similarly to a VACUUM, throwing out many dirty blocks and using additional I/O

[HACKERS] [proof of concept] Evolving postgresql.conf using genetic algorithm

2013-03-24 Thread Greg Jaskiewicz
Hi folks, I've always been fascinated with genetic algorithms. Having had a chance to implement it once before, to solve real life issue - I knew they can be brilliant at searching for right solutions in multi dimensional space. Thinking about just the postgresql.conf and number of possible

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-24 Thread Ants Aasma
On Sun, Mar 24, 2013 at 9:29 AM, Atri Sharma atri.j...@gmail.com wrote: I'll have to take a look. Removing *all spinning* from from page allocation though feels like it might be worthwhile to test (got to give some bonus points for being a very local change and simple to implement). I wonder

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-24 Thread Greg Smith
On 3/22/13 8:45 AM, Ants Aasma wrote: However, I think the main issue isn't finding new algorithms that are better in some specific circumstances. The hard part is figuring out whether their performance is better in general. Right. The current page replacement method works as expected. Many

Re: [HACKERS] Limiting setting of hint bits by read-only queries; vacuum_delay

2013-03-24 Thread Greg Stark
On Sun, Mar 24, 2013 at 11:14 AM, Simon Riggs si...@2ndquadrant.com wrote: Proposal is to prevent SELECTs from causing more than N buffers from being dirtied by hint bit setting and block cleanup. I think you need to clarify what you mean by this. I'm guessing you mean that there would be a

Re: [HACKERS] Limiting setting of hint bits by read-only queries; vacuum_delay

2013-03-24 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: Can we do this now? No. We are trying to get 9.3 out the door, not undertake design, coding, and testing of entirely new behaviors which will have complex performance tradeoffs. This sounds like a great project for early in the 9.4 development cycle,

Re: [HACKERS] Let's invent a function to report lock-wait-blocking PIDs

2013-03-24 Thread Greg Smith
On 3/20/13 2:02 PM, Tom Lane wrote: If isolationtester were the only market for this type of information, maybe it wouldn't be worth worrying about. But I'm pretty sure that there are a *lot* of monitoring applications out there that are trying to extract who-blocks-whom information from

[HACKERS] parallel pg_dump

2013-03-24 Thread Andrew Dunstan
I have committed this patch, after a good deal of review and testing, including on Windows. While I did some editorializing, there is more work to do, particularly on improving comments and messages, but I wanted to get this in as time is getting very short, and it's important that we not

[HACKERS] Interesting post-mortem on a near disaster with git

2013-03-24 Thread Tom Lane
Over the weekend, KDE came within a gnat's eyelash of losing *all* their authoritative git repos, despite having seemingly-extensive redundancy. Read about it here: http://jefferai.org/2013/03/24/too-perfect-a-mirror/ We should think about protecting our own repo a bit better, especially after

Re: [HACKERS] Interesting post-mortem on a near disaster with git

2013-03-24 Thread Martijn van Oosterhout
On Sun, Mar 24, 2013 at 11:52:17AM -0400, Tom Lane wrote: Over the weekend, KDE came within a gnat's eyelash of losing *all* their authoritative git repos, despite having seemingly-extensive redundancy. Read about it here: http://jefferai.org/2013/03/24/too-perfect-a-mirror/ We should

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-24 Thread Atri Sharma
On Sun, Mar 24, 2013 at 6:11 AM, Greg Smith g...@2ndquadrant.com wrote: On 3/22/13 8:45 AM, Ants Aasma wrote: However, I think the main issue isn't finding new algorithms that are better in some specific circumstances. The hard part is figuring out whether their performance is better in

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-24 Thread Atri Sharma
If we use the value calculated locally to decide on eviction, the highly used buffers where this is likely will get at least one clock sweep cycle of grace time. If they are indeed highly used it's likely that someone will manage to bump the usage_count in the meanwhile. If they are not hot,

Re: [HACKERS] pkg-config files for libpq and ecpg

2013-03-24 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: On 1/15/13 6:53 PM, Tom Lane wrote: This bit: + echo 'Libs.private: $(filter-out $(PKG_CONFIG_REQUIRES_PRIVATE:lib%=-l%),$(filter-out -L..%, $(SHLIB_LINK)))' $@ appears to assume that SHLIB_LINK contains nothing except -L and -l switches.

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-24 Thread Ants Aasma
On Sun, Mar 24, 2013 at 7:03 PM, Atri Sharma atri.j...@gmail.com wrote: So, essentially, we decide locally which page to evict, then try to get a lock on the header only when we want to evict the victim page? Sounds like the contention for the header should go down considerably. Not exactly

Re: [HACKERS] Interesting post-mortem on a near disaster with git

2013-03-24 Thread Stefan Kaltenbrunner
On 03/24/2013 05:08 PM, Martijn van Oosterhout wrote: On Sun, Mar 24, 2013 at 11:52:17AM -0400, Tom Lane wrote: Over the weekend, KDE came within a gnat's eyelash of losing *all* their authoritative git repos, despite having seemingly-extensive redundancy. Read about it here:

Re: [HACKERS] patch to add \watch to psql

2013-03-24 Thread Tom Lane
Daniel Farina dan...@heroku.com writes: The name \repeat has grown on me, but I haven't bothered renaming it for the time being. I think sameness with the familiar 'watch' program may not be such a big deal as I thought originally, but 'repeat' sounds a lot more like a kind of flow control

Re: [HACKERS] Limiting setting of hint bits by read-only queries; vacuum_delay

2013-03-24 Thread Simon Riggs
On 24 March 2013 13:29, Greg Stark st...@mit.edu wrote: On Sun, Mar 24, 2013 at 11:14 AM, Simon Riggs si...@2ndquadrant.com wrote: Proposal is to prevent SELECTs from causing more than N buffers from being dirtied by hint bit setting and block cleanup. I'm guessing you mean that there would

Re: [HACKERS] Limiting setting of hint bits by read-only queries; vacuum_delay

2013-03-24 Thread Simon Riggs
On 24 March 2013 13:38, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Can we do this now? No. We are trying to get 9.3 out the door, not undertake design, coding, and testing of entirely new behaviors which will have complex performance tradeoffs. This

Re: [HACKERS] WIP: index support for regexp search

2013-03-24 Thread Tom Lane
Alexander Korotkov aekorot...@gmail.com writes: Now I have working implemetation of this API. Comments still need rework. Could you give me any feedback? I looked at this a little bit, but it's not very far along at all towards resolving my API worries. The basic point that I'm concerned about

Re: [HACKERS] Interesting post-mortem on a near disaster with git

2013-03-24 Thread Michael Paquier
On Mon, Mar 25, 2013 at 12:52 AM, Tom Lane t...@sss.pgh.pa.us wrote: Over the weekend, KDE came within a gnat's eyelash of losing *all* their authoritative git repos, despite having seemingly-extensive redundancy. Read about it here: http://jefferai.org/2013/03/24/too-perfect-a-mirror/ It

Re: [HACKERS] Limiting setting of hint bits by read-only queries; vacuum_delay

2013-03-24 Thread Greg Stark
On Sun, Mar 24, 2013 at 9:02 PM, Simon Riggs si...@2ndquadrant.com wrote: ...and as a result, the rest of your comments don't apply at all to the proposal. Sorry about that confusion. How do you figure that? -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] Interesting post-mortem on a near disaster with git

2013-03-24 Thread Andrew Dunstan
On 03/24/2013 06:06 PM, Michael Paquier wrote: On Mon, Mar 25, 2013 at 12:52 AM, Tom Lane t...@sss.pgh.pa.us mailto:t...@sss.pgh.pa.us wrote: Over the weekend, KDE came within a gnat's eyelash of losing *all* their authoritative git repos, despite having seemingly-extensive

Re: [HACKERS] WIP: index support for regexp search

2013-03-24 Thread Alexander Korotkov
On Mon, Mar 25, 2013 at 1:50 AM, Tom Lane t...@sss.pgh.pa.us wrote: Alexander Korotkov aekorot...@gmail.com writes: Now I have working implemetation of this API. Comments still need rework. Could you give me any feedback? I looked at this a little bit, but it's not very far along at all

Re: [HACKERS] WIP: index support for regexp search

2013-03-24 Thread Tom Lane
Alexander Korotkov aekorot...@gmail.com writes: On Mon, Mar 25, 2013 at 1:50 AM, Tom Lane t...@sss.pgh.pa.us wrote: Similarly, pushing PG-specific declarations like RE_compile_and_cache() into regex/regex.h is completely not the right thing for preserving a clear library boundary (even

Re: [HACKERS] Request for Patch Feedback: Lag Lead Window Functions Can Ignore Nulls

2013-03-24 Thread Nicholas White
Thanks for the feedback. For the parsing changes, it seems I can either make RESPECT and IGNORE reserved keywords, or add a lookahead to construct synthetic RESPECT NULLS and IGNORE NULLS keywords. The grammar wouldn't compile if RESPECT and IGNORE were just normal unreserved keywords due to

Re: [PATCH] Exorcise zero-dimensional arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-03-24 Thread Josh Berkus
On 03/20/2013 04:45 PM, Brendan Jurd wrote: Incompatibility: This patch introduces an incompatible change in the behaviour of the aforementioned array functions -- instead of returning NULL for empty arrays they return meaningful values. Applications relying on the old behaviour to test for

Re: [HACKERS] Limiting setting of hint bits by read-only queries; vacuum_delay

2013-03-24 Thread Greg Smith
On 3/24/13 7:14 AM, Simon Riggs wrote: Patch to implement is a few hours work. The only complexity is deciding how to handle SQL in functions to which I would say, as simply as possible. Like the Page replacement ideas, the throttle on how fast something like this will get done depends

[HACKERS] adding support for zero-attribute unique/etc keys

2013-03-24 Thread Darren Duncan
From my usage and http://www.postgresql.org/docs/9.2/interactive/sql-createtable.html I see that Postgres requires constraints like unique (and primary) keys, and foreign keys, to range over at least 1 attribute/column. I propose that this be generalized so that constraints may instead be

Re: [HACKERS] Limiting setting of hint bits by read-only queries; vacuum_delay

2013-03-24 Thread Simon Riggs
On 25 March 2013 02:50, Greg Smith g...@2ndquadrant.com wrote: On 3/24/13 7:14 AM, Simon Riggs wrote: Patch to implement is a few hours work. The only complexity is deciding how to handle SQL in functions to which I would say, as simply as possible. Like the Page replacement ideas, the

Re: [HACKERS] Review of Row Level Security

2013-03-24 Thread Simon Riggs
On 19 March 2013 15:08, Kohei KaiGai kai...@kaigai.gr.jp wrote: It is much helpful if someone give me comments around these arguable portions from the standpoint with fresh eyes. My feeling at this point is that I don't think I should commit this and related patches without more work. I

Re: [HACKERS] Request for vote to move forward with recovery.conf overhaul

2013-03-24 Thread Simon Riggs
On 25 March 2013 04:08, Michael Paquier michael.paqu...@gmail.com wrote: Feedback is warmly welcome. I'll look at this in the coming week. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers

Re: [PATCH] Exorcise zero-dimensional arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-03-24 Thread Brendan Jurd
On 25 March 2013 13:02, Josh Berkus j...@agliodbs.com wrote: On 03/20/2013 04:45 PM, Brendan Jurd wrote: Incompatibility: This patch introduces an incompatible change in the behaviour of the aforementioned array functions -- instead of returning NULL for empty arrays they return meaningful