Re: [HACKERS] SSI predicate locking on heap -- tuple or row?

2011-05-26 Thread Heikki Linnakangas
On 26.05.2011 06:19, Kevin Grittner wrote: Dan and I went around a couple times chasing down all code, comment, and patch changes needed, resulting in the attached patch. We found and fixed the bug which originally manifested in a way which I confused with a need for row locks, as well as

Re: [HACKERS] The way to know whether the standby has caught up with the master

2011-05-26 Thread Fujii Masao
On Wed, May 25, 2011 at 3:11 PM, Jaime Casanova ja...@2ndquadrant.com wrote: On Wed, May 25, 2011 at 12:28 AM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, May 25, 2011 at 2:16 PM, Heikki Linnakangas By the time the standby has received that message, it might not be caught-up anymore

Re: [HACKERS] The way to know whether the standby has caught up with the master

2011-05-26 Thread Fujii Masao
On Wed, May 25, 2011 at 11:07 PM, Tom Lane t...@sss.pgh.pa.us wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: On 25.05.2011 07:42, Fujii Masao wrote: To achieve that, I'm thinking to change walsender so that, when the standby has caught up with the master, it sends back

[HACKERS] Re: Latch implementation that wakes on postmaster death on both win32 and Unix

2011-05-26 Thread Peter Geoghegan
I'm a bit disappointed that no one has commented on this yet. I would have appreciated some preliminary feedback. Anyway, I've added it to CommitFest 2011-06. -- Peter Geoghegan       http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via

Re: [HACKERS] Latch implementation that wakes on postmaster death on both win32 and Unix

2011-05-26 Thread Heikki Linnakangas
On 24.05.2011 23:43, Peter Geoghegan wrote: Attached is the latest revision of the latch implementation that monitors postmaster death, plus the archiver client that now relies on that new functionality and thereby works well without a tight PostmasterIsAlive() polling loop. The Unix-stuff

Re: [HACKERS] Proposal: Another attempt at vacuum improvements

2011-05-26 Thread Pavan Deolasee
On Thu, May 26, 2011 at 9:40 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, May 25, 2011 at 11:51 PM, Pavan Deolasee Having said that, it doesn't excite me too much because I think we should do the dead line pointer reclaim operation during page pruning and we are already holding cleanup

[HACKERS] Database research papers

2011-05-26 Thread Pavan Deolasee
Just a trivia. I remember spending weeks on reading the ARIES paper during my post graduation and I loved the depth of knowledge in that paper. In fact, if I re-read it again now, I would appreciate it even more. Are there other papers in the same league, especially which are more closely related

Re: [HACKERS] Latch implementation that wakes on postmaster death on both win32 and Unix

2011-05-26 Thread Peter Geoghegan
On 26 May 2011 11:22, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: The Unix-stuff looks good to me at a first glance. Good. There's one reference left to life sign in comments. (FWIW, I don't have a problem with that terminology myself) Should have caught that one. Removed.

Re: [HACKERS] Latch implementation that wakes on postmaster death on both win32 and Unix

2011-05-26 Thread Dave Page
On Thu, May 26, 2011 at 11:58 AM, Peter Geoghegan pe...@2ndquadrant.com wrote: Attached revision doesn't use any threads or pipes on win32. It's far neater there. I'm still seeing that lagger process (which is an overstatement) at times, so I guess it is normal. On Windows, there is no

Re: [HACKERS] Should partial dumps include extensions?

2011-05-26 Thread Peter Eisentraut
On tis, 2011-05-24 at 23:26 -0400, Robert Haas wrote: On Tue, May 24, 2011 at 4:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: There's a complaint here http://archives.postgresql.org/pgsql-general/2011-05/msg00714.php about the fact that 9.1 pg_dump always dumps CREATE EXTENSION commands for

Re: [HACKERS] Proposal: Another attempt at vacuum improvements

2011-05-26 Thread Pavan Deolasee
On Thu, May 26, 2011 at 4:10 PM, Pavan Deolasee pavan.deola...@gmail.com wrote: On Thu, May 26, 2011 at 9:40 AM, Robert Haas robertmh...@gmail.com wrote: Currently, I believe the only way a page can get marked all-visible is by vacuum.  But if we make this change, then it would be possible for

Re: [HACKERS] Proposal: Another attempt at vacuum improvements

2011-05-26 Thread Robert Haas
On Thu, May 26, 2011 at 6:40 AM, Pavan Deolasee pavan.deola...@gmail.com wrote: There are some other issues that we should think about too. Like recording free space  and managing visibility map. The free space is recorded in the second pass pass today, but I don't see any reason why that

Re: [HACKERS] Proposal: Another attempt at vacuum improvements

2011-05-26 Thread Robert Haas
On Thu, May 26, 2011 at 8:57 AM, Pavan Deolasee pavan.deola...@gmail.com wrote: On Thu, May 26, 2011 at 4:10 PM, Pavan Deolasee pavan.deola...@gmail.com wrote: On Thu, May 26, 2011 at 9:40 AM, Robert Haas robertmh...@gmail.com wrote: Currently, I believe the only way a page can get marked

Re: [HACKERS] Should partial dumps include extensions?

2011-05-26 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: On tis, 2011-05-24 at 23:26 -0400, Robert Haas wrote: On Tue, May 24, 2011 at 4:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: There's a complaint here http://archives.postgresql.org/pgsql-general/2011-05/msg00714.php about the fact that 9.1 pg_dump always

[HACKERS] patch for distinguishing PG instances in event log

2011-05-26 Thread MauMau
Hello, I wrote and attached a patch for the TODO item below (which I proposed). Allow multiple Postgres clusters running on the same machine to distinguish themselves in the event log http://archives.postgresql.org/pgsql-hackers/2011-03/msg01297.php

Re: [HACKERS] [ADMIN] pg_class reltuples/relpages not updated by autovacuum/vacuum

2011-05-26 Thread Tom Lane
Greg Stark gsst...@mit.edu writes: On Wed, May 25, 2011 at 9:41 AM, Tom Lane t...@sss.pgh.pa.us wrote: ... What I'm currently imagining is to do a smoothed moving average, where we factor in the new density estimate with a weight dependent on the percentage of the table we did scan. That is,

Re: [HACKERS] [ADMIN] pg_class reltuples/relpages not updated by autovacuum/vacuum

2011-05-26 Thread Robert Haas
On Thu, May 26, 2011 at 11:25 AM, Tom Lane t...@sss.pgh.pa.us wrote: I'm still of the opinion that an incremental estimation process like the above is a lot saner than what we're doing now, snarky Dilbert references notwithstanding.  The only thing that seems worthy of debate from here is

Re: [HACKERS] errno not set in case of libm functions (HPUX)

2011-05-26 Thread Tom Lane
Ibrar Ahmed ibrar.ah...@gmail.com writes: Please find the updated patch. I have added this +Olibmerrno compile flag check in configure/configure.in file. I tried this on my HP-UX 10.20 box, and it didn't work very nicely: configure decided that the compiler accepted +Olibmerrno, so I got a

Re: [HACKERS] [ADMIN] pg_class reltuples/relpages not updated by autovacuum/vacuum

2011-05-26 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I would feel a lot better about something that is deterministic, like, I dunno, if VACUUM visits more than 25% of the table, we use its estimate. And we always use ANALYZE's estimate. Or something. This argument seems to rather miss the point. The

Re: [HACKERS] LOCK DATABASE

2011-05-26 Thread Ross J. Reedstrom
On Thu, May 19, 2011 at 04:13:12PM -0400, Alvaro Herrera wrote: Excerpts from Robert Haas's message of jue may 19 15:32:57 -0400 2011: That's a bit of a self-defeating argument though, since it implies that the effect of taking an exclusive lock via LockSharedObject() will not simply

Re: [HACKERS] Pre-alloc ListCell's optimization

2011-05-26 Thread Alvaro Herrera
Excerpts from Stephen Frost's message of mar may 24 22:56:21 -0400 2011: Greetings, Someone (*cough*Haas*cough) made a claim over beers at PGCon that it would be very difficult to come up with a way to pre-allocate List entries and maintain the current List API. I'll admit that it

Re: [HACKERS] about EDITOR_LINENUMBER_SWITCH

2011-05-26 Thread Alvaro Herrera
Excerpts from Tom Lane's message of mié may 25 16:07:55 -0400 2011: Alvaro Herrera alvhe...@commandprompt.com writes: Excerpts from Tom Lane's message of mar may 24 17:11:17 -0400 2011: Right. It would also increase the cognitive load on the user to have to remember the command-line

Re: [HACKERS] Pre-alloc ListCell's optimization

2011-05-26 Thread Stephen Frost
* Alvaro Herrera (alvhe...@commandprompt.com) wrote: I think what this patch is mainly missing is a description of how the new allocation is supposed to work, so that we can discuss the details without having to reverse-engineer them from the code. Sure, sorry I didn't include something more

Re: [HACKERS] [ADMIN] pg_class reltuples/relpages not updated by autovacuum/vacuum

2011-05-26 Thread Robert Haas
On Thu, May 26, 2011 at 12:23 PM, Tom Lane t...@sss.pgh.pa.us wrote: Another thought: Couldn't relation_needs_vacanalyze() just scale up reltuples by the ratio of the current number of pages in the relation to relpages, just as the query planner does? Hmm ... that would fix Florian's

Re: [HACKERS] LOCK DATABASE

2011-05-26 Thread Robert Haas
On Thu, May 26, 2011 at 12:28 PM, Ross J. Reedstrom reeds...@rice.edu wrote: Perhaps the approach to restricting connections should not be a database object lock, but rather an admin function that does the equivalent of flipping datallowconn in pg_database? To me, that seems like a better

Re: [HACKERS] Pre-alloc ListCell's optimization

2011-05-26 Thread Tom Lane
Stephen Frost sfr...@snowman.net writes: Basically, I added a ListCell array into the List structure and then added a bitmap to keep track of which positions in the array were filled. Hm. I've gotten the impression from previous testing that there are an awful lot of extremely short lists (1

Re: [HACKERS] [ADMIN] pg_class reltuples/relpages not updated by autovacuum/vacuum

2011-05-26 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: I think we should really consider replacing reltuples with reltupledensity at some point. I continue to be afraid that using a decaying average in this case is going to end up overweighting the values from some portion of the table that's getting

Re: [HACKERS] timezone GUC

2011-05-26 Thread Alvaro Herrera
Excerpts from Robert Haas's message of dom may 22 23:09:47 -0400 2011: On Sun, May 22, 2011 at 10:24 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Sun, May 22, 2011 at 9:54 PM, Tom Lane t...@sss.pgh.pa.us wrote: But also, 99.999% of the time it would

Re: [HACKERS] [ADMIN] pg_class reltuples/relpages not updated by autovacuum/vacuum

2011-05-26 Thread Robert Haas
On Thu, May 26, 2011 at 1:28 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: I think we should really consider replacing reltuples with reltupledensity at some point.  I continue to be afraid that using a decaying average in this case is going to

Re: [HACKERS] Pre-alloc ListCell's optimization

2011-05-26 Thread Robert Haas
On Tue, May 24, 2011 at 10:56 PM, Stephen Frost sfr...@snowman.net wrote:  Someone (*cough*Haas*cough) made a claim over beers at PGCon that it  would be very difficult to come up with a way to pre-allocate List  entries and maintain the current List API.  I'll admit that it wasn't  quite as

Re: [HACKERS] [ADMIN] pg_class reltuples/relpages not updated by autovacuum/vacuum

2011-05-26 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: Given how trivial it would be to adjust reltuples to keep its ratio to relpages about the same when we don't have a new hard number, but some evidence that we should fudge our previous value, I don't

[HACKERS] #PgWest 2011: CFP now open

2011-05-26 Thread Joshua D. Drake
Hello, The CFP for #PgWest is now open. We are holding it at the San Jose Convention Center from September 27th - 30th. We look forward to seeing your submissions. http://www.postgresqlconference.org/ Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL

Re: [HACKERS] Pre-alloc ListCell's optimization

2011-05-26 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: I'm worried that this type of approach would bloat the storage required in those cases to a degree that would make the patch unattractive. While I agree that there is some bloat that'll happen with this approach, we could reduce it by just having a

Re: [HACKERS] SSI predicate locking on heap -- tuple or row?

2011-05-26 Thread Kevin Grittner
Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Could you explain in the README, why it is safe to only take the lock on the visible row version, please? Sure. I actually intended to do this last night but ran out of steam and posted what I had, planning on following up with

Re: [HACKERS] inconvenient compression options in pg_basebackup

2011-05-26 Thread Peter Eisentraut
On tis, 2011-05-24 at 15:34 -0400, Tom Lane wrote: I would argue that -Z ought to turn on gzip without my having to write -z as well (at least when the argument is greater than zero; possibly -Z0 should be allowed as meaning no compression). My concern with that is that if we ever add

Re: [HACKERS] [ADMIN] pg_class reltuples/relpages not updated by autovacuum/vacuum

2011-05-26 Thread Robert Haas
On Thu, May 26, 2011 at 2:05 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: I'm a bit confused by this - what the current design obfuscates is the fact that reltuples and relpages are not really independent columns; you can't update one without updating the other, unless you want screwy

Re: [HACKERS] [ADMIN] pg_class reltuples/relpages not updated by autovacuum/vacuum

2011-05-26 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: Except that's not how it works. At least in the case of ANALYZE, we *aren't* counting all the tuples in the table. We're selecting a random sample of pages and inferring a tuple density, which we then extrapolate to the whole table and store. Then

Re: [HACKERS] [ADMIN] pg_class reltuples/relpages not updated by autovacuum/vacuum

2011-05-26 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, May 26, 2011 at 12:23 PM, Tom Lane t...@sss.pgh.pa.us wrote: Another thought: Couldn't relation_needs_vacanalyze() just scale up reltuples by the ratio of the current number of pages in the relation to relpages, just as the query planner does?

Re: [HACKERS] inconvenient compression options in pg_basebackup

2011-05-26 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: On tis, 2011-05-24 at 15:34 -0400, Tom Lane wrote: I would argue that -Z ought to turn on gzip without my having to write -z as well (at least when the argument is greater than zero; possibly -Z0 should be allowed as meaning no compression). My

[HACKERS] pg_basebackup compressed tar to stdout

2011-05-26 Thread Peter Eisentraut
pg_basebackup currently doesn't allow compressed tar to stdout. That should be added to make the interface consistent, and specifically to allow common idoms like pg_basebackup -Ft -z -D - | ssh tar -x -z -f - Small patch attached. diff --git i/doc/src/sgml/ref/pg_basebackup.sgml

Re: [HACKERS] inconvenient compression options in pg_basebackup

2011-05-26 Thread Peter Eisentraut
On tor, 2011-05-26 at 16:54 -0400, Tom Lane wrote: But if you want to take such an extension into account right now, maybe we ought to design that feature now. What are you seeing it as looking like? My thought is that -z should just mean give me compression; a good default compression

Re: [HACKERS] pg_basebackup compressed tar to stdout

2011-05-26 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: pg_basebackup currently doesn't allow compressed tar to stdout. That should be added to make the interface consistent, and specifically to allow common idoms like pg_basebackup -Ft -z -D - | ssh tar -x -z -f - Small patch attached. I have not

Re: [HACKERS] pg_basebackup compressed tar to stdout

2011-05-26 Thread Peter Eisentraut
On tor, 2011-05-26 at 17:06 -0400, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: pg_basebackup currently doesn't allow compressed tar to stdout. That should be added to make the interface consistent, and specifically to allow common idoms like pg_basebackup -Ft -z -D - | ssh

Re: [HACKERS] inconvenient compression options in pg_basebackup

2011-05-26 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: On tor, 2011-05-26 at 16:54 -0400, Tom Lane wrote: But if you want to take such an extension into account right now, maybe we ought to design that feature now. What are you seeing it as looking like? My thought is that -z should just mean give me

Re: [HACKERS] errno not set in case of libm functions (HPUX)

2011-05-26 Thread Peter Eisentraut
On tor, 2011-05-26 at 12:14 -0400, Tom Lane wrote: Ibrar Ahmed ibrar.ah...@gmail.com writes: Please find the updated patch. I have added this +Olibmerrno compile flag check in configure/configure.in file. I tried this on my HP-UX 10.20 box, and it didn't work very nicely: configure

Re: [HACKERS] [ADMIN] pg_class reltuples/relpages not updated by autovacuum/vacuum

2011-05-26 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: By storing the ratio and one count you make changes to the other count implied and less visible. It seems more understandable and less prone to error (to me, anyway) to keep the two raw numbers and

Re: [HACKERS] errno not set in case of libm functions (HPUX)

2011-05-26 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: On tor, 2011-05-26 at 12:14 -0400, Tom Lane wrote: I tried this on my HP-UX 10.20 box, and it didn't work very nicely: configure decided that the compiler accepted +Olibmerrno, so I got a compile full of cc: warning 450: Unrecognized option

Re: [HACKERS] [ADMIN] pg_class reltuples/relpages not updated by autovacuum/vacuum

2011-05-26 Thread Tom Lane
Kevin Grittner kevin.gritt...@wicourts.gov writes: When we prune or vacuum a page, I don't suppose we have enough information about that page's previous state to calculate a tuple count delta, do we? That would allow a far more accurate number to be maintained than anything suggested so far,

Re: [HACKERS] LOCK DATABASE

2011-05-26 Thread Michael Paquier
Hi all, On Fri, May 27, 2011 at 2:13 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, May 26, 2011 at 12:28 PM, Ross J. Reedstrom reeds...@rice.edu wrote: Perhaps the approach to restricting connections should not be a database object lock, but rather an admin function that does the

Re: [HACKERS] patch for new feature: Buffer Cache Hibernation

2011-05-26 Thread Greg Smith
On 05/07/2011 03:32 AM, Mitsuru IWASAKI wrote: For 1, I've just finish my work. The latest patch is available at: http://people.freebsd.org/~iwasaki/postgres/buffer-cache-hibernation-postgresql-20110507.patch Reminder here--we can't accept code based on it being published to a web page.

Re: [HACKERS] Pre-alloc ListCell's optimization

2011-05-26 Thread Greg Stark
On Thu, May 26, 2011 at 11:57 AM, Stephen Frost sfr...@snowman.net wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: I'm worried that this type of approach would bloat the storage required in those cases to a degree that would make the patch unattractive. While I agree that there is some bloat

Re: [HACKERS] Expression Evaluator used for creating the plan tree / stmt ?

2011-05-26 Thread Vaibhav Kaushal
OK, I ran a GDB trace into ExecScan and here is a part of it: # (gdb) finish Run till exit from #0 ExecScanFetch (node=0x1d5c3c0, accessMtd=0x55dd10 SeqNext, recheckMtd=0x55db70 SeqRecheck) at execScan.c:44 194 if (TupIsNull(slot)) (gdb) s 205

Re: [HACKERS] Pre-alloc ListCell's optimization

2011-05-26 Thread Greg Stark
On Thu, May 26, 2011 at 11:57 AM, Stephen Frost sfr...@snowman.net wrote: Handling the 1-entry case would likely be pretty straight-forward, but you need book-keeping as soon as you go to two, and all that book-keeping feels like overkill for just a 2-entry cache to me. Incidentally what if I

Re: [HACKERS] Expression Evaluator used for creating the plan tree / stmt ?

2011-05-26 Thread Tom Lane
Vaibhav Kaushal vaibhavkaushal...@gmail.com writes: Why do these lines: ... repeat twice? Hm, you're new to using gdb, no? That's pretty normal: gdb is just reflecting back the fact that the compiler rearranges individual instructions as it sees fit. You could eliminate most, though perhaps

Re: [HACKERS] Pre-alloc ListCell's optimization

2011-05-26 Thread Stephen Frost
* Greg Stark (gsst...@mit.edu) wrote: On Thu, May 26, 2011 at 11:57 AM, Stephen Frost sfr...@snowman.net wrote: Handling the 1-entry case would likely be pretty straight-forward, but you need book-keeping as soon as you go to two, and all that book-keeping feels like overkill for just a

Re: [HACKERS] Expression Evaluator used for creating the plan tree / stmt ?

2011-05-26 Thread Vaibhav Kaushal
Thanks Tom. Comparing to you people, I am definitely new to almost everything here. I did debug a few smaller programs and never seen anything as such. So asked. Moreover, those programs I compiled never used any optimization. While everything seems to be working, it looks like the slot values do

Re: [HACKERS] Pre-alloc ListCell's optimization

2011-05-26 Thread Greg Stark
On Thu, May 26, 2011 at 8:52 PM, Stephen Frost sfr...@snowman.net wrote:  list_concat() does explicitly say that cells will be shared afterwards and that you can't pfree() either list (note that there's actually a couple cases currently that I discovered which were also addressed in the

Re: [HACKERS] Pre-alloc ListCell's optimization

2011-05-26 Thread Stephen Frost
* Greg Stark (gsst...@mit.edu) wrote: On Thu, May 26, 2011 at 11:57 AM, Stephen Frost sfr...@snowman.net wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: While I agree that there is some bloat that'll happen with this approach, we could reduce it by just having a 4-entry cache instead of an

Re: [HACKERS] Pre-alloc ListCell's optimization

2011-05-26 Thread Stephen Frost
Greg, * Greg Stark (gsst...@mit.edu) wrote: On Thu, May 26, 2011 at 8:52 PM, Stephen Frost sfr...@snowman.net wrote:  list_concat() does explicitly say that cells will be shared afterwards and that you can't pfree() either list (note that there's actually a couple cases currently that I