Re: [HACKERS] WAL replay bugs

2014-04-17 Thread Peter Geoghegan
On Thu, Apr 17, 2014 at 10:33 AM, Tom Lane wrote: >> Any objections to changing those two? > > Not here. I've always suspected #2 was going to bite us someday anyway. +1 -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscri

Re: [HACKERS] assertion failure 9.3.4

2014-04-17 Thread Andrew Dunstan
On 04/17/2014 09:04 PM, Peter Geoghegan wrote: On Thu, Apr 17, 2014 at 7:15 AM, Andrew Dunstan wrote: track_activity_query_size = 10240 shared_preload_libraries = 'auto_explain,pg_stat_statements' As you can see, auto_explain's log_min_duration hasn't been set, so it shouldn't be doing

Re: [HACKERS] DISCARD ALL (Again)

2014-04-17 Thread David Johnston
On Thursday, April 17, 2014, Joshua D. Drake wrote: > > On 04/17/2014 07:07 PM, David G Johnston wrote: > >> >> On 04/17/2014 05:24 PM, Tom Lane wrote: >> > On the whole I'm not sure this is something we ought to get into. >> > If you really need a fresh session, maybe you should st

Re: [HACKERS] DISCARD ALL (Again)

2014-04-17 Thread Joshua D. Drake
On 04/17/2014 07:07 PM, David G Johnston wrote: On 04/17/2014 05:24 PM, Tom Lane wrote: > On the whole I'm not sure this is something we ought to get into. > If you really need a fresh session, maybe you should start a > fresh session. Isn't the whole point to avoid the rec

Re: [HACKERS] DISCARD ALL (Again)

2014-04-17 Thread Stephen Frost
* David G Johnston (david.g.johns...@gmail.com) wrote: > > On 04/17/2014 05:24 PM, Tom Lane wrote: > > > On the whole I'm not sure this is something we ought to get into. > > > If you really need a fresh session, maybe you should start a > > > fresh session. > > > Isn't the whole point to avoid the

Re: [HACKERS] DISCARD ALL (Again)

2014-04-17 Thread David G Johnston
> On 04/17/2014 05:24 PM, Tom Lane wrote: > > On the whole I'm not sure this is something we ought to get into. > > If you really need a fresh session, maybe you should start a > > fresh session. > > Isn't the whole point to avoid the reconnection overhead, especially for connection poolers? DISCA

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Peter Geoghegan
On Thu, Apr 17, 2014 at 6:50 PM, Greg Stark wrote: > On Thu, Apr 17, 2014 at 4:48 PM, Peter Geoghegan wrote: >> Although I will add that not caching highly useful inner pages for the >> medium term, because that index isn't being used at all for 5 minutes >> probably is very bad. Using the 4,828

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Greg Stark
On Thu, Apr 17, 2014 at 4:48 PM, Peter Geoghegan wrote: > Although I will add that not caching highly useful inner pages for the > medium term, because that index isn't being used at all for 5 minutes > probably is very bad. Using the 4,828 buffers that it would take to > store all the inner pages

Re: [HACKERS] [bug fix] pg_ctl always uses the same event source

2014-04-17 Thread Amit Kapila
On Sat, Apr 12, 2014 at 1:21 PM, MauMau wrote: > Hello, Amit san, Tom san, > > I'm sorry for my late response. I've just caught up with the discussion. > I'm almost convinced. > > Please find attached the revised patch. I'd like to follow the idea of > adding a switch to pg_ctl. The newly added

Re: [HACKERS] How can we make beta testing better?

2014-04-17 Thread Greg Stark
On Thu, Apr 17, 2014 at 5:26 PM, Jeff Janes wrote: > A lot of the bugs that turned up are not the kind I would expect to have > been found in most beta testing done by non-hacking users. Weren't they > mostly around rare race conditions, crash recovery, and freezing? Actually I was struck by how

Re: [HACKERS] Buildfarm "master-next" branch?

2014-04-17 Thread Craig Ringer
On 04/17/2014 10:38 PM, Tom Lane wrote: > IMO the best single thing that could happen for the buildfarm is if > we had more critters (at least twice as many) running a wider variety of > platforms, compilers, and configuration options than there are today. > More frequent runs would come out of tha

Re: [HACKERS] assertion failure 9.3.4

2014-04-17 Thread Peter Geoghegan
On Thu, Apr 17, 2014 at 7:15 AM, Andrew Dunstan wrote: > track_activity_query_size = 10240 >shared_preload_libraries = 'auto_explain,pg_stat_statements' > > As you can see, auto_explain's log_min_duration hasn't been set, so it > shouldn't be doing anything very much, I should think. track_a

Re: [HACKERS] DISCARD ALL (Again)

2014-04-17 Thread Tom Lane
"Joshua D. Drake" writes: > On 04/17/2014 05:24 PM, Tom Lane wrote: >> 1. The things that DISCARD ALL resets are explicitly enumerated in its >> documentation page; it is not an open-ended promise to clean up anything >> anybody happens to think of. > Actually, it is. Unless we change the wording

Re: [HACKERS] DISCARD ALL (Again)

2014-04-17 Thread Joshua D. Drake
On 04/17/2014 05:24 PM, Tom Lane wrote: "Joshua D. Drake" writes: Does it seem reasonable based on the docs: DISCARD ALL: Releases all temporary resources associated with the current session and resets the session to its initial state. That we should also release the GD? There are a couple o

Re: [HACKERS] DISCARD ALL (Again)

2014-04-17 Thread Tom Lane
"Joshua D. Drake" writes: > Does it seem reasonable based on the docs: > DISCARD ALL: > Releases all temporary resources associated with the current session and > resets the session to its initial state. > That we should also release the GD? There are a couple of reasons why this isn't anywhere

Re: [HACKERS] Verbose output of pg_dump not show schema name

2014-04-17 Thread Michael Paquier
On Fri, Apr 18, 2014 at 4:29 AM, Fabrízio de Royes Mello wrote: > > On Thu, Apr 17, 2014 at 12:46 PM, Bruce Momjian wrote: >> >> On Thu, Apr 17, 2014 at 11:44:37AM -0400, Tom Lane wrote: >> > Bruce Momjian writes: >> > > The idea is that we only need quotes when there are odd characters in >> >

Re: [HACKERS] assertion failure 9.3.4

2014-04-17 Thread Josh Berkus
All, So have encountered a 2nd report of this issue, or of an issue which sounds very similar: - corruption in two "queue" tables - the tables are written in a high-concurrency, lock-contested environment - user uses SELECT FOR UPDATE with these tables. - pg_stat_statements .so is loaded, but

Re: [HACKERS] How can we make beta testing better?

2014-04-17 Thread Josh Berkus
On 04/15/2014 09:53 PM, Rod Taylor wrote: > A documented beta test process/toolset which does the following would help: > 1) Enables full query logging > 2) Creates a replica of a production DB, record $TIME when it stops. > 3) Allow user to make changes (upgrade to 9.4, change hardware, change > k

Re: [HACKERS] DISCARD ALL (Again)

2014-04-17 Thread Fabrízio de Royes Mello
On Thu, Apr 17, 2014 at 6:51 PM, Alvaro Herrera wrote: > > It does sounds a legitimate feature request to me. I don't remember if > we honored the request to add resetting of cached sequences, though; if > we didn't, this one is probably going to be tough too. > +1 > Another point is that to i

Re: [HACKERS] DISCARD ALL (Again)

2014-04-17 Thread David G Johnston
Alvaro Herrera-9 wrote > Are we going to backpatch a doc change that says "releases all temporary > resources, except for plptyhon's and plperl's GD"? Surely not ... GD = Global Dictionary I don't see why something like the following wouldn't have value. "For those languages that make use of a

Re: [HACKERS] DISCARD ALL (Again)

2014-04-17 Thread Alvaro Herrera
Joshua D. Drake wrote: > > On 04/17/2014 02:17 PM, Josh Berkus wrote: > > > >On 04/17/2014 01:44 PM, Joshua D. Drake wrote: > >>Does it seem reasonable based on the docs: > >> > >>DISCARD ALL: > >> > >>Releases all temporary resources associated with the current session and > >>resets the session

Re: [HACKERS] DISCARD ALL (Again)

2014-04-17 Thread Joshua D. Drake
On 04/17/2014 02:17 PM, Josh Berkus wrote: On 04/17/2014 01:44 PM, Joshua D. Drake wrote: Does it seem reasonable based on the docs: DISCARD ALL: Releases all temporary resources associated with the current session and resets the session to its initial state. That we should also release the

Re: [HACKERS] How can we make beta testing better?

2014-04-17 Thread Jan Wieck
On 04/17/14 15:16, Merlin Moncure wrote: On Tue, Apr 15, 2014 at 4:47 PM, Josh Berkus wrote: Hackers, I think 9.3 has given us evidence that our users aren't giving new versions of PostgreSQL substantial beta testing, or if they are, they aren't sharing the results with us. How can we make be

Re: [HACKERS] How can we make beta testing better?

2014-04-17 Thread Jeff Janes
On Tue, Apr 15, 2014 at 2:47 PM, Josh Berkus wrote: > Hackers, > > I think 9.3 has given us evidence that our users aren't giving new > versions of PostgreSQL substantial beta testing, or if they are, they > aren't sharing the results with us. > A lot of the bugs that turned up are not the kind

Re: [HACKERS] Patch: iff -> if

2014-04-17 Thread Andreas 'ads' Scherbaum
On 04/17/2014 12:33 PM, Nicolas Barbier wrote: 2014-04-17 Michael Paquier : Is there no equivalent in German? For example in French there is "ssi". gdw (genau dann, wenn) More likely that you see \equiv or: \leftrightarrow Regards, -- Andreas 'ads' Sc

Re: [HACKERS] DISCARD ALL (Again)

2014-04-17 Thread Josh Berkus
On 04/17/2014 01:44 PM, Joshua D. Drake wrote: > Does it seem reasonable based on the docs: > > DISCARD ALL: > > Releases all temporary resources associated with the current session and > resets the session to its initial state. > > That we should also release the GD? It does, but that's a feat

Re: [HACKERS] assertion in 9.4 with wal_level=logical

2014-04-17 Thread Steve Singer
On 04/17/2014 04:33 PM, Andres Freund wrote: Hi, On 2014-04-17 16:23:54 -0400, Steve Singer wrote: With master/9.4 from today (52e757420fa98a76015c2c88432db94269f3e8f4) I am getting an assertion when doing a truncate via SPI when I have wal_level=logical. Stack trace is below. I am just repl

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Peter Geoghegan
On Thu, Apr 17, 2014 at 1:33 PM, Peter Geoghegan wrote: > I can't imagine that this is much of a problem in practice. Although I will add that not caching highly useful inner pages for the medium term, because that index isn't being used at all for 5 minutes probably is very bad. Using the 4,828

Re: [HACKERS] assertion in 9.4 with wal_level=logical

2014-04-17 Thread Andres Freund
On 2014-04-17 17:40:01 -0300, Alvaro Herrera wrote: > For once, this looks more like a problem in logical decoding, which is > trying to assert about the tuple being updated; the assertion failing is > the one added a week ago about not palloc'ing in a critical section. It's this (older) assertion

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Peter Geoghegan
On Thu, Apr 17, 2014 at 1:39 PM, Andres Freund wrote: > On 2014-04-17 13:33:27 -0700, Peter Geoghegan wrote: >> Just over 99.6% of pages (leaving aside the meta page) in the big 10 >> GB pgbench_accounts_pkey index are leaf pages. > > That's a rather nice number. I knew it was big, but I'd have gu

[HACKERS] DISCARD ALL (Again)

2014-04-17 Thread Joshua D. Drake
Hello, Over at my quaint establishment we have been working on some plython work that makes use of GD. We wrote this code with the assumption (per the docs) that when you issued a DISCARD ALL, the GD would be cleared. Apparently this is not the case. The docs themselves are clearly wrong, an

Re: [HACKERS] assertion in 9.4 with wal_level=logical

2014-04-17 Thread Alvaro Herrera
Steve Singer wrote: > With master/9.4 from today (52e757420fa98a76015c2c88432db94269f3e8f4) > > I am getting an assertion when doing a truncate via SPI when I have > wal_level=logical. > > Stack trace is below. > > I am just replicating a table with normal slony (2.2) I don't need > to establish

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Andres Freund
On 2014-04-17 13:33:27 -0700, Peter Geoghegan wrote: > Just over 99.6% of pages (leaving aside the meta page) in the big 10 > GB pgbench_accounts_pkey index are leaf pages. That's a rather nice number. I knew it was big, but I'd have guessed it'd be a percent lower. Do you happen to have the same

Re: [HACKERS] assertion in 9.4 with wal_level=logical

2014-04-17 Thread Andres Freund
Hi, On 2014-04-17 16:23:54 -0400, Steve Singer wrote: > With master/9.4 from today (52e757420fa98a76015c2c88432db94269f3e8f4) > > I am getting an assertion when doing a truncate via SPI when I have > wal_level=logical. > > Stack trace is below. > > I am just replicating a table with normal slon

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Peter Geoghegan
On Thu, Apr 17, 2014 at 8:10 AM, Greg Stark wrote: > I don't think "common sense" is compelling. I think you need to pin > down exactly what it is about btree intermediate pages that the LRU > isn't capturing and not just argue they're more useful. The LRU is > already capturing which pages are mo

[HACKERS] assertion in 9.4 with wal_level=logical

2014-04-17 Thread Steve Singer
With master/9.4 from today (52e757420fa98a76015c2c88432db94269f3e8f4) I am getting an assertion when doing a truncate via SPI when I have wal_level=logical. Stack trace is below. I am just replicating a table with normal slony (2.2) I don't need to establish any replication slots to get this

Re: [HACKERS] New windows compiler warning from 585bca39

2014-04-17 Thread Tom Lane
David Rowley writes: > I've attached a tiny patch that fixes a new compiler warning on the windows > build... Applied, thanks. > Perhaps the #ifndef could be placed in a nicer spot in the patch, but the > attached should at least describe where the problem lies... Yeah, I thought it better to m

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Stephen Frost
On Thursday, April 17, 2014, Merlin Moncure wrote: > > no -- I got you. My point was, that's a pure guess unless you base it > on evidence recorded on the page itself. Without that evidence, > (which requires writing) the operating is in a a better place to make > that guess so it's probably bett

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Merlin Moncure
On Thu, Apr 17, 2014 at 2:28 PM, Stephen Frost wrote: > > > On Thursday, April 17, 2014, Merlin Moncure wrote: >> >> yeah -- the thing is, we are already too spendy already on >> supplemental write i/o (hint bits, visible bits, freezing, etc) and >> likely not worth it to throw something else on

Re: [HACKERS] Verbose output of pg_dump not show schema name

2014-04-17 Thread Fabrízio de Royes Mello
On Thu, Apr 17, 2014 at 12:46 PM, Bruce Momjian wrote: > > On Thu, Apr 17, 2014 at 11:44:37AM -0400, Tom Lane wrote: > > Bruce Momjian writes: > > > The idea is that we only need quotes when there are odd characters in > > > the identifier. We do that right now in some places, though I can't > >

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Stephen Frost
On Thursday, April 17, 2014, Merlin Moncure wrote: > > yeah -- the thing is, we are already too spendy already on > supplemental write i/o (hint bits, visible bits, freezing, etc) and > likely not worth it to throw something else on the pile unless the > page is already dirty; the medium term tren

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Merlin Moncure
On Thu, Apr 17, 2014 at 2:16 PM, Stephen Frost wrote: > * Merlin Moncure (mmonc...@gmail.com) wrote: >> I don't think this would work unless we would keep some kind of >> tracking information on the page itself which seems not worth a write >> operation to do (maybe if the page is dirtied it coul

Re: [HACKERS] How can we make beta testing better?

2014-04-17 Thread Merlin Moncure
On Tue, Apr 15, 2014 at 4:47 PM, Josh Berkus wrote: > Hackers, > > I think 9.3 has given us evidence that our users aren't giving new > versions of PostgreSQL substantial beta testing, or if they are, they > aren't sharing the results with us. > > How can we make beta testing better and more effec

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > I wonder if it would help to actually tell the OS to read in buffers > > that we're *evicting*... On the general notion that if the OS already > > has them buffered then it's almost a no-op, and if it doesn't and it's > > actually

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Stephen Frost
* Merlin Moncure (mmonc...@gmail.com) wrote: > I don't think this would work unless we would keep some kind of > tracking information on the page itself which seems not worth a write > operation to do (maybe if the page is dirtied it could be snuck in > there though...). IOW, it would only make s

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Tom Lane
Stephen Frost writes: > I wonder if it would help to actually tell the OS to read in buffers > that we're *evicting*... On the general notion that if the OS already > has them buffered then it's almost a no-op, and if it doesn't and it's > actually a 'hot' buffer that we're gonna need again short

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Merlin Moncure
On Thu, Apr 17, 2014 at 2:00 PM, Stephen Frost wrote: > * Merlin Moncure (mmonc...@gmail.com) wrote: >> I doubt that's necessary though -- if the postgres caching algorithm >> improves such that there is a better tendency for hot pages to stay in >> s_b, Eventually the O/S will deschedule the pag

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Stephen Frost
* Merlin Moncure (mmonc...@gmail.com) wrote: > I doubt that's necessary though -- if the postgres caching algorithm > improves such that there is a better tendency for hot pages to stay in > s_b, Eventually the O/S will deschedule the page for something else > that needs it. In other words, othe

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Peter Geoghegan
On Thu, Apr 17, 2014 at 11:53 AM, Merlin Moncure wrote: > No. but if you were very judicious, maybe you could hint the o/s > (posix_fadvise) about pages that are likely to stay hot that you don't > need them. Mitsumasa KONDO wrote a patch like that. I don't think the results were that promising,

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Merlin Moncure
On Thu, Apr 17, 2014 at 1:48 PM, Andres Freund wrote: > On 2014-04-17 21:44:47 +0300, Heikki Linnakangas wrote: >> On 04/17/2014 09:38 PM, Stephen Frost wrote: >> >* Greg Stark (st...@mit.edu) wrote: >> >>On Thu, Apr 17, 2014 at 12:21 PM, Stephen Frost wrote: >> >>>Ehhh. No. If it's a hot page

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Heikki Linnakangas
On 04/17/2014 09:38 PM, Stephen Frost wrote: * Greg Stark (st...@mit.edu) wrote: On Thu, Apr 17, 2014 at 12:21 PM, Stephen Frost wrote: Ehhh. No. If it's a hot page that we've been holding in *our* cache long enough, the kernel will happily evict it as 'cold' from *its* cache, leading to...

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > > Note that if we somehow come up with a page replacement algorithm that tends > > to evict pages that are in the OS cache, we have effectively solved the > > double buffering problem. When a page is cached in both caches, evicting it > > from one o

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Andres Freund
On 2014-04-17 21:44:47 +0300, Heikki Linnakangas wrote: > On 04/17/2014 09:38 PM, Stephen Frost wrote: > >* Greg Stark (st...@mit.edu) wrote: > >>On Thu, Apr 17, 2014 at 12:21 PM, Stephen Frost wrote: > >>>Ehhh. No. If it's a hot page that we've been holding in *our* cache > >>>long enough, the

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-04-17 Thread Peter Geoghegan
On Thu, Apr 17, 2014 at 9:52 AM, Bruce Momjian wrote: > Where are we on this? My hope is that I can get agreement on a way forward during pgCon. Or, at the very least, explain the issues as I see them in a relatively accessible and succinct way to those interested. -- Peter Geoghegan -- Sen

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Stephen Frost
* Greg Stark (st...@mit.edu) wrote: > On Thu, Apr 17, 2014 at 12:21 PM, Stephen Frost wrote: > > Ehhh. No. If it's a hot page that we've been holding in *our* cache > > long enough, the kernel will happily evict it as 'cold' from *its* > > cache, leading to... > > This is a whole nother problem

Re: [HACKERS] How can we make beta testing better?

2014-04-17 Thread Josh Berkus
On 04/17/2014 05:39 AM, Greg Stark wrote: > On Wed, Apr 16, 2014 at 12:53 AM, Rod Taylor wrote: >> 4) Plays queries from the CSV logs starting from $TIME mimicking actual >> timing and transaction boundaries > > This ^^ > > But I recall a number of previous attempts including plugins for > gener

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Peter Geoghegan
On Thu, Apr 17, 2014 at 9:21 AM, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: >> several orders of magnitude more often. That's clearly bad. On >> systems that are not too heavily loaded it doesn't matter too much >> because we just fault the page right back in from the OS

Re: [HACKERS] WAL replay bugs

2014-04-17 Thread Tom Lane
Heikki Linnakangas writes: > Two things that are not bugs, but I'd like to change just to make this > tool easier to maintain, and to generally clean things up: > 1. When creating a sequence, we first use simple_heap_insert() to insert > the sequence tuple, which creates a WAL record. Then we w

Re: [HACKERS] WAL replay bugs

2014-04-17 Thread Heikki Linnakangas
On 04/08/2014 06:41 AM, Michael Paquier wrote: On Tue, Apr 8, 2014 at 3:16 AM, Heikki Linnakangas wrote: I've been playing with a little hack that records a before and after image of every page modification that is WAL-logged, and writes the images to a file along with the LSN of the correspon

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Greg Stark
On Thu, Apr 17, 2014 at 12:21 PM, Stephen Frost wrote: > Ehhh. No. If it's a hot page that we've been holding in *our* cache > long enough, the kernel will happily evict it as 'cold' from *its* > cache, leading to... This is a whole nother problem. It is worrisome that we could be benchmarkin

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-04-17 Thread Bruce Momjian
On Mon, Feb 10, 2014 at 06:40:30PM +, Peter Geoghegan wrote: > On Sun, Jan 19, 2014 at 2:17 AM, Peter Geoghegan wrote: > > I'm just throwing an error when locking the tuple returns > > HeapTupleInvisible, and the xmin of the tuple is our xid. > > I would like some feedback on this point. We n

Re: [HACKERS] Fix memset usage in pgcrypto

2014-04-17 Thread Bruce Momjian
On Mon, Mar 31, 2014 at 09:03:41PM -0400, Bruce Momjian wrote: > On Thu, Dec 26, 2013 at 03:42:12PM +0200, Marko Kreen wrote: > > http://www.viva64.com/en/b/0227/ reported that on-stack memset()s > > might be optimized away by compilers. Fix it. > > > > * Replace memset() with px_memset() > > * A

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > several orders of magnitude more often. That's clearly bad. On > systems that are not too heavily loaded it doesn't matter too much > because we just fault the page right back in from the OS pagecache. Ehhh. No. If it's a hot page that we've been

Re: [HACKERS] shouldn't we log permission errors when accessing the configured trigger file?

2014-04-17 Thread Bruce Momjian
On Wed, Apr 16, 2014 at 06:55:14PM -0400, Bruce Momjian wrote: > > Seems reasonable. It could lead to quite a bit of log spam, I > > suppose, but the way things are now could be pretty mystifying if > > you've located your trigger file somewhere outside $PGDATA, and a > > parent di

Re: [HACKERS] Verbose output of pg_dump not show schema name

2014-04-17 Thread Bruce Momjian
On Thu, Apr 17, 2014 at 11:44:37AM -0400, Tom Lane wrote: > Bruce Momjian writes: > > The idea is that we only need quotes when there are odd characters in > > the identifier. We do that right now in some places, though I can't > > find them in pg_dump. I know psql does that, see quote_ident().

Re: [HACKERS] Verbose output of pg_dump not show schema name

2014-04-17 Thread Tom Lane
Bruce Momjian writes: > The idea is that we only need quotes when there are odd characters in > the identifier. We do that right now in some places, though I can't > find them in pg_dump. I know psql does that, see quote_ident(). I think our general style rule is that identifiers embedded in me

Re: [HACKERS] [GENERAL] pg_upgrade & tablespaces

2014-04-17 Thread Bruce Momjian
On Wed, Apr 16, 2014 at 01:49:20PM -0400, Bruce Momjian wrote: > On Sun, Jan 12, 2014 at 11:04:41PM -0500, Bruce Momjian wrote: > > > In the pgsql_old installation you have symlinks pointing back to the > > > current default location. As well pg_tablespace points back to > > > /usr/local/pgsql/data

Re: [HACKERS] Verbose output of pg_dump not show schema name

2014-04-17 Thread Bruce Momjian
On Thu, Apr 17, 2014 at 12:07:39PM -0300, Fabrízio de Royes Mello wrote: > > Can you get that to _conditionally_ double-quote the strings?  > > Sorry, I didn't understand what you means? Your idea is to check if the > namespace is available and then don't show the double-quote, is that? The idea

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Greg Stark
On Tue, Apr 15, 2014 at 7:30 PM, Peter Geoghegan wrote: > Frankly, there doesn't need to be any research on this, because it's > just common sense that probabilistically, leaf pages are much more > useful than heap pages in servicing index scan queries if we assume a > uniform distribution. If we

Re: [HACKERS] Verbose output of pg_dump not show schema name

2014-04-17 Thread Fabrízio de Royes Mello
On Thu, Apr 17, 2014 at 11:36 AM, Bruce Momjian wrote: > > On Thu, Apr 17, 2014 at 11:29:03AM -0300, Fabrízio de Royes Mello wrote: > > > > This database have a lot of different schemas with same structure and if I > > > > need do view the status of dump I don't know what schema the table are dump

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Greg Stark
On Thu, Apr 17, 2014 at 10:18 AM, Robert Haas wrote: > Because all the usage counts are the same, the eviction at > this point is completely indiscriminate. We're just as likely to kick > out a btree root page or a visibility map page as we are to kick out a > random heap page, even though the fo

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Andres Freund
On 2014-04-17 10:48:15 -0400, Bruce Momjian wrote: > On Thu, Apr 17, 2014 at 10:40:40AM -0400, Robert Haas wrote: > > That can happen, but the real problem I was trying to get at is that > > when all the buffers get up to max usage count, they all appear > > equally important. But in reality they'

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Robert Haas
On Thu, Apr 17, 2014 at 10:48 AM, Bruce Momjian wrote: >> > I understand now. If there is no memory pressure, every buffer gets the >> > max usage count, and when a new buffer comes in, it isn't the max so it >> > is swiftly removed until the clock sweep has time to decrement the old >> > buffers

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Bruce Momjian
On Thu, Apr 17, 2014 at 10:40:40AM -0400, Robert Haas wrote: > On Thu, Apr 17, 2014 at 10:32 AM, Bruce Momjian wrote: > > On Thu, Apr 17, 2014 at 10:18:43AM -0400, Robert Haas wrote: > >> I also believe this to be the case on first principles and my own > >> experiments. Suppose you have a worklo

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Robert Haas
On Thu, Apr 17, 2014 at 10:32 AM, Bruce Momjian wrote: > On Thu, Apr 17, 2014 at 10:18:43AM -0400, Robert Haas wrote: >> I also believe this to be the case on first principles and my own >> experiments. Suppose you have a workload that fits inside >> shared_buffers. All of the usage counts will

Re: [HACKERS] Buildfarm "master-next" branch?

2014-04-17 Thread Tom Lane
Andrew Dunstan writes: > On 04/17/2014 09:17 AM, Robert Haas wrote: >> In terms of improving the buildfarm infrastructure, the thing I would >> most like to have is more frequent runs. IMO the best single thing that could happen for the buildfarm is if we had more critters (at least twice as many

Re: [HACKERS] Verbose output of pg_dump not show schema name

2014-04-17 Thread Bruce Momjian
On Thu, Apr 17, 2014 at 11:29:03AM -0300, Fabrízio de Royes Mello wrote: > > > This database have a lot of different schemas with same structure and if I > > > need do view the status of dump I don't know what schema the table are > > > dump > > > from. > > Yes this may be helpful. The attached qu

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Bruce Momjian
On Thu, Apr 17, 2014 at 10:18:43AM -0400, Robert Haas wrote: > I also believe this to be the case on first principles and my own > experiments. Suppose you have a workload that fits inside > shared_buffers. All of the usage counts will converge to 5. Then, > somebody accesses a table that is not

Re: [HACKERS] Verbose output of pg_dump not show schema name

2014-04-17 Thread Fabrízio de Royes Mello
On Thu, Apr 17, 2014 at 2:14 AM, Michael Paquier wrote: > > On Thu, Apr 17, 2014 at 11:41 AM, Fabrízio de Royes Mello > wrote: > > Hi all, > > > > There are some reason to verbose output of pg_dump don't show schema name? > > > > A output example of using "pg_dump -Fd -j8 -v" > Specifying a targe

Re: [HACKERS] Buildfarm "master-next" branch?

2014-04-17 Thread Andrew Dunstan
On 04/17/2014 09:17 AM, Robert Haas wrote: In terms of improving the buildfarm infrastructure, the thing I would most like to have is more frequent runs. It would be great if pushing a commit to the master repository triggered an immediate build on every buildfarm animal so that you could see

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Robert Haas
On Thu, Apr 17, 2014 at 9:40 AM, Greg Stark wrote: > On Wed, Apr 16, 2014 at 12:44 AM, Robert Haas wrote: >> This isn't a fundamental property of the usage-count idea; it's an >> artifact of the fact that usage count decreases are tied to eviction >> pressure rather than access pressure. For exa

Re: [HACKERS] assertion failure 9.3.4

2014-04-17 Thread Andrew Dunstan
On 04/16/2014 10:28 PM, Tom Lane wrote: Andrew Dunstan writes: On 04/16/2014 07:19 PM, Tom Lane wrote: Yeah, it would be real nice to see a self-contained test case for this. Well, that might be hard to put together, but I did try running without pg_stat_statements and auto_explain loaded an

Re: [HACKERS] BGWorkers, shared memory pointers, and postmaster restart

2014-04-17 Thread Robert Haas
On Wed, Apr 16, 2014 at 8:46 PM, Craig Ringer wrote: > On 04/17/2014 12:16 AM, Robert Haas wrote: >> On Wed, Apr 16, 2014 at 7:11 AM, Craig Ringer wrote: > >>> - A flag like BGW_UNREGISTER_ON_RESTART; >> >> I would be OK with this, maybe modulo the name. >> >>> - To always unregister dynamic bgws

Re: [HACKERS] four minor proposals for 9.5

2014-04-17 Thread Tom Lane
Pavel Stehule writes: > We can introduce new feature without hard dependency on CSV format Look, the long and the short of it is that there is not consensus that this measurement is worth creating a new CSV log column for. And from that, there is also not consensus that it's worth putting into lo

Re: [HACKERS] four minor proposals for 9.5

2014-04-17 Thread Pavel Stehule
2014-04-17 7:12 GMT+02:00 Amit Kapila : > On Mon, Apr 14, 2014 at 6:27 PM, Robert Haas > wrote: > > I agree. I don't think the idea of pushing this into the > > log_line_prefix stuff as a one-off is a very good one. Sure, we could > > wedge it in there, but we've got an existing precedent that

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-17 Thread Greg Stark
On Wed, Apr 16, 2014 at 12:44 AM, Robert Haas wrote: > This isn't a fundamental property of the usage-count idea; it's an > artifact of the fact that usage count decreases are tied to eviction > pressure rather than access pressure. For example, suppose we made a > rule that if the total usage co

Re: [HACKERS] Minor performance improvement in transition to external sort

2014-04-17 Thread Robert Haas
On Wed, Apr 16, 2014 at 7:38 PM, Bruce Momjian wrote: > On Thu, Apr 10, 2014 at 06:03:15PM +0100, Simon Riggs wrote: >> On 6 February 2014 18:21, Jeff Janes wrote: >> > On Tue, Feb 4, 2014 at 2:22 PM, Jeremy Harris wrote: >> >> >> >> The attached patch replaces the existing siftup method for hea

Re: [HACKERS] Need Multixact Freezing Docs

2014-04-17 Thread Robert Haas
On Wed, Apr 16, 2014 at 4:39 PM, Josh Berkus wrote: >> Hmm, are you sure it's INT_MAX and not 4244967297? Heikki reported >> that: http://www.postgresql.org/message-id/52401aea.9000...@vmware.com >> The absolute value is not important; I think that's mostly harmless. I >> don't think applying ag

Re: [HACKERS] Buildfarm "master-next" branch? (was: Dynamic Shared Memory stuff)

2014-04-17 Thread Robert Haas
On Wed, Apr 16, 2014 at 8:24 PM, Craig Ringer wrote: > On 04/17/2014 12:08 AM, Robert Haas wrote: >> On Tue, Apr 15, 2014 at 10:46 PM, Amit Kapila >> wrote: >>> On Wed, Apr 16, 2014 at 3:01 AM, Robert Haas wrote: On Tue, Apr 15, 2014 at 12:33 AM, Amit Kapila wrote: > On Mon, Apr

Re: [HACKERS] How can we make beta testing better?

2014-04-17 Thread Greg Stark
On Wed, Apr 16, 2014 at 12:53 AM, Rod Taylor wrote: > 4) Plays queries from the CSV logs starting from $TIME mimicking actual > timing and transaction boundaries This ^^ But I recall a number of previous attempts including plugins for general load testing systems, what happened to them? Honestl

Re: [HACKERS] Patch: iff -> if

2014-04-17 Thread Nicolas Barbier
2014-04-17 Michael Paquier : > Is there no equivalent in German? For example in French there is "ssi". gdw (genau dann, wenn) Nicolas -- A. Because it breaks the logical sequence of discussion. Q. Why is top posting bad? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] [COMMITTERS] pgsql: Include planning time in EXPLAIN ANALYZE output.

2014-04-17 Thread Andreas Karlsson
On 04/17/2014 01:35 AM, Tom Lane wrote: I'll go change it. Thanks for fixing this. The new name "Execution time" is much clearer. -- Andreas Karlsson -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

Re: [HACKERS] Misaligned BufferDescriptors causing major performance problems on AMD

2014-04-17 Thread Andres Freund
On 2014-04-16 19:18:02 -0400, Bruce Momjian wrote: > On Thu, Feb 6, 2014 at 09:40:32AM +0100, Andres Freund wrote: > > On 2014-02-05 12:36:42 -0500, Robert Haas wrote: > > > >> It may well be that your proposal is spot on. But I'd like to see some > > > >> data-structure-by-data-structure measure

Re: [HACKERS] slow startup due to LWLockAssign() spinlock

2014-04-17 Thread Andres Freund
On 2014-04-16 19:33:52 -0400, Bruce Momjian wrote: > On Tue, Feb 4, 2014 at 12:58:49AM +0100, Andres Freund wrote: > > On 2014-02-03 11:22:45 -0500, Tom Lane wrote: > > > Andres Freund writes: > > > > On larger, multi-socket, machines, startup takes a fair bit of time. As > > > > I was profiling

[HACKERS] New windows compiler warning from 585bca39

2014-04-17 Thread David Rowley
I've attached a tiny patch that fixes a new compiler warning on the windows build... Perhaps the #ifndef could be placed in a nicer spot in the patch, but the attached should at least describe where the problem lies... (ClCompile target) -> src\interfaces\libpq\fe-connect.c(3365): warning C4101

Re: [HACKERS] Question about optimising (Postgres_)FDW

2014-04-17 Thread Etsuro Fujita
(2014/04/16 22:16), Hannu Krosing wrote: On 04/16/2014 01:35 PM, Etsuro Fujita wrote: Maybe I'm missing something, but I think that you can do what I think you'd like to do by the following procedure: No, what I'd like PostgreSQL to do is to 1. select the id+set from local table 2. select th

Re: [HACKERS] [COMMITTERS] pgsql: Include planning time in EXPLAIN ANALYZE output.

2014-04-17 Thread Oleg Bartunov
I found a bit confusing, when planning time is greater total time, so +1 for execution time. On Thu, Apr 17, 2014 at 3:35 AM, Tom Lane wrote: > Bruce Momjian writes: >> Where are we on this? I still see: > >> test=> EXPLAIN ANALYZE SELECT 1; >>Q