Re: [HACKERS] Odd out of memory problem.

2012-03-31 Thread Hitoshi Harada
On Thu, Mar 29, 2012 at 7:38 PM, Peter Eisentraut pete...@gmx.net wrote: On tis, 2012-03-27 at 00:53 +0100, Greg Stark wrote: Hm. So my original plan was dependent on adding the state-merge function we've talked about in the past. Not all aggregate functions necessarily can support such a

[HACKERS] Tab completion of double quoted identifiers broken

2012-03-31 Thread Dean Rasheed
Hi, I just spotted that tab completion of double quoted identifiers seems to be broken in 9.2devel. For example things like this which worked in 9.1 no longer work: UPDATE foo bar tab It looks like the problem is in get_previous_words() here: if (buf[start] == '')

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Simon Riggs
On Sat, Mar 31, 2012 at 4:41 AM, Robert Haas robertmh...@gmail.com wrote: which means, if I'm not confused here, that every single lwlock-related stall 1s happened while waiting for a buffer content lock.  Moreover, each event affected a different buffer.  I find this result so surprising

[HACKERS] Http Frontend implemented using pgsql?

2012-03-31 Thread Dobes Vandermeer
I had a thought that it might be interesting to have a simple C fronted that converts HTTP to and from some pgsql friendly structure and delegates all the core logic to a stored procedure in the database. This might make it easier to hack on the API without worrying about memory management and

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Robert Haas
On Sat, Mar 31, 2012 at 4:53 AM, Simon Riggs si...@2ndquadrant.com wrote: The next logical step in measuring lock waits is to track the reason for the lock wait, not just the lock wait itself. I had the same thought. I'm not immediately sure what the best way to do that is, but I'll see if I

Re: [HACKERS] Odd out of memory problem.

2012-03-31 Thread Peter Eisentraut
On fre, 2012-03-30 at 22:59 -0700, Hitoshi Harada wrote: I don't know exactly how PL/Proxy or pgpool accomplish the multi-phase aggregate, They don't. but in theory the proposal above is state-merge function, so it doesn't apply to general aggregate results that passed through the final

Re: [HACKERS] Patch pg_is_in_backup()

2012-03-31 Thread Gabriele Bartolini
Hi Gilles, first and foremost, sorry for jumping in this thread so late. I read all previous discussions and I'd be happy to help you with this patch. Agreed and sorry for the response delay. I've attached 2 patches here, the first one is the same as before with just the renaming of the

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Greg Stark
On Sat, Mar 31, 2012 at 4:41 AM, Robert Haas robertmh...@gmail.com wrote: But I didn't think we were ever supposed to hold content locks for that long. Isn't that lock held while doing visibility checks? What about I/O waiting for a clog page to be read? -- greg -- Sent via pgsql-hackers

Re: [HACKERS] Http Frontend implemented using pgsql?

2012-03-31 Thread Aidan Van Dyk
On Sat, Mar 31, 2012 at 6:27 AM, Dobes Vandermeer dob...@gmail.com wrote: I had a thought that it might be interesting to have a simple C fronted that converts HTTP to and from some pgsql friendly structure and delegates all the core logic to a stored procedure in the database. This might

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-31 Thread Dobes Vandermeer
On Sat, Mar 31, 2012 at 1:44 AM, Daniel Farina dan...@heroku.com wrote: On Fri, Mar 30, 2012 at 10:21 AM, Daniel Farina dan...@heroku.com wrote: Any enhancement here that can't be used with libpq via, say, drop-in .so seems unworkable to me, and that's why any solution that is basically

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Simon Riggs
On Sat, Mar 31, 2012 at 1:58 PM, Greg Stark st...@mit.edu wrote: On Sat, Mar 31, 2012 at 4:41 AM, Robert Haas robertmh...@gmail.com wrote: But I didn't think we were ever supposed to hold content locks for that long. Isn't that lock held while doing visibility checks? What about I/O waiting

[HACKERS] narwhal versus gnu_printf

2012-03-31 Thread Tom Lane
I noticed that the build logs for buildfarm member narwhal (a mingw critter) contain an awful lot of occurrences of ../../src/include/utils/elog.h:159: warning: `gnu_printf' is an unrecognized format function type Evidently, the following hunk in pg_config_manual.h failed to consider mingw. Is

Re: [HACKERS] Tab completion of double quoted identifiers broken

2012-03-31 Thread Tom Lane
Dean Rasheed dean.a.rash...@gmail.com writes: I just spotted that tab completion of double quoted identifiers seems to be broken in 9.2devel. Yeah, looks like I broke it :-( --- I think I missed the fact that the last check with WORD_BREAKS was looking at the previous character not the current

Re: [HACKERS] narwhal versus gnu_printf

2012-03-31 Thread Andrew Dunstan
On 03/31/2012 11:01 AM, Tom Lane wrote: I noticed that the build logs for buildfarm member narwhal (a mingw critter) contain an awful lot of occurrences of ../../src/include/utils/elog.h:159: warning: `gnu_printf' is an unrecognized format function type Evidently, the following hunk in

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-31 Thread Tom Lane
Marko Kreen mark...@gmail.com writes: On Thu, Mar 29, 2012 at 06:56:30PM -0400, Tom Lane wrote: Yeah. Perhaps we should tweak the row-processor callback API so that it gets an explicit notification that this is a new resultset. Duplicating PQexec's behavior would then involve having the

Re: [HACKERS] [COMMITTERS] pgsql: Add PGDLLIMPORT to ScanKeywords and NumScanKeywords.

2012-03-31 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: On 03/31/2012 10:56 AM, Tom Lane wrote: Add PGDLLIMPORT to ScanKeywords and NumScanKeywords. Per buildfarm, this is now needed by contrib/pg_stat_statements. It seems to have broken mingw earlier now :-( Ugh. It looks like ecpg (and also pg_dump)

Re: [HACKERS] [GENERAL] pg_dump incredibly slow dumping a single schema from a large db

2012-03-31 Thread Tom Lane
Mike Roest mike.ro...@replicon.com writes: The file is 6 megs so I've dropped it here. That was doing perf for the length of the pg_dump command and then a perf report -n http://dl.dropbox.com/u/13153/output.txt Hmm ... that's a remarkably verbose output format, but the useful part of this

Re: [HACKERS] [COMMITTERS] pgsql: Add PGDLLIMPORT to ScanKeywords and NumScanKeywords.

2012-03-31 Thread Andrew Dunstan
On 03/31/2012 11:59 AM, Tom Lane wrote: Andrew Dunstanand...@dunslane.net writes: On 03/31/2012 10:56 AM, Tom Lane wrote: Add PGDLLIMPORT to ScanKeywords and NumScanKeywords. Per buildfarm, this is now needed by contrib/pg_stat_statements. It seems to have broken mingw earlier now :-(

Re: [HACKERS] [COMMITTERS] pgsql: Add PGDLLIMPORT to ScanKeywords and NumScanKeywords.

2012-03-31 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: On 03/31/2012 11:59 AM, Tom Lane wrote: The only simple fix I can see is to rename the symbols in ecpg and pg_dump to something else. This is probably a good thing anyway to reduce confusion. Anybody have another idea? Seems the sane thing to do.

Re: [HACKERS] [GENERAL] pg_dump incredibly slow dumping a single schema from a large db

2012-03-31 Thread Tom Lane
I wrote: So this is dumb; we should manage the is the object already processed component of that with an O(1) check, like a bool array or some such, rather than an O(N) search loop. As for the getTables slowdown, the only part of that I can see that looks to be both significant and entirely

[HACKERS] Re: [GENERAL] pg_dump incredibly slow dumping a single schema from a large db

2012-03-31 Thread Mike Roest
I've committed fixes for both these issues. If you are in a position to test with 9.1 branch tip from git, it'd be nice to have confirmation that these patches actually cure your problem. For both of them, the issue seems to only show up in a subset of cases, which may explain why we'd not

[HACKERS] Re: [GENERAL] pg_dump incredibly slow dumping a single schema from a large db

2012-03-31 Thread Mike Roest
I'm just pulling another backup using the stock 9.1.1 pg_dump to ensure the backups are equivalent. Schema data are identical between the 2 backups. the new backup passes all our tests for validating a tenant. Thank you again for the quick response! --Mike

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Robert Haas
On Sat, Mar 31, 2012 at 8:58 AM, Greg Stark st...@mit.edu wrote: On Sat, Mar 31, 2012 at 4:41 AM, Robert Haas robertmh...@gmail.com wrote: But I didn't think we were ever supposed to hold content locks for that long. Isn't that lock held while doing visibility checks? Nope. heap_update()

Re: [HACKERS] Publish checkpoint timing and sync files summary data to pg_stat_bgwriter

2012-03-31 Thread Peter Geoghegan
On 28 March 2012 15:23, Robert Haas robertmh...@gmail.com wrote:  At any rate, I strongly agree that counting the number of strategy allocations is not really a viable proxy for counting the number of backend writes.  You can't know how many of those actually got dirtied. Sure. Since any

Re: [HACKERS] Re: [GENERAL] pg_dump incredibly slow dumping a single schema from a large db

2012-03-31 Thread Tom Lane
Mike Roest mike.ro...@replicon.com writes: Any idea when 9.1.4 with this change will be out so we can pull the cluster up. Well, we just did some releases last month, so unless somebody finds a really nasty security or data-loss issue, I'd think it will be a couple of months before the next

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Greg Stark
On Sat, Mar 31, 2012 at 10:14 PM, Robert Haas robertmh...@gmail.com wrote: Isn't that lock held while doing visibility checks? Nope.  heap_update() and friends do a very complicated little dance to avoid that. ... What about I/O waiting for a clog page to be read? I'm pretty sure that can

[HACKERS] new group commit behavior not helping?

2012-03-31 Thread Robert Haas
Hoping to demonstrate the wonders of our new group commit code, I ran some benchmarks on the IBM POWER7 machine with synchronous_commit = on. But, it didn't come out much better than 9.1. pgbench, scale factor 300, median of 3 30-minute test runs, # clients = #threads, shared_buffers = 8GB,

Re: [HACKERS] new group commit behavior not helping?

2012-03-31 Thread Peter Geoghegan
On 1 April 2012 01:10, Robert Haas robertmh...@gmail.com wrote: Hoping to demonstrate the wonders of our new group commit code, I ran some benchmarks on the IBM POWER7 machine with synchronous_commit = on.  But, it didn't come out much better than 9.1.  pgbench, scale factor 300, median of 3

Re: [HACKERS] new group commit behavior not helping?

2012-03-31 Thread Robert Haas
On Sat, Mar 31, 2012 at 8:31 PM, Peter Geoghegan pe...@2ndquadrant.com wrote: Why the low value for wal_writer_delay? A while back I was getting a benefit from cranking that down. I could try leaving it out and see if it matters. master: 01 tps = 118.968446 (including connections

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Robert Haas
On Sat, Mar 31, 2012 at 6:01 PM, Greg Stark st...@mit.edu wrote: On Sat, Mar 31, 2012 at 10:14 PM, Robert Haas robertmh...@gmail.com wrote: Isn't that lock held while doing visibility checks? Nope.  heap_update() and friends do a very complicated little dance to avoid that. ... What about

Re: [HACKERS] Command Triggers patch v18

2012-03-31 Thread Robert Haas
On Fri, Mar 30, 2012 at 11:19 AM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Yeah context is not explicit, we could call that toplevel: the command tag of the command that the user typed. When toplevel is null, the event trigger is fired on a command the user sent, when it's not null, the

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Robert Haas
On Sat, Mar 31, 2012 at 9:29 PM, Robert Haas robertmh...@gmail.com wrote: I've discovered a bug in my code that was causing it to print at most 2 histogram buckets per lwlock, which obviously means that my previous results were totally inaccurate.  Ah, the joys of benchmarking.  I found the

[HACKERS] new group commit behavior not helping?

2012-03-31 Thread Jeff Janes
On Saturday, March 31, 2012, Robert Haas robertmh...@gmail.com wrote: Hoping to demonstrate the wonders of our new group commit code, I ran some benchmarks on the IBM POWER7 machine with synchronous_commit = on. But, it didn't come out much better than 9.1. Where I would expect (and have

Re: [HACKERS] new group commit behavior not helping?

2012-03-31 Thread Robert Haas
On Sat, Mar 31, 2012 at 8:31 PM, Peter Geoghegan pe...@2ndquadrant.com wrote: The exact benchmark that I ran was the update.sql pgbench-tools benchmark, on my laptop. The idea was to produce a sympathetic benchmark with a workload that was maximally commit-bound. Heikki reproduced similar

Re: [HACKERS] new group commit behavior not helping?

2012-03-31 Thread Robert Haas
On Sun, Apr 1, 2012 at 1:40 AM, Jeff Janes jeff.ja...@gmail.com wrote: On Saturday, March 31, 2012, Robert Haas robertmh...@gmail.com wrote: Hoping to demonstrate the wonders of our new group commit code, I ran some benchmarks on the IBM POWER7 machine with synchronous_commit = on.  But, it