Re: [HACKERS] Initial 9.2 pgbench write results

2012-02-19 Thread Simon Riggs
On Sun, Feb 19, 2012 at 4:17 AM, Robert Haas robertmh...@gmail.com wrote: Here's what's bugging me.  Greg seemed to be assuming that the business of the background writer might be the cause of the performance drop-off he measured on certain test cases.  But you and I both seem to feel that

Re: [HACKERS] Initial 9.2 pgbench write results

2012-02-19 Thread Simon Riggs
On Tue, Feb 14, 2012 at 6:45 PM, Greg Smith g...@2ndquadrant.com wrote: Minimal changes were made to the postgresql.conf.  shared_buffers=2GB, checkpoint_segments=64, and I left wal_buffers at its default so that 9.1 got credit for that going up.  See

[HACKERS] pg_upgrade --logfile option documentation

2012-02-19 Thread Peter Eisentraut
The documentation of the pg_upgrade -l/--logfile option never made much sense to me: -l, --logfile=FILENAMElog session activity to file I don't know what session means for pg_upgrade, so I never used it. What it actually does is log the output of all the programs that pg_upgrade calls

Re: [HACKERS] pg_restore ignores PGDATABASE

2012-02-19 Thread Robert Haas
On Sun, Feb 19, 2012 at 1:18 AM, Erik Rijkers e...@xs4all.nl wrote: On Sun, February 19, 2012 06:27, Robert Haas wrote: On Sat, Feb 18, 2012 at 11:58 AM, Erik Rijkers e...@xs4all.nl wrote: pg_restore ignores environment variable PGDATABASE. What exactly do you mean by ignores?  pg_restore

Re: [HACKERS] wal_buffers

2012-02-19 Thread Euler Taveira de Oliveira
On 19-02-2012 02:24, Robert Haas wrote: I have attached tps scatterplots. The obvious conclusion appears to be that, with only 16MB of wal_buffers, the buffer wraps around with some regularity Isn't it useful to print some messages on the log when we have wrap around? In this case, we have an

Re: [HACKERS] pg_restore ignores PGDATABASE

2012-02-19 Thread Andrew Dunstan
On 02/19/2012 08:02 AM, Robert Haas wrote: On Sun, Feb 19, 2012 at 1:18 AM, Erik Rijkerse...@xs4all.nl wrote: On Sun, February 19, 2012 06:27, Robert Haas wrote: On Sat, Feb 18, 2012 at 11:58 AM, Erik Rijkerse...@xs4all.nl wrote: pg_restore ignores environment variable PGDATABASE. What

Re: [HACKERS] 16-bit page checksums for 9.2

2012-02-19 Thread Simon Riggs
On Thu, Feb 16, 2012 at 1:53 PM, Robert Haas robertmh...@gmail.com wrote: +                       /* +                        * If we're in recovery we cannot dirty a page because of a hint. +                        * We can set the hint, just not dirty the page as a result so +          

Re: [HACKERS] wal_buffers

2012-02-19 Thread Robert Haas
On Sun, Feb 19, 2012 at 9:46 AM, Euler Taveira de Oliveira eu...@timbira.com wrote: On 19-02-2012 02:24, Robert Haas wrote: I have attached tps scatterplots.  The obvious conclusion appears to be that, with only 16MB of wal_buffers, the buffer wraps around with some regularity Isn't it

Re: [HACKERS] pg_upgrade --logfile option documentation

2012-02-19 Thread Robert Haas
On Sun, Feb 19, 2012 at 6:13 AM, Peter Eisentraut pete...@gmx.net wrote: The documentation of the pg_upgrade -l/--logfile option never made much sense to me:  -l, --logfile=FILENAME        log session activity to file I don't know what session means for pg_upgrade, so I never used it. What

Re: [HACKERS] wal_buffers

2012-02-19 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sun, Feb 19, 2012 at 9:46 AM, Euler Taveira de Oliveira eu...@timbira.com wrote: Isn't it useful to print some messages on the log when we have wrap around? In this case, we have an idea that wal_buffers needs to be increased. I was thinking about

Re: [HACKERS] wal_buffers

2012-02-19 Thread Simon Riggs
On Sun, Feb 19, 2012 at 6:33 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Sun, Feb 19, 2012 at 9:46 AM, Euler Taveira de Oliveira eu...@timbira.com wrote: Isn't it useful to print some messages on the log when we have wrap around? In this case, we have

[HACKERS] Reducing bgwriter wakeups

2012-02-19 Thread Simon Riggs
Recent changes for power reduction mean that we now issue a wakeup call to the bgwriter every time we set a hint bit. However cheap that is, its still overkill. My proposal is that we wakeup the bgwriter whenever a backend is forced to write a dirty buffer, a job the bgwriter should have been

[HACKERS] patch: autocomplete for functions

2012-02-19 Thread Pavel Stehule
Hello I found so this extremely simple patch should be useful. It helps for pattern SELECT fx(); There was thread about it. Regards Pavel *** ./src/bin/psql/tab-complete.c.orig 2012-02-05 11:28:48.0 +0100 --- ./src/bin/psql/tab-complete.c 2012-02-19 20:05:05.241626625 +0100

Re: [HACKERS] 16-bit page checksums for 9.2

2012-02-19 Thread Simon Riggs
On Sun, Feb 19, 2012 at 4:35 PM, Simon Riggs si...@2ndquadrant.com wrote: We don't need to wait until nobody has it set, we just need to wait for the people that had it set when we first checked to be out of that state momentarily. I've just finished doing some performance analysis on various

[HACKERS] patch: CREATE OR REPLACE FUNCTION autocomplete

2012-02-19 Thread Pavel Stehule
Hello other very simple patch - enhance autocomplete to support CREATE OR REPLACE FUNCTION statement Regards Pavel Stehule *** ./src/bin/psql/tab-complete.c.orig 2012-02-19 20:05:05.0 +0100 --- ./src/bin/psql/tab-complete.c 2012-02-19 20:20:43.817202512 +0100 *** *** 644,649

Re: [HACKERS] Reducing bgwriter wakeups

2012-02-19 Thread Robert Haas
On Sun, Feb 19, 2012 at 1:53 PM, Simon Riggs si...@2ndquadrant.com wrote: Recent changes for power reduction mean that we now issue a wakeup call to the bgwriter every time we set a hint bit. However cheap that is, its still overkill. My proposal is that we wakeup the bgwriter whenever a

Re: [HACKERS] Reducing bgwriter wakeups

2012-02-19 Thread Simon Riggs
On Sun, Feb 19, 2012 at 8:15 PM, Robert Haas robertmh...@gmail.com wrote: On Sun, Feb 19, 2012 at 1:53 PM, Simon Riggs si...@2ndquadrant.com wrote: Recent changes for power reduction mean that we now issue a wakeup call to the bgwriter every time we set a hint bit. However cheap that is, its

Re: [HACKERS] [COMMITTERS] pgsql: Improve pretty printing of viewdefs.

2012-02-19 Thread Andrew Dunstan
On 02/19/2012 04:18 PM, Andrew Dunstan wrote: [redirecting to -hackers] Arghh, this time redirecting ... On 02/19/2012 12:04 PM, Pavel Stehule wrote: Hello nice should be this functionality used for query too? some like pg_pretty_query('SELECT ... ', 80) when we have this

Re: [HACKERS] 16-bit page checksums for 9.2

2012-02-19 Thread Robert Haas
On Sun, Feb 19, 2012 at 11:35 AM, Simon Riggs si...@2ndquadrant.com wrote: Doesn't this seem awfully bad for performance on Hot Standby servers? I agree that it fixes the problem with un-WAL-logged pages there, but I seem to recall some recent complaining about performance features that work

Re: [HACKERS] Reducing bgwriter wakeups

2012-02-19 Thread Robert Haas
On Sun, Feb 19, 2012 at 4:11 PM, Simon Riggs si...@2ndquadrant.com wrote: On Sun, Feb 19, 2012 at 8:15 PM, Robert Haas robertmh...@gmail.com wrote: On Sun, Feb 19, 2012 at 1:53 PM, Simon Riggs si...@2ndquadrant.com wrote: Recent changes for power reduction mean that we now issue a wakeup call

[HACKERS] leakproof

2012-02-19 Thread Andrew Dunstan
I missed all the fun while the leakproof addition to function attributes was being decided, so I know I'm late to the party. Today I had to go and look up what it actually meant. I have to say that I was a bit surprised. I expected it to refer to memory management in some way. I don't honestly

Re: [HACKERS] Future of our regular expression code

2012-02-19 Thread Brendan Jurd
On 19 February 2012 15:49, Tom Lane t...@sss.pgh.pa.us wrote: That sounds great. BTW, if you don't have it already, I'd highly recommend getting a copy of Friedl's Mastering Regular Expressions.  It's aimed at users not implementers, but there is a wealth of valuable context information in

Re: [HACKERS] Reducing bgwriter wakeups

2012-02-19 Thread Jeff Janes
On Sun, Feb 19, 2012 at 2:18 PM, Robert Haas robertmh...@gmail.com wrote: Also, in general, I think that it's not a good idea to let dirty data sit in shared_buffers forever.  I'm unhappy about the change this release cycle to skip checkpoints if we've written less than a full WAL segment,

Re: [HACKERS] 16-bit page checksums for 9.2

2012-02-19 Thread Simon Riggs
On Sun, Feb 19, 2012 at 10:04 PM, Robert Haas robertmh...@gmail.com wrote: To me, it seems that you are applying a double standard.  You have twice attempted to insist that I do extra work to make major features that I worked on - unlogged tables and index-only scans - work in Hot Standby

Re: [HACKERS] Future of our regular expression code

2012-02-19 Thread Tom Lane
Brendan Jurd dire...@gmail.com writes: Are you far enough into the backrefs bug that you'd prefer to see it through, or would you like me to pick it up? Actually, what I've been doing today is a brain dump. This code is never going to be maintainable by anybody except its original author

Re: [HACKERS] 16-bit page checksums for 9.2

2012-02-19 Thread Simon Riggs
On Sun, Feb 19, 2012 at 10:04 PM, Robert Haas robertmh...@gmail.com wrote: As explained in detailed comments, the purpose of this is to implement Heikki's suggestion that we have a bit set to zero so we can detect failures that cause a run of 1s. I think it's nonsensical to pretend that

Re: [HACKERS] Potential reference miscounts and segfaults in plpython.c

2012-02-19 Thread Jan Urbański
On 18/02/12 21:18, Jan Urbański wrote: On 18/02/12 21:17, Tom Lane wrote: =?UTF-8?B?SmFuIFVyYmHFhHNraQ==?= wulc...@wulczer.org writes: On 18/02/12 20:30, Tom Lane wrote: Dave Malcolm at Red Hat has been working on a static code analysis tool for Python-related C code. He reports here on some

Re: [HACKERS] pl/python long-lived allocations in datum-dict transformation

2012-02-19 Thread Jan Urbański
On 14/02/12 01:35, Tom Lane wrote: =?UTF-8?B?SmFuIFVyYmHFhHNraQ==?= wulc...@wulczer.org writes: It's not very comfortable, but I think PLyDict_FromTuple can be allowed to be non-reentrant. I think that's pretty short-sighted. Even if it's safe today (which I am not 100% convinced of),

Re: [HACKERS] leakproof

2012-02-19 Thread Robert Haas
On Sun, Feb 19, 2012 at 5:29 PM, Andrew Dunstan and...@dunslane.net wrote: I missed all the fun while the leakproof addition to function attributes was being decided, so I know I'm late to the party. Today I had to go and look up what it actually meant. I have to say that I was a bit surprised.

Re: [HACKERS] Reducing bgwriter wakeups

2012-02-19 Thread Robert Haas
On Sun, Feb 19, 2012 at 5:56 PM, Jeff Janes jeff.ja...@gmail.com wrote: Would the log really have been archived in 9.1?  I don't think checkpoint_timeout caused a log switch, just a checkpoint which could happily be in the same file as the previous checkpoint. The log segment doesn't need to

Re: [HACKERS] 16-bit page checksums for 9.2

2012-02-19 Thread Robert Haas
On Sun, Feb 19, 2012 at 6:33 PM, Simon Riggs si...@2ndquadrant.com wrote: On Sun, Feb 19, 2012 at 10:04 PM, Robert Haas robertmh...@gmail.com wrote: To me, it seems that you are applying a double standard.  You have twice attempted to insist that I do extra work to make major features that I

Re: [HACKERS] Future of our regular expression code

2012-02-19 Thread Brendan Jurd
On 20 February 2012 10:42, Tom Lane t...@sss.pgh.pa.us wrote: I have also got a bunch of text about the colormap management code, which I think is interesting right now because that is what we are going to have to fix if we want decent performance for Unicode \w and related classes (cf the

Re: [HACKERS] Future of our regular expression code

2012-02-19 Thread Billy Earney
Tom, I did a google search, and found the following: http://www.arglist.com/regex/ Which states that Tcl uses the same library from Henry. Maybe someone involved with that project would help explain the library? Also I noticed at the url above is a few ports people did from Henry's code. I

Re: [HACKERS] 16-bit page checksums for 9.2

2012-02-19 Thread Robert Haas
On Sun, Feb 19, 2012 at 6:57 PM, Simon Riggs si...@2ndquadrant.com wrote: I thought it was a reasonable and practical idea from Heikki. The bit is not selected arbitrarily, it is by design adjacent to one of the other bits. So overall, 3 bits need to be set to a precise value and a run of 1s

Re: [HACKERS] Future of our regular expression code

2012-02-19 Thread Tom Lane
Billy Earney billy.ear...@gmail.com writes: I did a google search, and found the following: http://www.arglist.com/regex/ Hmm ... might be worth looking at those two pre-existing attempts at making a standalone library from Henry's code, just to see what choices they made. Which states that

Re: [HACKERS] leakproof

2012-02-19 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sun, Feb 19, 2012 at 5:29 PM, Andrew Dunstan and...@dunslane.net wrote: Can we come up with a more descriptive term? We bikeshed on that topic a while back and nobody suggested anything that got more than 1 or 2 votes. But I'm still happy to

Re: [HACKERS] Future of our regular expression code

2012-02-19 Thread Billy Earney
Thanks Tom. I looked at the code in the libraries I referred to earlier, and it looks like the code in the regex directory is exactly the same as Walter Waldo's version, which has at least one comment from the middle of last decade (~ 2003). Has people thought about migrating to the pcre

Re: [HACKERS] Future of our regular expression code

2012-02-19 Thread Stephen Frost
Billy, * Billy Earney (billy.ear...@gmail.com) wrote: Thanks Tom. I looked at the code in the libraries I referred to earlier, and it looks like the code in the regex directory is exactly the same as Walter Waldo's version, which has at least one comment from the middle of last decade (~

Re: [HACKERS] leakproof

2012-02-19 Thread Don Baccus
On Feb 19, 2012, at 5:42 PM, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: Having now spent far too much time in bed with that patch, I'm feeling like the concept that we are really looking for there is what some languages call pure - that is, there must be no side effects,

Re: [HACKERS] leakproof

2012-02-19 Thread Greg Stark
I suspect this is wrong for similar reasons as pure but I'll throw it out there: hermetic -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] leakproof

2012-02-19 Thread Tom Lane
Don Baccus dhog...@pacifier.com writes: On Feb 19, 2012, at 5:42 PM, Tom Lane wrote: Hmm, pure doesn't sound bad to me. Nice and short. Technically, pure is stronger than has no side effects: http://en.wikipedia.org/wiki/Pure_function Result can't depend on state (for instance, database

Re: [HACKERS] Future of our regular expression code

2012-02-19 Thread Greg Stark
On Sat, Feb 18, 2012 at 6:15 PM, Tom Lane t...@sss.pgh.pa.us wrote:  A larger point is that it'd be a real shame for the Spencer regex engine to die off, because it is in fact one of the best pieces of regex technology on the planet. ... Another possible long-term answer is to finish the work

Re: [HACKERS] Potential reference miscounts and segfaults in plpython.c

2012-02-19 Thread Tom Lane
=?UTF-8?B?SmFuIFVyYmHFhHNraQ==?= wulc...@wulczer.org writes: On 18/02/12 21:17, Tom Lane wrote: Dave Malcolm at Red Hat has been working on a static code analysis tool for Python-related C code. He reports here on some preliminary results for plpython.c:

Re: [HACKERS] Future of our regular expression code

2012-02-19 Thread Stephen Frost
Greg, * Greg Stark (st...@mit.edu) wrote: I can't see how your first claim that the Spencer code is worth keeping around because it's just a superior regex implementation has much force unless we can accomplish the latter. If the library can be split off into a standalone library then it

Re: [HACKERS] Future of our regular expression code

2012-02-19 Thread Tom Lane
Greg Stark st...@mit.edu writes: ... We need a library that can be used to defend against malicious regexes and i suspect neither Perl's nor Python's library will suffice for this. Yeah. Did you read the Russ Cox papers referenced upthread? One of the things Google wanted was provably

Re: [HACKERS] wal_buffers

2012-02-19 Thread Fujii Masao
On Mon, Feb 20, 2012 at 3:08 AM, Robert Haas robertmh...@gmail.com wrote: On Sun, Feb 19, 2012 at 9:46 AM, Euler Taveira de Oliveira eu...@timbira.com wrote: On 19-02-2012 02:24, Robert Haas wrote: I have attached tps scatterplots.  The obvious conclusion appears to be that, with only 16MB of

Re: [HACKERS] Initial 9.2 pgbench write results

2012-02-19 Thread Greg Smith
On 02/18/2012 02:35 PM, Robert Haas wrote: I see CheckpointWriteDelay calling BgBufferSync in 9.1. Background writing would stop during the sync phase and perhaps slow down a bit during checkpoint writing, but I don't think it was stopped completely. The sync phase can be pretty long

Re: [HACKERS] Initial 9.2 pgbench write results

2012-02-19 Thread Greg Smith
On 02/19/2012 05:37 AM, Simon Riggs wrote: Please retest with wal_buffers 128MB, checkpoint_segments 1024 The test parameters I'm using aim to run through several checkpoint cycles in 10 minutes of time. Bumping up against the ugly edges of resource bottlenecks is part of the test.

Re: [HACKERS] wal_buffers

2012-02-19 Thread Greg Smith
On 02/19/2012 12:24 AM, Robert Haas wrote: I think we might want to consider adjusting our auto-tuning formula for wal_buffers to allow for a higher cap, although this is obviously not enough data to draw any firm conclusions. That's an easy enough idea to throw into my testing queue. The

Re: [HACKERS] Future of our regular expression code

2012-02-19 Thread Greg Smith
On 02/19/2012 10:28 PM, Greg Stark wrote: One thing that concerns me more and more is that most sufficiently powerful regex implementations are susceptible to DOS attacks. There's a list of evil regexes at http://en.wikipedia.org/wiki/ReDoS The Perl community's reaction to Russ Cox's regex

Re: [HACKERS] leakproof

2012-02-19 Thread Don Baccus
On Feb 19, 2012, at 7:24 PM, Tom Lane wrote: Don Baccus dhog...@pacifier.com writes: On Feb 19, 2012, at 5:42 PM, Tom Lane wrote: Hmm, pure doesn't sound bad to me. Nice and short. Technically, pure is stronger than has no side effects: http://en.wikipedia.org/wiki/Pure_function Result

Re: [HACKERS] Future of our regular expression code

2012-02-19 Thread Jay Levitt
Stephen Frost wrote: Alright, I'll bite.. Which existing regexp implementation that's well written, well maintained, and which is well protected against malicious regexes should we be considering then? FWIW, there's a benchmark here that compares a number of regexp engines, including PCRE,

Re: [HACKERS] leakproof

2012-02-19 Thread Robert Haas
On Sun, Feb 19, 2012 at 10:24 PM, Tom Lane t...@sss.pgh.pa.us wrote: Don Baccus dhog...@pacifier.com writes: On Feb 19, 2012, at 5:42 PM, Tom Lane wrote: Hmm, pure doesn't sound bad to me.  Nice and short. Technically, pure is stronger than has no side effects:

Re: [HACKERS] Initial 9.2 pgbench write results

2012-02-19 Thread Robert Haas
On Sun, Feb 19, 2012 at 11:12 PM, Greg Smith g...@2ndquadrant.com wrote: I'm collecting one last bit of data before posting another full set of results, but I'm getting more comfortable the issue here is simply changes in the BGW behavior.  The performance regression tracks the background

Re: [HACKERS] Proposal: XML helper functions

2012-02-19 Thread Larry
But by using the above code: how do we deal with multiple matching values? For example: question name=“my_question tag java /tag tag c++ /tag /question In this case, perhaps I would want something like ---+- my_question | java my_question | c++ -- View this message in

Re: Scaling XLog insertion (was Re: [HACKERS] Moving more work outside WALInsertLock)

2012-02-19 Thread Amit Kapila
I was trying to understand this patch and had few doubts: 1. In PerformXLogInsert(), why there is need to check freespace when already during ReserveXLogInsertLocation(), the space is reserved. Is it possible that the record size is more than actually calculted in

Re: [HACKERS] Displaying accumulated autovacuum cost

2012-02-19 Thread Fujii Masao
On Sat, Feb 18, 2012 at 2:16 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Feb 17, 2012 at 5:04 AM, Fujii Masao masao.fu...@gmail.com wrote: Here are review comments: The document about EXPLAIN needs to be updated. You forgot to add the long-integer-valued property of