Re: [HACKERS] rc tarball built with older flex version?

2009-06-22 Thread Peter Eisentraut
On Monday 22 June 2009 00:17:06 Marc G. Fournier wrote: On Thu, 18 Jun 2009, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: I noticed that the rc1 tarball includes scanner files that are built with an older flex version that generates warnings with our default compilation flags.

Re: [HACKERS] rc tarball built with older flex version?

2009-06-22 Thread Marc G. Fournier
On Mon, 22 Jun 2009, Peter Eisentraut wrote: Well, I rarely test the actual release source tarball, so it might have been like that forever. 'k ... I swore I haven't changed anything over there in awhile, so was most confused as to where this sudden error came from ... Marc G.

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Alan Li
Hi Tom, How much concern is there for the contention for use cases where the WAL can't be bypassed? Thanks, Alan On Sun, Jun 21, 2009 at 10:00 AM, Tom Lane t...@sss.pgh.pa.us wrote: Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: The following copying 3M rows(each) into a seperate

Re: [HACKERS] Missing Docs for MOVE direction?

2009-06-22 Thread Peter Eisentraut
On Monday 22 June 2009 03:15:57 David E. Wheeler wrote: On Jun 21, 2009, at 5:07 PM, David E. Wheeler wrote: I was just looking at the documentation for cursors command, and noticed that the MOVE command's direction argument doesn't seem to have documentation for its possible values.

Re: [HACKERS] Synch Rep: communication between backends and walsender

2009-06-22 Thread Fujii Masao
Hi, On Sat, Jun 20, 2009 at 6:05 PM, Markus Wannermar...@bluegap.ch wrote: Reading the past messages on this topic, I realized that this problem so far only affects HPUX. I fear the proposed UDP/semaphores approach might have a similar gotcha on at least one of the supported platforms, too.

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Heikki Linnakangas
Tom Lane wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: I was going to say that since we flush the WAL every 16MB anyway (at every XLOG file switch), you shouldn't see any benefit with larger ring buffers, since to fill 16MB of data you're not going to write more than

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Simon Riggs
On Mon, 2009-06-22 at 10:52 +0300, Heikki Linnakangas wrote: Tom Lane wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: I was going to say that since we flush the WAL every 16MB anyway (at every XLOG file switch), you shouldn't see any benefit with larger ring

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread David Fetter
On Mon, Jun 22, 2009 at 02:00:10PM +0900, KaiGai Kohei wrote: KaiGai Kohei wrote: The todo list says, as follows: * Binary Data o Add security checks for large objects http://wiki.postgresql.org/wiki/Todo#Binary_Data Is anyone working on? or interested in? OK, it seems to me

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
Alan Li a...@truviso.com writes: How much concern is there for the contention for use cases where the WAL can't be bypassed? If you mean is something going to be done about it in 8.4, the answer is no. This is a pre-existing issue that there is no simple fix for.

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread Peter Eisentraut
On Monday 22 June 2009 16:24:44 David Fetter wrote: On Mon, Jun 22, 2009 at 02:00:10PM +0900, KaiGai Kohei wrote: KaiGai Kohei wrote: The todo list says, as follows: * Binary Data o Add security checks for large objects http://wiki.postgresql.org/wiki/Todo#Binary_Data Is

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Andrew Dunstan
Tom Lane wrote: Alan Li a...@truviso.com writes: How much concern is there for the contention for use cases where the WAL can't be bypassed? If you mean is something going to be done about it in 8.4, the answer is no. This is a pre-existing issue that there is no simple fix for.

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: Tom Lane wrote: Alan Li a...@truviso.com writes: How much concern is there for the contention for use cases where the WAL can't be bypassed? If you mean is something going to be done about it in 8.4, the answer is no. This is a pre-existing issue

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Andrew Dunstan
Tom Lane wrote: I thought he was asking for a solution to the problem of WALInsertLock contention. In any case, we have WAL bypass on a table by table basis now, don't we? If we do I'm ignorant of it ;-) How do we say Never WAL this table? cheers andrew --

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: Tom Lane wrote: I thought he was asking for a solution to the problem of WALInsertLock contention. In any case, we have WAL bypass on a table by table basis now, don't we? If we do I'm ignorant of it ;-) How do we say Never WAL this table? Make it

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Aidan Van Dyk
* Andrew Dunstan and...@dunslane.net [090622 10:47]: If we do I'm ignorant of it ;-) How do we say Never WAL this table? CREATE TEMPORARY TABLE ... a. -- Aidan Van Dyk Create like a god, ai...@highrise.ca

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: The more useful case for data load is create or truncate it in the same transaction, of course. Unfortunately, WAL bypass also requires not being in archive mode with no way to turn that off w/o a server restart, aiui. Thanks,

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
Stephen Frost sfr...@snowman.net writes: * Tom Lane (t...@sss.pgh.pa.us) wrote: The more useful case for data load is create or truncate it in the same transaction, of course. Unfortunately, WAL bypass also requires not being in archive mode with no way to turn that off w/o a server restart,

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: Stephen Frost sfr...@snowman.net writes: Unfortunately, WAL bypass also requires not being in archive mode with no way to turn that off w/o a server restart, aiui. Well, if you're trying to archive then you certainly wouldn't want WAL off, so I'm

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread David Fetter
On Mon, Jun 22, 2009 at 05:18:51PM +0300, Peter Eisentraut wrote: On Monday 22 June 2009 16:24:44 David Fetter wrote: On Mon, Jun 22, 2009 at 02:00:10PM +0900, KaiGai Kohei wrote: KaiGai Kohei wrote: The todo list says, as follows: * Binary Data o Add security checks for large

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Simon Riggs
On Mon, 2009-06-22 at 11:14 -0400, Tom Lane wrote: Stephen Frost sfr...@snowman.net writes: * Tom Lane (t...@sss.pgh.pa.us) wrote: The more useful case for data load is create or truncate it in the same transaction, of course. Unfortunately, WAL bypass also requires not being in

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: I was thinking it might be beneficial to be able to defer writing WAL until COPY is complete, so heap_sync would either fsync the whole heap file or copy the whole file to WAL. What about indexes? regards, tom lane -- Sent

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Simon Riggs
On Mon, 2009-06-22 at 11:24 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: I was thinking it might be beneficial to be able to defer writing WAL until COPY is complete, so heap_sync would either fsync the whole heap file or copy the whole file to WAL. What about

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread Tom Lane
David Fetter da...@fetter.org writes: On Mon, Jun 22, 2009 at 05:18:51PM +0300, Peter Eisentraut wrote: MED is management of external data, whereas the large objects are internal, no? It depends on your definition. The lo interface is pretty much to objects on the file system directly.

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Andrew Dunstan
Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Tom Lane wrote: I thought he was asking for a solution to the problem of WALInsertLock contention. In any case, we have WAL bypass on a table by table basis now, don't we? If we do I'm ignorant of it ;-) How do

Re: [HACKERS] Synch Rep: communication between backends and walsender

2009-06-22 Thread Markus Wanner
Hi, Fujii Masao wrote: The perfect solution seems to be to remove SA_RESTART and handle EINTR in an appropriate way after every syscalls. But, this is very tough job and has much influence on whole source code, so I don't think that this should be done as part of synch rep. Especially note

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Tom Lane wrote: I'm not convinced that WAL segment boundaries are particularly relevant to this. The unit of flushing is an 8K page, not a segment. We fsync() the old WAL segment every time we switch to a new WAL segment.

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread David Fetter
On Mon, Jun 22, 2009 at 11:31:45AM -0400, Tom Lane wrote: David Fetter da...@fetter.org writes: On Mon, Jun 22, 2009 at 05:18:51PM +0300, Peter Eisentraut wrote: MED is management of external data, whereas the large objects are internal, no? It depends on your definition. The lo

[HACKERS] Re: [BUGS] BUG #4862: different results in to_date() between 8.3.7 8.4.RC1

2009-06-22 Thread Brendan Jurd
2009/6/23 Tom Lane t...@sss.pgh.pa.us: Brendan Jurd dire...@gmail.com writes: I should be able to get the same results by snipping an extra strspace_len() characters in the new code path in from_char_parse_int_len().  This ought to be a one-line fix that doesn't clobber the good parts of my

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Dimitri Fontaine
Le 22 juin 2009 à 17:24, Tom Lane t...@sss.pgh.pa.us a écrit : Simon Riggs si...@2ndquadrant.com writes: I was thinking it might be beneficial to be able to defer writing WAL until COPY is complete, so heap_sync would either fsync the whole heap file or copy the whole file to WAL. What

Re: [HACKERS] [BUGS] BUG #4862: different results in to_date() between 8.3.7 8.4.RC1

2009-06-22 Thread Tom Lane
Brendan Jurd dire...@gmail.com writes: Here's a one-line patch to fix a regression in the new from_char code I introduced into 8.4. Versions = 8.3 skipped over any whitespace immediately preceding any integer field, and this behaviour was lost in HEAD when my from_char patch was committed

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: I wonder though whether the wal_buffers setting interacts with the ring size. Has everyone who's tested this used the same 16MB wal_buffers setting as in Alan's original scenario? I had been using his postgresql.conf file, then added autovacuum = off.

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
Kevin Grittner kevin.gritt...@wicourts.gov writes: Tom Lane t...@sss.pgh.pa.us wrote: I wonder though whether the wal_buffers setting interacts with the ring size. Has everyone who's tested this used the same 16MB wal_buffers setting as in Alan's original scenario? I had been using his

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Greg Smith
On Mon, 22 Jun 2009, Kevin Grittner wrote: When I tried with setting the ring size to 16MB, I accidentally left off the step to copy the postgresql.conf file, and got better performance. Do you have happen to have a build with assertions turned on? That is one common cause of performance

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Kevin Grittner
Greg Smith gsm...@gregsmith.com wrote: Do you have happen to have a build with assertions turned on? Nope. I showed my ./configure options upthread, but can confirm with pg_config: BINDIR = /usr/local/pgsql-8.4rc1/bin DOCDIR = /usr/local/pgsql-8.4rc1/share/doc HTMLDIR =

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Kevin Grittner
Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: A 25-30% performance regression in our main bulk loading mechanism should at least be explained before the release... I think a performance regression of that magnitude merits holding up a release to resolve. Note that in a follow-up

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Huh, that's bizarre. I can see that increasing shared_buffers should make no difference in this test case (we're not using them all anyway). But why should increasing wal_buffers make it slower? I forget the walwriter's control algorithm at the moment

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Kevin Grittner
I wrote: Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: A 25-30% performance regression in our main bulk loading mechanism should at least be explained before the release... I think a performance regression of that magnitude merits holding up a release to resolve. Wow. That

[HACKERS] Changed error message for blocks by prepared transactions

2009-06-22 Thread Andreas 'ads' Scherbaum
Hello, the small attached patch changes the error message for a blocked database in case there are prepared transactions. The original message accessed by other users is misleading. Example: - snip - postgres=# begin; BEGIN postgres=# prepare transaction 'abc'; PREPARE TRANSACTION

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
Kevin Grittner kevin.gritt...@wicourts.gov writes: The checkpoint_segments seems dramatic enough to be real. I wonder if the test is short enough that it never got around to re-using any of them, so it was doing extra writes for the initial creation during the test? That's exactly what I was

Re: [HACKERS] Changed error message for blocks by prepared transactions

2009-06-22 Thread Tom Lane
Andreas 'ads' Scherbaum adsm...@wars-nicht.de writes: the small attached patch changes the error message for a blocked database in case there are prepared transactions. Isn't this duplicative of the errdetail_busy_db code? And anyway I do not see a reason not to consider prepared transactions

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Alan Li
On Mon, Jun 22, 2009 at 7:16 AM, Tom Lane t...@sss.pgh.pa.us wrote: Alan Li a...@truviso.com writes: How much concern is there for the contention for use cases where the WAL can't be bypassed? If you mean is something going to be done about it in 8.4, the answer is no. This is a

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread KaiGai Kohei
David Fetter wrote: On Mon, Jun 22, 2009 at 11:31:45AM -0400, Tom Lane wrote: David Fetter da...@fetter.org writes: On Mon, Jun 22, 2009 at 05:18:51PM +0300, Peter Eisentraut wrote: MED is management of external data, whereas the large objects are internal, no? It depends on your definition.

[HACKERS] building without perl

2009-06-22 Thread Robert Haas
I'm a bit confused as to what we're trying to support. While I was poking around tonight, I notice that we have both ./src/backend/utils/Gen_fmgrtab.sh and ./src/backend/utils/Gen_fmgrtab.pl, ostensibly because: # The reason for implementing this functionality twice is that we don't # require

Re: [HACKERS] building without perl

2009-06-22 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I'm a bit confused as to what we're trying to support. You are not supposed to need perl to build from a release tarball (except on Windows which doesn't have a decent make or shell, so we depend on perl instead). You do need it to build from a CVS

Re: [HACKERS] building without perl

2009-06-22 Thread Robert Haas
On Mon, Jun 22, 2009 at 10:12 PM, Tom Lanet...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: I'm a bit confused as to what we're trying to support. You are not supposed to need perl to build from a release tarball (except on Windows which doesn't have a decent make or shell,

Re: [HACKERS] building without perl

2009-06-22 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: So, is our policy that it's OK to use perl to generate things, as long as they are things that are architecture-independent and configuration-independent therefore could be pregenerated for the release tarball? If so, it seems we might be able to punt

Re: [HACKERS] building without perl

2009-06-22 Thread Andrew Dunstan
Tom Lane wrote: Actually, further investigation discloses that someone broke the no-Perl defense in psql's makefile back in 8.0, and we hadn't noticed till now. So it certainly seems to be the case that nobody associated with the Postgres project works in a Perl-free environment.

Re: [HACKERS] building without perl

2009-06-22 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: Tom Lane wrote: Actually, further investigation discloses that someone broke the no-Perl defense in psql's makefile back in 8.0, and we hadn't noticed till now. Don't we ship sql_help.h in the tarball? I thought that's all the perl was used for in

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread David Fetter
On Tue, Jun 23, 2009 at 10:38:59AM +0900, KaiGai Kohei wrote: David Fetter wrote: On Mon, Jun 22, 2009 at 11:31:45AM -0400, Tom Lane wrote: David Fetter da...@fetter.org writes: On Mon, Jun 22, 2009 at 05:18:51PM +0300, Peter Eisentraut wrote: MED is management of external data, whereas

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread KaiGai Kohei
KaiGai Kohei wrote: KaiGai Kohei wrote: The todo list says, as follows: * Binary Data o Add security checks for large objects http://wiki.postgresql.org/wiki/Todo#Binary_Data Is anyone working on? or interested in? OK, it seems to me nobody is working on the todo item. I'll also

Re: [HACKERS] building without perl

2009-06-22 Thread Andrew Dunstan
Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Tom Lane wrote: Actually, further investigation discloses that someone broke the no-Perl defense in psql's makefile back in 8.0, and we hadn't noticed till now. Don't we ship sql_help.h in the tarball? I thought

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread KaiGai Kohei
David Fetter wrote: On Tue, Jun 23, 2009 at 10:38:59AM +0900, KaiGai Kohei wrote: David Fetter wrote: On Mon, Jun 22, 2009 at 11:31:45AM -0400, Tom Lane wrote: David Fetter da...@fetter.org writes: On Mon, Jun 22, 2009 at 05:18:51PM +0300, Peter Eisentraut wrote: MED is management of