Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Peter Eisentraut
Tom Lane wrote: I think this needs to be treated as experimental until it's got a few more than zero miles under its belt. OK, then maybe we should document that. I wouldn't be too surprised to find that we have to implement it as a run-time switch instead of compile-time, in order to not

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Zeugswetter Andreas OSB SD
Why is this not the default when supported? Fear. Maybe eventually, but right now I think it's too risky. One point that I already found out the hard way is that sizeof(off_t) = 8 does not guarantee the availability of largefile support; there can also be filesystem-level

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Zeugswetter Andreas OSB SD
Why is this not the default when supported? I am wondering both from the point of view of the user, and in terms of development direction. Also it would get more buildfarm coverage if it were default. If it breaks something we'll notice earlier. No we don't, because the buildfarm does

Re: [PATCHES] [PERFORM] Very slow (2 tuples/second) sequential scan after bulk insert; speed returns to ~500 tuples/second after commit

2008-03-11 Thread Heikki Linnakangas
(moved to pgsql-patches, as there's a patch attached) Tom Lane wrote: Getting rid of the linked-list representation would be a win in a couple of ways --- we'd not need the bogus list of XIDs support in pg_list.h, and xactGetCommittedChildren would go away. OTOH AtSubCommit_childXids would get

Re: [PATCHES] Bulk Insert tuning

2008-03-11 Thread Heikki Linnakangas
Gavin Sherry wrote: On Tue, Feb 26, 2008 at 02:43:51PM +, Simon Riggs wrote: Following patch implements a simple mechanism to keep a buffer pinned while we are bulk loading. CK Tan and I worked on something similar but the problem we discovered was self locking. Consider a primary key: we

Re: [PATCHES] TransactionIdIsInProgress() cache

2008-03-11 Thread Alvaro Herrera
I didn't check whether your transformation is correct, but if so then it can be changed like this and save the extra XidDidCommit call: xvac_committed = TransactionIdDidCommit(xvac); if (xvac_committed) { /* committed */ } else if (!TransactionIdIsInProgress(xvac))

Re: [PATCHES] [PERFORM] Very slow (2 tuples/second) sequential scan after bulk insert; speed returns to ~500 tuples/second after commit

2008-03-11 Thread Pavan Deolasee
On Tue, Mar 11, 2008 at 6:04 PM, Heikki Linnakangas [EMAIL PROTECTED] wrote: (moved to pgsql-patches, as there's a patch attached) I couldn't let this case go, so I wrote a patch. I replaced the linked list with an array that's enlarged at AtSubCommit_childXids when necessary. We can

Re: [PATCHES] TransactionIdIsInProgress() cache

2008-03-11 Thread Pavan Deolasee
On Tue, Mar 11, 2008 at 6:37 PM, Alvaro Herrera [EMAIL PROTECTED] wrote: I didn't check whether your transformation is correct, but if so then it can be changed like this and save the extra XidDidCommit call: xvac_committed = TransactionIdDidCommit(xvac); if (xvac_committed) {

Re: [PATCHES] [PERFORM] Very slow (2 tuples/second) sequential scan after bulk insert; speed returns to ~500 tuples/second after commit

2008-03-11 Thread Alvaro Herrera
Heikki Linnakangas wrote: I couldn't let this case go, so I wrote a patch. I replaced the linked list with an array that's enlarged at AtSubCommit_childXids when necessary. Do you still need to palloc the return value from xactGetCommittedChildren? Perhaps you can save the

Re: [PATCHES] TransactionIdIsInProgress() cache

2008-03-11 Thread Heikki Linnakangas
Alvaro Herrera wrote: I didn't check whether your transformation is correct, but if so then it can be changed like this and save the extra XidDidCommit call: xvac_committed = TransactionIdDidCommit(xvac); if (xvac_committed) { /* committed */ } else if

Re: [PATCHES] [PERFORM] Very slow (2 tuples/second) sequentialscan after bulk insert; speed returns to ~500 tuples/second aftercommit

2008-03-11 Thread Heikki Linnakangas
Alvaro Herrera wrote: Heikki Linnakangas wrote: I couldn't let this case go, so I wrote a patch. I replaced the linked list with an array that's enlarged at AtSubCommit_childXids when necessary. Do you still need to palloc the return value from xactGetCommittedChildren? Perhaps you can

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Larry Rosenman
On Mon, 10 Mar 2008, Tom Lane wrote: Peter Eisentraut [EMAIL PROTECTED] writes: Tom Lane wrote: Applied with minor corrections. Why is this not the default when supported? Fear. Maybe eventually, but right now I think it's too risky. One point that I already found out the hard way is

Re: [PATCHES] TransactionIdIsInProgress() cache

2008-03-11 Thread Alvaro Herrera
Heikki Linnakangas wrote: Nope, that's not good. Per comments in tqual.c, you have to call TransactionIdIsInProgress *before* TransactionIdDidCommit, to avoid this race condition: 1. Xact A inserts a record 2. Xact B does TransactionIdDidCommit, which retuns false because it's still

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Tom Lane
Peter Eisentraut [EMAIL PROTECTED] writes: Tom Lane wrote: I think this needs to be treated as experimental until it's got a few more than zero miles under its belt. OK, then maybe we should document that. Agreed, but at this point we don't even know what hazards we need to document.

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Tom Lane
Zeugswetter Andreas OSB SD [EMAIL PROTECTED] writes: Exactly, e.g. AIX is one of those. jfs (not the newer jfs2) has an option to enable large files, which is not the default and cannot be changed post crfs. And even if it is enabled, jfs has a 64 Gb filesize limit ! Anybody know others that

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Zdenek Kotala
Tom Lane napsal(a): Zeugswetter Andreas OSB SD [EMAIL PROTECTED] writes: Exactly, e.g. AIX is one of those. jfs (not the newer jfs2) has an option to enable large files, which is not the default and cannot be changed post crfs. And even if it is enabled, jfs has a 64 Gb filesize limit ! Anybody

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Tom Lane
Zdenek Kotala [EMAIL PROTECTED] writes: Tom Lane napsal(a): These examples suggest that maybe what we want is not so much a no segments ever mode as a segment size larger than 1GB. PS: ZFS is happy with 2^64bit size and UFS has 1TB file size limit (depends on solaris version) So even on

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Bruce Momjian
Tom Lane wrote: Zdenek Kotala [EMAIL PROTECTED] writes: Tom Lane napsal(a): These examples suggest that maybe what we want is not so much a no segments ever mode as a segment size larger than 1GB. PS: ZFS is happy with 2^64bit size and UFS has 1TB file size limit (depends on solaris

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Zdenek Kotala
Tom Lane napsal(a): Zdenek Kotala [EMAIL PROTECTED] writes: Tom Lane napsal(a): These examples suggest that maybe what we want is not so much a no segments ever mode as a segment size larger than 1GB. PS: ZFS is happy with 2^64bit size and UFS has 1TB file size limit (depends on solaris

Re: [PATCHES] Remove FATAL from pg_lzdecompress

2008-03-11 Thread Zdenek Kotala
Tom Lane napsal(a): Zdenek Kotala [EMAIL PROTECTED] writes: I attach patch which adds boundaries check and memory overwriting protection when compressed data are corrupted. Applied with revisions --- it appeared to me that it got the corner case wrong where we find a tag just at the end of

Re: [PATCHES] TransactionIdIsInProgress() cache

2008-03-11 Thread Tom Lane
Heikki Linnakangas [EMAIL PROTECTED] writes: I presume the case where this would help would be when you populate a large table, with COPY for example, and the run a seq scan on it. As all rows in the table have the same xmin, you keep testing for the same XID over and over again. Yeah, that's

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Tom Lane
Zdenek Kotala [EMAIL PROTECTED] writes: I think current patch could stay in CVS and I will rip out non segment code path in a new patch. Sure, I feel no need to revert what's applied. Have at it. regards, tom lane -- Sent via pgsql-patches mailing list

Re: [PATCHES] actualised forgotten Magnus's patch for plpgsql MOVE statement

2008-03-11 Thread Bruce Momjian
Added to TODO for pl/pgsql: o Review handling of MOVE and FETCH http://archives.postgresql.org/pgsql-patches/2007-04/msg00527.php --- Pavel Stehule wrote: I would argue that we should likewise not allow them in

Re: [PATCHES] Remove FATAL from pg_lzdecompress

2008-03-11 Thread Tom Lane
Zdenek Kotala [EMAIL PROTECTED] writes: I attach patch which adds boundaries check and memory overwriting protection when compressed data are corrupted. Good point. Is there plan to applied also on other branch? I wasn't planning to back-patch it. Given the lack of field reports of

Re: [PATCHES] TransactionIdIsInProgress() cache

2008-03-11 Thread Simon Riggs
On Tue, 2008-03-11 at 12:57 +, Heikki Linnakangas wrote: We could do this instead: if (TransactionIdDidCommit(xvac)) { /* committed */ } else if (!TransactionIdIsInProgress(xvac)) { if (TransactionIdDidCommit(xvac)) { /* committed */ }

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Peter Eisentraut
Zdenek Kotala wrote: Yes, agree. It seems only ZFS is OK at this moment and if somebody sets 32TB he gets nonsegment mode anyway. Surely if you set the segment size to INT64_MAX, you will get nonsegmented behavior anyway, so two code paths might not be necessary at all. I looked into posix

Re: [PATCHES] TransactionIdIsInProgress() cache

2008-03-11 Thread Simon Riggs
On Tue, 2008-03-11 at 13:15 -0400, Tom Lane wrote: Maybe instead TransactionIdIsKnownCompleted? Works for me. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk -- Sent via pgsql-patches mailing list

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Tom Lane
Peter Eisentraut [EMAIL PROTECTED] writes: Zdenek Kotala wrote: Maybe we can put additional test into configure and collect appropriate data from buildfarm. It might be good to just check first if it returns realistic values for the example cases that have been mentioned. Yeah, please just

Re: [PATCHES] Have vacuum emit a warning when it runs out of maintenance_work_mem

2008-03-11 Thread Bruce Momjian
Added to TODO for VACUUM: * Consider a more compact data representation for dead rows http://archives.postgresql.org/pgsql-patches/2007-05/msg00143.php --- Tom Lane wrote: Heikki Linnakangas [EMAIL PROTECTED] writes:

Re: [PATCHES] TransactionIdIsInProgress() cache

2008-03-11 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: No, I haven't done any formal performance testing on it. It seemed an obvious hole that everyone would agree we should avoid, since we can do it so cheaply: one integer comparison against scanning the whole procarray and taking an LWlock. [ after reading

Re: [PATCHES] TransactionIdIsInProgress() cache

2008-03-11 Thread Tom Lane
I wrote: [ after reading the patch a bit closer... ] Actually, it's not as obvious as all that. TransactionIdIsInProgress already falls out before the proposed test for any XID RecentXmin, which means that in reality it's fairly probable that the target XID is running. So while this test

Re: [PATCHES] trace_checkpoint parameter patch

2008-03-11 Thread Bruce Momjian
Now that we have log_checkpoints, consider resubmitting this with just the new information you want to add. Thanks. --- Satoshi Nagayasu wrote: Hi all, Here is a brand new patch to log a checkpointing load information

Re: [PATCHES] TransactionIdIsInProgress() cache

2008-03-11 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: We currently have a single item cache of the last checked TransactionId, which optimises the call to TransactionIdDidCommit() during HeapTupleSatisfiesMVCC() and partners. Before we call TransactionIdDidCommit() we always call

Re: [PATCHES] [PERFORM] Very slow (2 tuples/second) sequential scan after bulk insert; speed returns to ~500 tuples/second after commit

2008-03-11 Thread Tom Lane
Heikki Linnakangas [EMAIL PROTECTED] writes: I initially thought that using a single palloc'd array to hold all the XIDs would introduce a new limit on the number committed subtransactions, thanks to MaxAllocSize, but that's not the case. Without patch, we actually allocate an array like

Re: [PATCHES] Load Distributed Checkpoints, final patch

2008-03-11 Thread Bruce Momjian
Added to TODO: * Test to see if calling PreallocXlogFiles() from the background writer will help with WAL segment creation latency http://archives.postgresql.org/pgsql-patches/2007-06/msg00340.php --- Tom Lane wrote:

Re: [PATCHES] SPI-header-files safe for C++-compiler

2008-03-11 Thread Bruce Momjian
Added to TODO: * Add automated check for invalid C++ source code constructs http://archives.postgresql.org/pgsql-patches/2007-07/msg00056.php --- Tom Lane wrote: Stefan Kaltenbrunner [EMAIL PROTECTED] writes: wild

[PATCHES] Proposed patch for LISTEN/NOTIFY race condition

2008-03-11 Thread Tom Lane
This patch responds to the problem reported by Laurent Birtz here: http://archives.postgresql.org/pgsql-bugs/2008-03/msg00094.php The difficulty is that LISTEN modifies pg_listener and then releases lock early, so that there is an interval where a concurrent NOTIFY doesn't see the pg_listener

Re: [PATCHES] [BUGS] psql \COPY accepts multiple NULL AS

2008-03-11 Thread Bruce Momjian
Stephen Frost wrote: -- Start of PGP signed section. Greetings, psql seems to accept syntax which would be rejected by the server, which seems a bit odd: sfrost*=# \copy billing_data from ../BillingSamplePricerFile.csv with csv header quote as '' null as 'abc' null as

Re: [PATCHES] Proposed patch for LISTEN/NOTIFY race condition

2008-03-11 Thread Tom Lane
I wrote: The patch disallows LISTEN/NOTIFY in a prepared transaction, I meant LISTEN/UNLISTEN, of course. See pghackers discussion here: http://archives.postgresql.org/pgsql-hackers/2008-03/msg00290.php regards, tom lane -- Sent via pgsql-patches mailing list

Re: [PATCHES] Memory leak in nodeAgg

2008-03-11 Thread Bruce Momjian
Added to TODO: * Consider simplifying how memory context resets handle child contexts http://archives.postgresql.org/pgsql-patches/2007-08/msg00067.php --- Neil Conway wrote: On Mon, 2007-08-06 at 18:52 -0400, Tom

Re: [PATCHES] Configurable Penalty Costs for Levenshtein

2008-03-11 Thread Tom Lane
Volkan YAZICI [EMAIL PROTECTED] writes: I noticed a small typo in the patch. prev = palloc((m + n) * sizeof(char)); line should look like prev = palloc(2 * m * sizeof(char)); instead. If that's wrong, aren't the comments and the length restriction limit also wrong?