Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Jan Wieck
On 6/23/2006 3:10 PM, Mark Woodward wrote: This is NOT an in-place update. The whole MVCC strategy of keeping old versions around doesn't change. The only thing that does change is one level of indirection. Rather than keep references to all versions of all rows in indexes, keep only a

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Jan Wieck
On 6/23/2006 9:56 PM, [EMAIL PROTECTED] wrote: On Fri, Jun 23, 2006 at 03:08:34PM -0400, Bruce Momjian wrote: Tom Lane wrote: ... suggesting. We're having a hard enough time debugging and optimizing *one* storage model. I think the correct path forward is to stick with the same basic

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread mark
On Sat, Jun 24, 2006 at 03:29:47AM -0400, Jan Wieck wrote: It sounds like everybody agrees that things need to be fixed, and genuinely motivated people are trying to offer what they have to the table. One singe core team member responds vaguely in a way, you feel being supportive of your

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Heikki Linnakangas
On Fri, 23 Jun 2006, Jonah H. Harris wrote: On 6/23/06, Mark Woodward [EMAIL PROTECTED] wrote: Rather than keep references to all versions of all rows in indexes, keep only a reference to the first or key row of each row, and have the first version of a row form the head of a linked list to

Re: [HACKERS] libpq Describe Extension [WAS: Bytea and perl]

2006-06-24 Thread Volkan YAZICI
On Jun 16 08:21, Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: Volkan YAZICI wrote: The problem is, AFAICS, it's not possible to distinguish between a tuple returning query (T, ..., C, Z or T, E) and a description of a portal (T, Z). Therefore, I've created a global flag

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Mark Woodward
On 6/23/2006 3:10 PM, Mark Woodward wrote: This is NOT an in-place update. The whole MVCC strategy of keeping old versions around doesn't change. The only thing that does change is one level of indirection. Rather than keep references to all versions of all rows in indexes, keep only a

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Martijn van Oosterhout
On Sat, Jun 24, 2006 at 08:14:10AM -0400, Mark Woodward wrote: On 6/23/2006 3:10 PM, Mark Woodward wrote: This is NOT an in-place update. The whole MVCC strategy of keeping old versions around doesn't change. The only thing that does change is one level of indirection. Rather than keep

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Martijn van Oosterhout
On Fri, Jun 23, 2006 at 03:10:39PM -0400, Mark Woodward wrote: This is NOT an in-place update. The whole MVCC strategy of keeping old versions around doesn't change. The only thing that does change is one level of indirection. Rather than keep references to all versions of all rows in indexes,

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Heikki Linnakangas
On Sat, 24 Jun 2006, Mark Woodward wrote: I'm probably mistaken, but aren't there already forward references in tuples to later versions? If so, I'm only sugesting reversing the order and referencing the latest version. I thought I understood your idea, but now you lost me again. I thought

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Mark Woodward
On Sat, 24 Jun 2006, Mark Woodward wrote: I'm probably mistaken, but aren't there already forward references in tuples to later versions? If so, I'm only sugesting reversing the order and referencing the latest version. I thought I understood your idea, but now you lost me again. I thought

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Jonah H. Harris
On 6/24/06, Mark Woodward [EMAIL PROTECTED] wrote: Currently it looks like this: ver001-ver002-ver003-...-verN That's what t_ctid does now, right? Well, that's sort of stupid. Why not have it do this: ver001-verN-...-ver003-ver002-| Heh, because that's crazy. The first time you insert a

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Jonah H. Harris
On 6/24/06, Jonah H. Harris [EMAIL PROTECTED] wrote: Grr... need coffee... s/c_tid/ctid/g -- Jonah H. Harris, Software Architect | phone: 732.331.1300 EnterpriseDB Corporation| fax: 732.331.1301 33 Wood Ave S, 2nd Floor| [EMAIL PROTECTED] Iselin, New Jersey 08830

Re: [HACKERS] cygwin breakage (was: GPL Source and Copyright Questions)

2006-06-24 Thread Magnus Hagander
And why do win32 and cygwin now not include at all pg_config_os.h? It's a way to avoid the step to copy win32\port.h in msvc. configure copies it to pg_config_os.h. Since for win32 platforms (unfortunatly, at this point it considers cygwin win32..)that will always be port/win32.h, it

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Mark Woodward
On 6/24/06, Mark Woodward [EMAIL PROTECTED] wrote: Currently it looks like this: ver001-ver002-ver003-...-verN That's what t_ctid does now, right? Well, that's sort of stupid. Why not have it do this: ver001-verN-...-ver003-ver002-| Heh, because that's crazy. The first time you insert

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Jochem van Dieten
On 6/24/06, Mark Woodward wrote: ver001-verN-...-ver003-ver002-| ^-/ This will speed up almost *all* queries when there are more than two version of rows. OK, here is the behavior of an update: (1) Find the latest version of the row (2) Duplicate row and

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Martijn van Oosterhout
On Sat, Jun 24, 2006 at 09:23:28AM -0400, Mark Woodward wrote: Can you try to explain more carefully how the whole thing would work? What would an index tuple point to? What pointers would a heap tuple have? What would an index scan do to find the row version it's interested in? What

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Jonah H. Harris
On 6/24/06, Mark Woodward [EMAIL PROTECTED] wrote: In the scenario, as previously outlined: ver001-verN-...-ver003-ver2-| ^-/ So you want to always keep an old version around? -- Jonah H. Harris, Software Architect | phone: 732.331.1300 EnterpriseDB Corporation

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Mark Woodward
On 6/24/06, Mark Woodward [EMAIL PROTECTED] wrote: In the scenario, as previously outlined: ver001-verN-...-ver003-ver2-| ^-/ So you want to always keep an old version around? Prior to vacuum, it will be there anyway, and after vacuum, the new version will

Re: [HACKERS] cygwin breakage (was: GPL Source and Copyright Questions)

2006-06-24 Thread Tom Lane
Magnus Hagander [EMAIL PROTECTED] writes: Attached simple patch reverts this, as it clearly broke cygwin. Applied ... hopefully it didn't also break mingw ;-) regards, tom lane ---(end of broadcast)--- TIP 4: Have you

Re: [HACKERS] cygwin breakage (was: GPL Source and Copyright Questions)

2006-06-24 Thread Magnus Hagander
Attached simple patch reverts this, as it clearly broke cygwin. Applied ... hopefully it didn't also break mingw ;-) Oh, I tested that. It also didn't break msvc. //Magnus ---(end of broadcast)--- TIP 1: if posting/reading through Usenet,

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Jonah H. Harris
On 6/24/06, Mark Woodward [EMAIL PROTECTED] wrote: On 6/24/06, Mark Woodward [EMAIL PROTECTED] wrote: In the scenario, as previously outlined: ver001-verN-...-ver003-ver2-| ^-/ So you want to always keep an old version around? Prior to vacuum, it will be

Re: [HACKERS] Exporting type OID macros in a cleaner fashion

2006-06-24 Thread Greg Sabino Mullane
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The alternative I'm currently thinking about is to build and install an auto-generated file comparable to fmgroids.h, containing *only* the type OID macro #defines extracted from pg_type.h. This would require just a trivial amount of sed

[HACKERS] Gist does not build with VC++ anymore

2006-06-24 Thread Magnus Hagander
I've updated my VC++ build env with latest CVS, and it no longer builds because of changes to GIST: src\backend\access\gist\gistutil.c(237) : error C2057: expected constant expression src\backend\access\gist\gistutil.c(237) : error C2466: cannot allocate an array of constant size 0

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread PFC
What I see in this discussion is a huge amount of the grass must be greener on the other side syndrome, and hardly any recognition that every technique has its downsides and complications. Sure ;) MVCC generates dead rows, by its very nature ; however I see two trends in this :

Re: [HACKERS] Gist does not build with VC++ anymore

2006-06-24 Thread Magnus Hagander
I've updated my VC++ build env with latest CVS, and it no longer builds because of changes to GIST: src\backend\access\gist\gistutil.c(237) : error C2057: expected constant expression src\backend\access\gist\gistutil.c(237) : error C2466: cannot allocate an array of constant size 0

Re: [HACKERS] Gist does not build with VC++ anymore

2006-06-24 Thread Magnus Hagander
I've updated my VC++ build env with latest CVS, and it no longer builds because of changes to GIST: src\backend\access\gist\gistutil.c(237) : error C2057: expected constant expression src\backend\access\gist\gistutil.c(237) : error C2466: cannot allocate an array of constant

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Bruce Momjian
bruce wrote: Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: I think at some point we have to admit that _polling_ the tables, which is what autovacuum does, just isn't going to work well, no matter how much it is tweeked, and another approach should be considered for

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Mark Woodward
On 6/24/06, Mark Woodward [EMAIL PROTECTED] wrote: On 6/24/06, Mark Woodward [EMAIL PROTECTED] wrote: In the scenario, as previously outlined: ver001-verN-...-ver003-ver2-| ^-/ So you want to always keep an old version around? Prior to vacuum, it will

[HACKERS] Crash on initdb in MSVC++

2006-06-24 Thread Magnus Hagander
HEAD built with msvc++ crashes on initdb. I'd appreciate any pointers as to where to start looking... Unhandled exception at 0x0046b9c7 in postgres.exe: 0xC005: Access violation reading location 0x06b3. Backtrace: postgres.exe!_bt_start_vacuum(RelationData * rel=0x067f) Line

[HACKERS] vacuum row?

2006-06-24 Thread Mark Woodward
I originally suggested a methodology for preserving MVCC and everyone is confusing it as update in place, this isnot what I intended. How about a form of vacuum that targets a particular row? Is this possible? Would if have to be by transaction? ---(end of

Re: [HACKERS] Crash on initdb in MSVC++

2006-06-24 Thread Magnus Hagander
Nevermind this whole mail. My .bki files weren't updated. (installed new version of them in the wrong directory. Oops.) Updating that to correct files for HEAD mdae it pass again. //Magnus HEAD built with msvc++ crashes on initdb. I'd appreciate any pointers as to where to start looking...

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Heikki Linnakangas
On Sat, 24 Jun 2006, Bruce Momjian wrote: OK, I have an idea. Right now, an UPDATE where the old and new rows are on the same page have two index entries. What if we made only one index entry for both? We already have UPDATE chaining, where the old row points to the new one. If no key

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Jan Wieck
On 6/24/2006 9:23 AM, Mark Woodward wrote: On Sat, 24 Jun 2006, Mark Woodward wrote: I'm probably mistaken, but aren't there already forward references in tuples to later versions? If so, I'm only sugesting reversing the order and referencing the latest version. I thought I understood your

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Hannu Krosing
Ühel kenal päeval, R, 2006-06-23 kell 13:08, kirjutas Tom Lane: Csaba Nagy [EMAIL PROTECTED] writes: Surprisingly its mostly WAL traffic, the heap/index pages themselves are often not yet synced to disk by time of vacuum, so no additional traffic there. If you had made 5 updates per page

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Hannu Krosing
Ühel kenal päeval, R, 2006-06-23 kell 17:27, kirjutas Bruce Momjian: Jonah H. Harris wrote: On 6/23/06, Tom Lane [EMAIL PROTECTED] wrote: What I see in this discussion is a huge amount of the grass must be greener on the other side syndrome, and hardly any recognition that every

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Jan Wieck
On 6/22/2006 2:37 PM, Alvaro Herrera wrote: Adding back pgsql-hackers. Mark Woodward wrote: Mark Woodward wrote: Hmm, OK, then the problem is more serious than I suspected. This means that every index on a row has to be updated on every transaction that modifies that row. Is that

Re: Anyone still care about Cygwin? (was Re: [HACKERS] [CORE] GPL

2006-06-24 Thread Robert Treat
On Friday 23 June 2006 14:30, Andrew Dunstan wrote: Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: There are several supported platforms not represented on the buildfarm - e.g. the one HPUX member has never actually reported any results. Yeah, and this is not a good thing.

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Hannu Krosing
Ühel kenal päeval, L, 2006-06-24 kell 15:44, kirjutas Jan Wieck: That fixes the symptom, not the problem. The problem is performance steadily degrades over time. No, you got it backwards. The performance degradation is the symptom. The problem is that there are too many dead tuples in

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Martijn van Oosterhout
On Sat, Jun 24, 2006 at 10:04:43PM +0300, Hannu Krosing wrote: Maybe we could start from reusing the index tuples which point to invisible tuples ? The index is not MVCC anyway, so maybe it is easier to do in-place replacement there ? This probably has the same obstacles which have prevented

Re: [HACKERS] Buffer for inner and outer table

2006-06-24 Thread Daniel Xavier de Sousa
Hi,Please, Somebody can tell me, where the postgres control the buffer for inner and outer table, when it execute Nest_loop_join? I would want how to change the size this buffer andsee all statistics about thisThere is another doubt, how can I see the pages access (on Ram and HD)

Re: [HACKERS] Buffer for inner and outer table

2006-06-24 Thread Alvaro Herrera
Daniel Xavier de Sousa wrote: Somebody can tell me, where the postgres control the buffer for inner and outer table, when it execute Nest_loop_join? I would want how to change the size this buffer and see all statistics about this There is no such buffer. Buffers used in scans are

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Christopher Browne
Martha Stewart called it a Good Thing when [EMAIL PROTECTED] (Jan Wieck) wrote: On 6/22/2006 2:37 PM, Alvaro Herrera wrote: Adding back pgsql-hackers. Mark Woodward wrote: Mark Woodward wrote: Hmm, OK, then the problem is more serious than I suspected. This means that every index on a

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Bruce Momjian
Heikki Linnakangas wrote: On Sat, 24 Jun 2006, Bruce Momjian wrote: OK, I have an idea. Right now, an UPDATE where the old and new rows are on the same page have two index entries. What if we made only one index entry for both? We already have UPDATE chaining, where the old row

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Alvaro Herrera
Mark Woodward wrote: The update behavior of PostgreSQL is probably the *last* serious issue. Debate all you want, vacuum mitigates the problem to varying levels, fixing the problem will be a huge win. If the update behavior gets fixed, I can't think of a single issue with postgresql that

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-24 Thread Bruce Momjian
Hannu Krosing wrote: ?hel kenal p?eval, R, 2006-06-23 kell 13:08, kirjutas Tom Lane: Csaba Nagy [EMAIL PROTECTED] writes: Surprisingly its mostly WAL traffic, the heap/index pages themselves are often not yet synced to disk by time of vacuum, so no additional traffic there. If you had

Re: [HACKERS] Gist does not build with VC++ anymore

2006-06-24 Thread Bruce Momjian
Patch applied. It seems offsetof() can only find structure members in MSVC, not the offset of array elements in structures. Anyway, offsetof() was finding the first element, so the [0] is not needed. --- Magnus Hagander

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-24 Thread Bruce Momjian
Tom Lane wrote: [ redirecting to -hackers, as I see no need for this to be a core issue ] Charles Comiskey [EMAIL PROTECTED] writes: Hello, I've recently looked through the PostgreSQL code and a couple of questions surfaced. I was hoping someone here may be able to answer them. Two

Re: [HACKERS] vacuum row?

2006-06-24 Thread Alvaro Herrera
Mark Woodward wrote: I originally suggested a methodology for preserving MVCC and everyone is confusing it as update in place, this isnot what I intended. It doesn't make sense, but maybe vacuuming a page would. Naturally, it would need to wholly scan all the indexes to clean'em up, so it's