Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Alvaro Herrera
Adding back pgsql-hackers. Mark Woodward wrote: > > Mark Woodward wrote: > > > >> Hmm, OK, then the problem is more serious than I suspected. > >> This means that every index on a row has to be updated on every > >> transaction that modifies that row. Is that correct? > > > > Add an index entry, y

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Josh Berkus
Tom, > [ shrug... ] I'm not planning to panic; we've still got explicit GPL > code that's not been cleaned out of contrib/ yet. (Um, weren't you on > the hook to move those modules to pgfoundry projects?) Yeah, thanks for reminding me. Will do before feature freeze. As soon as I can figure

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Mark Woodward
> >> As you can see, in about a minute at high load, this very simple table >> lost about 10% of its performance, and I've seen worse based on update >> frequency. Before you say this is an obscure problem, I can tell you it >> isn't. I have worked with more than a few projects that had to switch

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Bort, Paul
> > so presumably this is only needed for old Cygwin versions. Can anyone > say how old "1001" is and whether we still ought to care about it? > IIRC, I've been on 1.5.x for at least three years. 1.0/1.1 seems to be around 2000/2001, based on a quick Google. So it's definitely older than PG 7.3

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Yeah, you should be able to find the older version easily enough, if you > arrived at the newer version and realized you needed to visit the older > version. But this fails in scenarios where you are searching on a > column that's been updated --- the index

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
Josh Berkus writes: > Yeah, thanks for reminding me. Will do before feature freeze. As soon as > I can figure out how to generate a patch that removes directories. Don't worry about that; CVS never deletes directories. But anyway, I can easily handle removing the code. I just want someone e

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Josh Berkus
Tom, > adddepend > dbase > dbmirror > fulltextindex > mSQL-interface > mac > oracle > tips > userlock I think you're right. I will do this before I leave town on the 30th. -- --Josh Josh Berkus PostgreSQL @ Sun San Francisco ---(end of broadcast)--

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread PFC
What you seem not to grasp at this point is a large web-farm, about 10 or more servers running PHP, Java, ASP, or even perl. The database is usually the most convenient and, aside from the particular issue we are talking about, best suited. The answer is sticky sessions : each user is assi

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Stefan Kaltenbrunner
Tom Lane wrote: > Josh Berkus writes: >> Yeah, thanks for reminding me. Will do before feature freeze. As soon as >> I can figure out how to generate a patch that removes directories. > > Don't worry about that; CVS never deletes directories. But anyway, > I can easily handle removing the co

Re: [HACKERS] xlog viewer proposal

2006-06-22 Thread Jonah H. Harris
On 6/22/06, Tom Lane <[EMAIL PROTECTED]> wrote: Jonah, I've been working with this system for years, and it's not that easy to "handle the differences with a few macros". True, it is harder than just that. I didn't mean to make light of it at all, just that a good amount of design upfront woul

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Andrew Dunstan
Bort, Paul wrote: so presumably this is only needed for old Cygwin versions. Can anyone say how old "1001" is and whether we still ought to care about it? IIRC, I've been on 1.5.x for at least three years. 1.0/1.1 seems to be around 2000/2001, based on a quick Google. So it's definite

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Mark Woodward
> >> What you seem not to grasp at this point is a large web-farm, about 10 >> or >> more servers running PHP, Java, ASP, or even perl. The database is >> usually >> the most convenient and, aside from the particular issue we are talking >> about, best suited. > > The answer is sticky session

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Andrew Dunstan
PFC wrote: What you seem not to grasp at this point is a large web-farm, about 10 or more servers running PHP, Java, ASP, or even perl. The database is usually the most convenient and, aside from the particular issue we are talking about, best suited. The answer is sticky sessions

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Stefan Kaltenbrunner
Tom Lane wrote: > The bad news is that except in the stats_command_string cases, HEAD > is noticeably slower than 8.1 on the machine with slow gettimeofday. > In the single-transaction test this might be blamed on the addition > of statement_timestamp support (which requires a gettimeofday per > s

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread David Fetter
On Thu, Jun 22, 2006 at 07:01:38PM +0200, Lukas Smith wrote: > Jochem van Dieten wrote: > > >make the session handler smarter? And if you can't do that, put > >some logic in the session table that turns an update without > >changes into a no-op? > > err isnt that one the job of the database? By

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jim Nasby
On Jun 22, 2006, at 12:56 PM, Greg Stark wrote: Just for the record, if i understood correctly -- this was all a bit black magicky -- Oracle found the data in the rollback segment by storing a pointer to it in the block header where the updated data is. Ie, it could jump straight to the right

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jim Nasby
On Jun 22, 2006, at 2:00 PM, Mark Woodward wrote: I actually have a good number of years of experience in this topic, and memcached or file system files are NOT the best solutions for a server farm. What's wrong with memcached for session data? -- Jim C. Nasby, Sr. Engineering Consultant

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jim Nasby
On Jun 22, 2006, at 1:09 PM, Tom Lane wrote: Lukas Smith <[EMAIL PROTECTED]> writes: Jochem van Dieten wrote: make the session handler smarter? And if you can't do that, put some logic in the session table that turns an update without changes into a no-op? err isnt that one the job of the

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Tom Lane
Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes: > This is what I get on a fast AMD Dual Opteron box(Running Debian > Sarge/AMD64): > 8.1.4 HEAD > 100 SELECT 1; 74,74,7377,76,77 > stats_command_string=1; 105,99,106

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Stefan Kaltenbrunner
Tom Lane wrote: > Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes: >> This is what I get on a fast AMD Dual Opteron box(Running Debian >> Sarge/AMD64): > >>8.1.4 HEAD >> 100 SELECT 1;74,74,7377,76,77 >> stats_command_string=1;

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Tom Lane
Jim Nasby <[EMAIL PROTECTED]> writes: > Question: do we currently create new index entries even if the index > key hasn't changed? Yes. > If so, what's the purpose of storing the CTID of > the next version in the old version of the row? So that UPDATE can always find the newest version of th

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Tom Lane
Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> Or do you mean that you have stats_row_level and/or stats_block_level on >> in all four cases? > yes - stats_row_level and stats_block_level on in all cases (sorry for > the confusion) - I can easily redo the tests without those

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
Andrew Dunstan <[EMAIL PROTECTED]> writes: > Bort, Paul wrote: >>> so presumably this is only needed for old Cygwin versions. Can anyone >>> say how old "1001" is and whether we still ought to care about it? >> >> IIRC, I've been on 1.5.x for at least three years. 1.0/1.1 seems to be >> around 20

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Tom Lane
Jim Nasby <[EMAIL PROTECTED]> writes: > What would be nice to add is the ability to perform that check more > easily. As of 8.1... > ... > if NEW=OLD then > ... > ERROR: operator does not exist: test = test > HINT: No operator matches the given name and argument type(s). You > may need to add

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Stefan Kaltenbrunner
Tom Lane wrote: > Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes: >> Tom Lane wrote: >>> Or do you mean that you have stats_row_level and/or stats_block_level on >>> in all four cases? > >> yes - stats_row_level and stats_block_level on in all cases (sorry for >> the confusion) - I can easily red

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Tom Lane
Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> It'd be interesting to compare 8.1 and HEAD for the no-overhead case; >> I don't think you need to redo all four cases, but I'd like to see that one. > 8.1: 50,50,49 > HEAD: 49,48,49 OK, so that seems comparable to my results

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Alvaro Herrera
Tom Lane wrote: > Charles Comiskey <[EMAIL PROTECTED]> writes: > > item #3: Carsten Wolff copyright in informix.c file > > The file informix.c contains a copyright from Carsten Wolff. Did Carsten > > directly contribute this file to the PostgreSQL project? Wow, I see what mess we would be into

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Larry Rosenman
Josh Berkus wrote: > Tom, > >> adddepend >> dbase >> dbmirror >> fulltextindex >> mSQL-interface >> mac >> oracle >> tips >> userlock > > I think you're right. I will do this before I leave town on the 30th. before anyone asks, the files I wrote in contrib/mac are free to be licensed any way th

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Andrew Dunstan
Tom Lane wrote: OK, so let's yank the file altogether and see what happens. I can make a cut at fixing the makefiles based on removing references to DLLINIT, but it might be better if someone who's in a position to test the results on Windows did the patch ... Something has br

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Alvaro Herrera
Tom Lane wrote: > Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes: > > Tom Lane wrote: > >> It'd be interesting to compare 8.1 and HEAD for the no-overhead case; > >> I don't think you need to redo all four cases, but I'd like to see that > >> one. > > > 8.1:50,50,49 > > HEAD: 49,48

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Andrew Dunstan
Andrew Dunstan wrote: Tom Lane wrote: OK, so let's yank the file altogether and see what happens. I can make a cut at fixing the makefiles based on removing references to DLLINIT, but it might be better if someone who's in a position to test the results on Windows did the patch ...

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
Andrew Dunstan <[EMAIL PROTECTED]> writes: > On REL8_1_STABLE sources commenting out the DLLINIT definition in > Makefile.cygwin works just fine. Same goes for > Win32/HEAD/Makefile.win32. I just did complete (unreported) buildfarm > run with these changes made, so I think ripping that out sho

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Andrew Dunstan
Tom Lane wrote: Meanwhile, I'd like to know how to fix the Cygwin build on HEAD. I suspect it was the MSVC "improvements" that did it. Probably. This is the commit: 2006-06-07 18:24 momjian [file list snipped] Prepare code to be built by MSVC:

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
Andrew Dunstan <[EMAIL PROTECTED]> writes: > Meanwhile, I'd like to know how to fix the Cygwin build on HEAD. I > suspect it was the MSVC "improvements" that did it. The patch to c.h certainly had no compunction about possibly changing the behavior for Cygwin: *** *** 82,94 #e

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Tom Lane
Bruce Momjian <[EMAIL PROTECTED]> writes: > Seeing stats_command_string with almost zero overhead is great news! > Should we remove that setting and just have it enabled all > the time? If you don't need it, you shouldn't have to pay any overhead for it, I think. One could make an argument now fo

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Andrew Dunstan
Tom Lane wrote: Andrew Dunstan <[EMAIL PROTECTED]> writes: Meanwhile, I'd like to know how to fix the Cygwin build on HEAD. I suspect it was the MSVC "improvements" that did it. The patch to c.h certainly had no compunction about possibly changing the behavior for Cygwin: [sni

Re: [HACKERS] Interval aggregate regression failure (expected seems

2006-06-22 Thread Michael Glaesemann
Tom Lane wrote: I've also confirmed that the problem is in interval_div; you can reproduce the failure with select '41 years 1 mon 11 days'::interval / 10; which should give '4 years 1 mon 9 days 26:24:00', but when timestamp.o is compiled with "-mcpu=pentium4 -march=pentium4", you ge

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
Andrew Dunstan <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> The patch to c.h certainly had no compunction about possibly changing >> the behavior for Cygwin: > Maybe we need to divorce Cygwin and Win32. That seems like probably an overreaction. The impression I got was that this patch had ac

Re: [HACKERS] Interval aggregate regression failure (expected seems

2006-06-22 Thread Michael Glaesemann
On Jun 23, 2006, at 9:47 , Michael Glaesemann wrote: It also changes the result of the aggregate test for intervals, but I think that's to be expected. My goodness. Of course it changes the aggregate test results. That was what brought this up in the first place. (*kicks self for not rea

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Thomas Lockhart
1) Is any John Franks code really in this file? Possibly, maybe probably. I don't remember the details (9 years is a long time!) but almost certainly any code or algorithms were specifically for the "inside" or "outside" routines. 2) Did John provide a separate license for PostgreSQL to li

Re: [HACKERS] Going for "all green" buildfarm results

2006-06-22 Thread Tom Lane
Andrew Dunstan <[EMAIL PROTECTED]> writes: > Mark Wong wrote: >> Flex is 2.5.33 on both systems. I'm assuming that's too modern so >> I'll go ahead and stop building 7.3 for those systems. > You could be lucky the others build. I believe our supported version is > still 2.5.4, which is what all

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Christopher Kings-Lynne
The example is a very active web site, the flow is this: query for session information process HTTP request update session information This happens for EVERY http request. Chances are that you won't have concurrent requests for the same row, but you may have well over 100 HTTP server processes/t

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
Thomas Lockhart writes: >> This code seems to have been inserted by Tom Lockhart on 1997-07-29 >> (geo_ops.c rev 1.13). Tom, any info on the copyright status? > None, beyond the info you already resurrected. I vaguely recall that I > did take the LJ letter as an invitation to reuse algorithms.

Re: [HACKERS] Overhead for stats_command_string et al, take 2

2006-06-22 Thread Mark Kirkwood
Tom Lane wrote: Stefan Kaltenbrunner <[EMAIL PROTECTED]> writes: Tom Lane wrote: It'd be interesting to compare 8.1 and HEAD for the no-overhead case; I don't think you need to redo all four cases, but I'd like to see that one. 8.1:50,50,49 HEAD: 49,48,49 OK, so that seems comparable

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Agent M
On Jun 22, 2006, at 9:56 PM, Christopher Kings-Lynne wrote: The example is a very active web site, the flow is this: query for session information process HTTP request update session information This happens for EVERY http request. Chances are that you won't have concurrent requests for the sam

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Steve Atkins
On Jun 22, 2006, at 6:56 PM, Agent M wrote: On Jun 22, 2006, at 9:56 PM, Christopher Kings-Lynne wrote: The example is a very active web site, the flow is this: query for session information process HTTP request update session information This happens for EVERY http request. Chances are that

[HACKERS] Small overhead run time memory trace (Was Re: shall we have a TRACE_MEMORY mode)

2006-06-22 Thread Qingqing Zhou
"Tom Lane" <[EMAIL PROTECTED]> wrote > > One idea that comes to mind is to have a compile time option to record > the palloc __FILE__ and _LINE__ in every AllocChunk header. Then it > would not be so hard to identify the culprit while trawling through > memory. The overhead costs would be so hig

Row comparison for tables (was Re: [HACKERS] vacuum, performance, and MVCC)

2006-06-22 Thread Tom Lane
I wrote: > Jim Nasby <[EMAIL PROTECTED]> writes: >> What would be nice to add is the ability to perform that check more >> easily. As of 8.1... >> ... >> if NEW=OLD then >> ... >> ERROR: operator does not exist: test = test >> HINT: No operator matches the given name and argument type(s). You

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Gavin Sherry
On Thu, 22 Jun 2006, Agent M wrote: > > On Jun 22, 2006, at 9:56 PM, Christopher Kings-Lynne wrote: > > >> The example is a very active web site, the flow is this: > >> query for session information > >> process HTTP request > >> update session information > >> This happens for EVERY http request.

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Christopher Browne
The world rejoiced as [EMAIL PROTECTED] (Tom Lane) wrote: > Lukas Smith <[EMAIL PROTECTED]> writes: >> Jochem van Dieten wrote: >>> make the session handler smarter? And if you can't do that, put some >>> logic in the session table that turns an update without changes into a >>> no-op? > >> err isn

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jonah H. Harris
On 6/22/06, Gavin Sherry <[EMAIL PROTECTED]> wrote: There seems to be a bit of confusion about what MVCC is. Yes, PostgreSQL implements MVTO-style concurrency, Oracle implements MVRC, ... Let's not go into theory here, because there's plenty of papers and books on the subject. The other MVCC

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Gavin Sherry
On Thu, 22 Jun 2006, Jonah H. Harris wrote: > Not in all systems. A few now perform in-memory UNDO and only write > it to disk if and when it is required. Interesting... > > > Overwriting MVCC comes with its own baggage. Ask any Oracle user about > > error ORA-01555[1]. There's also the added c

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Alvaro Herrera
Jonah H. Harris wrote: > On 6/22/06, Gavin Sherry <[EMAIL PROTECTED]> wrote: > >When an update occurs, the existing row version is > >copied to te UNDO file > > Not in all systems. A few now perform in-memory UNDO and only write > it to disk if and when it is required. How does that work? If t

[HACKERS] Full Disjunction

2006-06-22 Thread Christopher Kings-Lynne
What IS this full disjunction business? User Pgstudy wrote: Log Message: --- Some more deformed tuple structures consolidation works. Now even faster but still half way done. Modified Files: -- fd: algutils.c (r1.9 -> r1.10) (http://cvs.pgfoundry.org/cg

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Heikki Linnakangas
On Thu, 22 Jun 2006, Jim Nasby wrote: DB2 switched to MVCC in version 8. ... Um, no it didn't. - Heikki ---(end of broadcast)--- TIP 6: explain analyze is your friend

Re: [HACKERS] postmaster.exe vs postgres.exe (was: CVS HEAD busted on

2006-06-22 Thread Bruce Momjian
Tom Lane wrote: > "Dave Page" writes: > >>> though - Magnus & > >>> I were wondering if Peter's change means we no longer need to ship > >>> postmaster.exe and postgres.exe with pgInstaller. Presumably > >>> we can just use postgres.exe for everything now? > > >> Won't we still need to know if w

Re: [HACKERS] CVS HEAD busted on Windows?

2006-06-22 Thread Dave Page
> -Original Message- > From: Tom Lane [mailto:[EMAIL PROTECTED] > Sent: 21 June 2006 20:43 > To: Dave Page > Cc: Andrew Dunstan; Peter Eisentraut; pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] CVS HEAD busted on Windows? > > "Dave Page" writes: > > Yeah - I've been seeing bot

Re: [HACKERS] checking on buildfarm member thrush

2006-06-22 Thread Gaetano Mendola
Tom Lane wrote: > I'm trying to determine why thrush has been failing on PG CVS HEAD for > the past few days. Could you try running the attached program on that > machine, and see what it prints? I suspect it will dump core :-( > > Note: you might need to use -D_GNU_SOURCE to get it to compile a

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Mark Woodward
> Clinging to sanity, [EMAIL PROTECTED] ("Mark Woodward") mumbled into > her beard: >> We all know that PostgreSQL suffers performance problems when rows are >> updated frequently prior to a vacuum. The most serious example can be >> seen >> by using PostgreSQL as a session handler for a busy we si

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Zeugswetter Andreas DCP SD
> >> Each time the record is updated, a new version is created, thus > >> lengthening the "correct" version search each time row is accessed, > >> until, of course, the next vacuum comes along and corrects the index > >> to point to the latest version of the record. > >> > >> Is that a fair exp

[HACKERS] xlog viewer proposal

2006-06-22 Thread Diogo Biazus
I'm developing the summer of code project to create a xlog viewer.The tool we want to create is a DBA tool used for inspect the xlog files, looking for some operations, statistcs of database usage and status of transactions. Some use cases:* Some user made a mistake and commited it to the database.

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Christopher Browne
After a long battle with technology, [EMAIL PROTECTED] ("Mark Woodward"), an earthling, wrote: >> Clinging to sanity, [EMAIL PROTECTED] ("Mark Woodward") mumbled into >> her beard: >>> We all know that PostgreSQL suffers performance problems when rows are >>> updated frequently prior to a vacuum.

Re: [HACKERS] CVS HEAD busted on Windows?

2006-06-22 Thread Andrew Dunstan
Dave Page wrote: As a sidenote on the postgres/postmaster merge subject though - Magnus & I were wondering if Peter's change means we no longer need to ship postmaster.exe and postgres.exe with pgInstaller. Presumably we can just use postgres.exe for everything now? Won't we still need

Re: [HACKERS] postmaster.exe vs postgres.exe (was: CVS HEAD busted on Windows?)

2006-06-22 Thread Dave Page
> -Original Message- > From: Andrew Dunstan [mailto:[EMAIL PROTECTED] > Sent: 22 June 2006 14:06 > To: Dave Page > Cc: Tom Lane; Peter Eisentraut; pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] CVS HEAD busted on Windows? > > > > Dave Page wrote: > > > > >As a sidenote on th

Re: [HACKERS] [PERFORM] Sun Donated a Sun Fire T2000 to the PostgreSQL

2006-06-22 Thread David Roussel
Arjen van der Meijden wrote: Here is a graph of our performance measured on PostgreSQL: http://achelois.tweakers.net/~acm/pgsql-t2000/T2000-schaling-postgresql.png ... The "perfect" line is based on the "Max" value for 1 core and then just multiplied by the amount of cores to have

Re: [HACKERS] postmaster.exe vs postgres.exe

2006-06-22 Thread Andrew Dunstan
Dave Page wrote: Won't we still need to know if we are called as postmaster or postgres? Unless the 'postmaster' instance starts all it's sub processes with an additional option to tell them they're children (I haven't looked at the code yet so I dunno if this is how it's done). For th

Re: [HACKERS] postmaster.exe vs postgres.exe (was: CVS HEAD busted on Windows?)

2006-06-22 Thread Hiroshi Saito
> Dave Page wrote: > > Won't we still need to know if we are called as postmaster or > > postgres? > > Unless the 'postmaster' instance starts all it's sub processes with an > additional option to tell them they're children (I haven't looked at the > code yet so I dunno if this is how it's done).

Re: [HACKERS] postmaster.exe vs postgres.exe

2006-06-22 Thread Dave Page
> -Original Message- > From: Andrew Dunstan [mailto:[EMAIL PROTECTED] > Sent: 22 June 2006 14:26 > To: Dave Page > Cc: Tom Lane; Peter Eisentraut; pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] postmaster.exe vs postgres.exe > > > Windows children could be handled, I think, but

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Mark Woodward
> After a long battle with technology, [EMAIL PROTECTED] ("Mark > Woodward"), an earthling, wrote: >>> Clinging to sanity, [EMAIL PROTECTED] ("Mark Woodward") mumbled into >>> her beard: [snip] >>> >>> 1. The index points to all the versions, until they get vacuumed out. >> >> It can't point to "a

Re: [HACKERS] checking on buildfarm member thrush

2006-06-22 Thread Tom Lane
Gaetano Mendola <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> I'm trying to determine why thrush has been failing on PG CVS HEAD for >> the past few days. Could you try running the attached program on that >> machine, and see what it prints? I suspect it will dump core :-( > $ gcc -D_GNU_SOUR

Re: [HACKERS] postmaster.exe vs postgres.exe (was: CVS HEAD busted on Windows?)

2006-06-22 Thread Tom Lane
"Dave Page" writes: >>> though - Magnus & >>> I were wondering if Peter's change means we no longer need to ship >>> postmaster.exe and postgres.exe with pgInstaller. Presumably >>> we can just use postgres.exe for everything now? >> Won't we still need to know if we are called as postmaster or

Re: [HACKERS] postmaster.exe vs postgres.exe (was: CVS HEAD busted

2006-06-22 Thread Andrew Dunstan
Tom Lane wrote: Won't we still need to know if we are called as postmaster or postgres? No. The entire point of the recent changes is that the behavior no longer depends on the name of the executable, only on the switches. Oh. My mistake. That sounds good. cheers andrew ---

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Alvaro Herrera
Mark Woodward wrote: > Hmm, OK, then the problem is more serious than I suspected. > This means that every index on a row has to be updated on every > transaction that modifies that row. Is that correct? Add an index entry, yes. > I am attaching some code that shows the problem with regard to >

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jonah H. Harris
On 6/22/06, Alvaro Herrera <[EMAIL PROTECTED]> wrote: > Hmm, OK, then the problem is more serious than I suspected. > This means that every index on a row has to be updated on every > transaction that modifies that row. Is that correct? Add an index entry, yes. Again, this is a case for update

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Csaba Nagy
> > [...] > > There has to be a more linear way of handling this scenario. > > So vacuum the table often. Good advice, except if the table is huge :-) Here we have for example some tables which are frequently updated but contain >100 million rows. Vacuuming that takes hours. And the dead row can

Re: [HACKERS] Problem to "current-status information in shared memory" patch

2006-06-22 Thread Tom Lane
"Qingqing Zhou" <[EMAIL PROTECTED]> writes: > "Tom Lane" <[EMAIL PROTECTED]> wrote >> "postgres --single" works for me. Maybe you need a "make >> distclean"/rebuild? > Sorry, because I forget to say that you need to turn "stats_command_string" > on. Ah. OK, fixed. regar

Re: [HACKERS] [PERFORM] Sun Donated a Sun Fire T2000 to the PostgreSQL

2006-06-22 Thread Arjen van der Meijden
On 22-6-2006 15:03, David Roussel wrote: Sureky the 'perfect' line ought to be linear? If the performance was perfectly linear, then the 'pages generated' ought to be G times the number (virtual) processors, where G is the gradient of the graph. In such a case the graph will go through the or

Re: [HACKERS] shall we have a TRACE_MEMORY mode

2006-06-22 Thread korry
> As I follow Relyea Mike's recent post of possible memory leak, I think that > we are lack of a good way of identifing memory usage. Maybe we should also > remember __FILE__, __LINE__ etc for better memory usage diagnose when > TRACE_MEMORY is on? I find __FILE__ and __LINE__ very helpful

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Mario Weilguni
Am Donnerstag, 22. Juni 2006 16:09 schrieb Csaba Nagy: > > > [...] > > > There has to be a more linear way of handling this scenario. > > > > So vacuum the table often. > > Good advice, except if the table is huge :-) > > Here we have for example some tables which are frequently updated but > conta

Re: [HACKERS] [CORE] GPL Source and Copyright Questions

2006-06-22 Thread Tom Lane
[ redirecting to -hackers, as I see no need for this to be a core issue ] Charles Comiskey <[EMAIL PROTECTED]> writes: > Hello, > I've recently looked through the PostgreSQL code and a couple of questions > surfaced. I was hoping someone here may be able to answer them. Two have > links to pos

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Hannu Krosing
Ühel kenal päeval, N, 2006-06-22 kell 09:59, kirjutas Mark Woodward: > > After a long battle with technology, [EMAIL PROTECTED] ("Mark > > Woodward"), an earthling, wrote: > >>> Clinging to sanity, [EMAIL PROTECTED] ("Mark Woodward") mumbled into > > It pointed to *ALL* the versions. > > Hmm, OK,

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Hannu Krosing
Ühel kenal päeval, N, 2006-06-22 kell 10:20, kirjutas Jonah H. Harris: > On 6/22/06, Alvaro Herrera <[EMAIL PROTECTED]> wrote: > > > Hmm, OK, then the problem is more serious than I suspected. > > > This means that every index on a row has to be updated on every > > > transaction that modifies that

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Mark Woodward
> Ühel kenal päeval, N, 2006-06-22 kell 09:59, kirjutas Mark Woodward: >> > After a long battle with technology, [EMAIL PROTECTED] ("Mark >> > Woodward"), an earthling, wrote: >> >>> Clinging to sanity, [EMAIL PROTECTED] ("Mark Woodward") mumbled >> into >> > It pointed to *ALL* the versions. >>

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jonah H. Harris
On 6/22/06, Hannu Krosing <[EMAIL PROTECTED]> wrote: I guess that MySQL on its original storage does that, but they allow only one concurrent update per table and no transactions. More like practically every commercial database. As ~97% of transactions commit (yes, some can argue that number),

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Mark Woodward
> Ühel kenal päeval, N, 2006-06-22 kell 10:20, kirjutas Jonah H. Harris: >> On 6/22/06, Alvaro Herrera <[EMAIL PROTECTED]> wrote: >> > > Hmm, OK, then the problem is more serious than I suspected. >> > > This means that every index on a row has to be updated on every >> > > transaction that modif

Re: [HACKERS] xlog viewer proposal

2006-06-22 Thread Tom Lane
"Diogo Biazus" <[EMAIL PROTECTED]> writes: > The idea I've been discussing with Simon Riggs is to create a set of > functions that can be called from within the database. I'd question that at the very start. I don't see any strong reason to do it that way, and as you admit further down it'd make

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Tom Lane
Christopher Browne <[EMAIL PROTECTED]> writes: > After a long battle with technology, [EMAIL PROTECTED] ("Mark Woodward"), an > earthling, wrote: >> Not true. Oracle does not seem to exhibit this problem. > Oracle suffers a problem in this regard that PostgreSQL doesn't; in > Oracle, rollbacks ar

Re: [HACKERS] Going for "all green" buildfarm results

2006-06-22 Thread Andrew Dunstan
Mark Wong wrote: Now why are we failing on 7.3? What version of flex do you have? If it's too modern we'll just need to take 7.3 out of the cobra and stoat rotations - we'd really only make supercritical fixes on that branch these days. Flex is 2.5.33 on both systems. I'm assuming that

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jochem van Dieten
On 6/22/06, Mark Woodward wrote: (..) thousand active sessions (..) If an active user causes a session update once a second (..) Generally speaking, sessions aren't updated when they change, they are usually updated per HTTP request. The data in a session may not change, but the session hand

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Lukas Smith
Tom Lane wrote: Basically there's no free lunch: if you want the benefits of MVCC it's going to cost you somewhere. In the Postgres design you pay by having to do VACUUM pretty often for heavily-updated tables. I don't think that decision is fundamentally wrong --- the attractive thing about i

Re: [HACKERS] xlog viewer proposal

2006-06-22 Thread Jonah H. Harris
On 6/22/06, Tom Lane <[EMAIL PROTECTED]> wrote: it'd make it impossible to use the viewer to work on extracting data from a failed cluster; which is, at least in my mind, one of the primary use-cases for the thing. While I too see this as something which could be used for this outside the datab

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Lukas Smith
Jochem van Dieten wrote: make the session handler smarter? And if you can't do that, put some logic in the session table that turns an update without changes into a no-op? err isnt that one the job of the database? regards, Lukas ---(end of broadcast)-

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Chris Browne
[EMAIL PROTECTED] (Csaba Nagy) writes: >> > [...] >> > There has to be a more linear way of handling this scenario. >> >> So vacuum the table often. > > Good advice, except if the table is huge :-) ... Then the table shouldn't be designed to be huge. That represents a design error. > Here we h

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Jonah H. Harris
On 6/22/06, Tom Lane <[EMAIL PROTECTED]> wrote: The Oracle design has got other drawbacks: if you need to access a row version other than than the very latest, you need to go searching in the rollback segments for it. There are ways to implement this functionality without implementing it exactl

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread D'Arcy J.M. Cain
On Thu, 22 Jun 2006 19:01:38 +0200 Lukas Smith <[EMAIL PROTECTED]> wrote: > Jochem van Dieten wrote: > > > make the session handler smarter? And if you can't do that, put some > > logic in the session table that turns an update without changes into a > > no-op? > > err isnt that one the job of t

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Rod Taylor
> > Here we have for example some tables which are frequently updated but > > contain >100 million rows. Vacuuming that takes hours. And the dead row > > candidates are the ones which are updated again and again and looked up > > frequently... > > This demonstrates that "archival" material and "ac

Re: [HACKERS] xlog viewer proposal

2006-06-22 Thread Tom Lane
"Jonah H. Harris" <[EMAIL PROTECTED]> writes: > I think it should certainly be able to run on it's own, but it > wouldn't be that hard to extend the functions so that they were usable > from within the database or vice-versa. Yes it would. The most obvious point is that memory management and erro

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Mark Woodward
> Christopher Browne <[EMAIL PROTECTED]> writes: >> After a long battle with technology, [EMAIL PROTECTED] ("Mark >> Woodward"), an earthling, wrote: >>> Not true. Oracle does not seem to exhibit this problem. > >> Oracle suffers a problem in this regard that PostgreSQL doesn't; in >> Oracle, rollb

Re: [HACKERS] vacuum, performance, and MVCC

2006-06-22 Thread Rod Taylor
> > You mean systems that are designed so exactly, that they can't take 10% > > performance change ? > > No, that's not really the point, performance degrades over time, in one > minute it degraded 10%. > > The update to session ratio has a HUGE impact on PostgreSQL. If you have a > thousand acti

Re: [HACKERS] xlog viewer proposal

2006-06-22 Thread Jonah H. Harris
On 6/22/06, Tom Lane <[EMAIL PROTECTED]> wrote: Yes it would. The most obvious point is that memory management and error handling conventions inside the backend are quite different from what you'd expect to employ in a standalone program. No, this wouldn't really be that hard, especially if he

  1   2   >