Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-21 Thread Kenneth Marshall
On Sun, Jun 18, 2006 at 11:07:41PM -0400, Tom Lane wrote: > "Bort, Paul" <[EMAIL PROTECTED]> writes: > >> Anyone know a variant of this that really works? > > > Here's a theory: If the counter is bumped to an odd number before > > modification, and an even number after it's done, then the reader w

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-21 Thread Kenneth Marshall
On Sun, Jun 18, 2006 at 05:26:16PM -0400, Tom Lane wrote: > I wrote: > > PFC <[EMAIL PROTECTED]> writes: > >> So, the proposal : > >> On executing a command, Backend stores the command string, then > >> overwrites the counter with (counter + 1) and with the timestamp of > >> command start. > >>

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-19 Thread Bort, Paul
> > * reader's read starts before and ends after writer's update: reader > will certainly note a change in update counter. > > * reader's read starts before and ends within writer's update: reader > will note a change in update counter. > > * reader's read starts within and ends after writer's u

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-19 Thread Magnus Hagander
> > Might it not be a win to also store "per backend global > values" in the > > shared memory segment? Things like "time of last command", > "number of > > transactions executed in this backend", "backend start > time" and other > > values that are fixed-size? > > I'm including backend star

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-19 Thread PFC
Great minds think alike ;-) ... I just committed exactly that protocol. I believe it is correct, because AFAICS there are only four possible risk cases: Congrats ! For general culture you might be interested in reading this : http://en.wikipedia.org/wiki/Software_tran

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Tom Lane
"Bort, Paul" <[EMAIL PROTECTED]> writes: >> Anyone know a variant of this that really works? > Here's a theory: If the counter is bumped to an odd number before > modification, and an even number after it's done, then the reader will > know it needs to re-read if the counter is an odd number. Gr

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Bort, Paul
> > > BTW, I think the writer would actually need to bump the > counter twice, > > once before and once after it modifies its stats area. > Else there's > > no way to detect that you've copied a partially-updated stats entry. > > Actually, neither of these ideas works: it's possible that >

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Tom Lane
I wrote: > PFC <[EMAIL PROTECTED]> writes: >> So, the proposal : >> On executing a command, Backend stores the command string, then >> overwrites the counter with (counter + 1) and with the timestamp of >> command start. >> Periodically, like every N seconds, a separate process reads the counte

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Hannu Krosing
Ühel kenal päeval, P, 2006-06-18 kell 15:09, kirjutas Tom Lane: > "Magnus Hagander" <[EMAIL PROTECTED]> writes: > > Might it not be a win to also store "per backend global values" in the > > shared memory segment? Things like "time of last command", "number of > > transactions executed in this back

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Tom Lane
"Magnus Hagander" <[EMAIL PROTECTED]> writes: > Might it not be a win to also store "per backend global values" in the > shared memory segment? Things like "time of last command", "number of > transactions executed in this backend", "backend start time" and other > values that are fixed-size? I'm

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Magnus Hagander
> The existing stats collection mechanism seems OK for event > counts, although I'd propose two changes: one, get rid of the > separate buffer process, and two, find a way to emit event > reports in a time-driven way rather than once per transaction > commit. I'm a bit vague about how to do th

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Greg Stark
Douglas McNaught <[EMAIL PROTECTED]> writes: > (a) and (b): of course you would only do it on a temporary basis for > problem diagnosis. Temporary or not it isn't really an option when you're dealing with high volumes. You could imagine a setup where say, 1% of page requests randomly turn on deb

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Douglas McNaught
Greg Stark <[EMAIL PROTECTED]> writes: > Douglas McNaught <[EMAIL PROTECTED]> writes: > >> Yeah, but if you turn on query logging in that case you'll see the >> bajillions of short queries, so you don't need the accurate snapshot >> to diagnose that. > > Query logging on a production OLTP machine?

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Greg Stark
Douglas McNaught <[EMAIL PROTECTED]> writes: > Yeah, but if you turn on query logging in that case you'll see the > bajillions of short queries, so you don't need the accurate snapshot > to diagnose that. Query logging on a production OLTP machine? a) that would be a huge performance drain on th

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-17 Thread Douglas McNaught
Greg Stark <[EMAIL PROTECTED]> writes: > PFC <[EMAIL PROTECTED]> writes: > >> - Will only be of use if the command is taking a long, long time. >> So, it need not be realtime ; no problem if the data comes with a >> little delay, or not at all if the command executes quickly. > > I woul

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-17 Thread Greg Stark
PFC <[EMAIL PROTECTED]> writes: > - Will only be of use if the command is taking a long, long time. > So, it need not be realtime ; no problem if the data comes with a > little delay, or not at all if the command executes quickly. I would dispute this point. Picture a system running

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-17 Thread Tom Lane
PFC <[EMAIL PROTECTED]> writes: > So, the proposal : > On executing a command, Backend stores the command string, then > overwrites the counter with (counter + 1) and with the timestamp of > command start. > Periodically, like every N seconds, a separate process reads the > c

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-17 Thread PFC
It strikes me that we are using a single communication mechanism to handle what are really two distinct kinds of data: Interesting. I recently read a paper on how to get rid of locks for this kind of pattern. * For the Command String - Problem : need to display the currently ex

[HACKERS] Rethinking stats communication mechanisms

2006-06-17 Thread Tom Lane
In view of my oprofile results http://archives.postgresql.org/pgsql-hackers/2006-06/msg00859.php I'm thinking we need some major surgery on the way that the stats collection mechanism works. It strikes me that we are using a single communication mechanism to handle what are really two distinct kin