Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-22 Thread Kenneth Marshall
On Sun, Jun 18, 2006 at 05:26:16PM -0400, Tom Lane wrote: I wrote: PFC [EMAIL PROTECTED] writes: So, the proposal : On executing a command, Backend stores the command string, then overwrites the counter with (counter + 1) and with the timestamp of command start. Periodically, like

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-22 Thread Kenneth Marshall
On Sun, Jun 18, 2006 at 11:07:41PM -0400, Tom Lane wrote: Bort, Paul [EMAIL PROTECTED] writes: Anyone know a variant of this that really works? Here's a theory: If the counter is bumped to an odd number before modification, and an even number after it's done, then the reader will know

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-19 Thread PFC
Great minds think alike ;-) ... I just committed exactly that protocol. I believe it is correct, because AFAICS there are only four possible risk cases: Congrats ! For general culture you might be interested in reading this :

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-19 Thread Magnus Hagander
Might it not be a win to also store per backend global values in the shared memory segment? Things like time of last command, number of transactions executed in this backend, backend start time and other values that are fixed-size? I'm including backend start time, command

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-19 Thread Bort, Paul
* reader's read starts before and ends after writer's update: reader will certainly note a change in update counter. * reader's read starts before and ends within writer's update: reader will note a change in update counter. * reader's read starts within and ends after writer's update:

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Greg Stark
Douglas McNaught [EMAIL PROTECTED] writes: Yeah, but if you turn on query logging in that case you'll see the bajillions of short queries, so you don't need the accurate snapshot to diagnose that. Query logging on a production OLTP machine? a) that would be a huge performance drain on the

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Douglas McNaught
Greg Stark [EMAIL PROTECTED] writes: Douglas McNaught [EMAIL PROTECTED] writes: Yeah, but if you turn on query logging in that case you'll see the bajillions of short queries, so you don't need the accurate snapshot to diagnose that. Query logging on a production OLTP machine? a) that

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Greg Stark
Douglas McNaught [EMAIL PROTECTED] writes: (a) and (b): of course you would only do it on a temporary basis for problem diagnosis. Temporary or not it isn't really an option when you're dealing with high volumes. You could imagine a setup where say, 1% of page requests randomly turn on

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Magnus Hagander
The existing stats collection mechanism seems OK for event counts, although I'd propose two changes: one, get rid of the separate buffer process, and two, find a way to emit event reports in a time-driven way rather than once per transaction commit. I'm a bit vague about how to do the

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Tom Lane
Magnus Hagander [EMAIL PROTECTED] writes: Might it not be a win to also store per backend global values in the shared memory segment? Things like time of last command, number of transactions executed in this backend, backend start time and other values that are fixed-size? I'm including

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Hannu Krosing
Ühel kenal päeval, P, 2006-06-18 kell 15:09, kirjutas Tom Lane: Magnus Hagander [EMAIL PROTECTED] writes: Might it not be a win to also store per backend global values in the shared memory segment? Things like time of last command, number of transactions executed in this backend, backend

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Tom Lane
I wrote: PFC [EMAIL PROTECTED] writes: So, the proposal : On executing a command, Backend stores the command string, then overwrites the counter with (counter + 1) and with the timestamp of command start. Periodically, like every N seconds, a separate process reads the counter, then

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Bort, Paul
BTW, I think the writer would actually need to bump the counter twice, once before and once after it modifies its stats area. Else there's no way to detect that you've copied a partially-updated stats entry. Actually, neither of these ideas works: it's possible that the reader

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-18 Thread Tom Lane
Bort, Paul [EMAIL PROTECTED] writes: Anyone know a variant of this that really works? Here's a theory: If the counter is bumped to an odd number before modification, and an even number after it's done, then the reader will know it needs to re-read if the counter is an odd number. Great

[HACKERS] Rethinking stats communication mechanisms

2006-06-17 Thread Tom Lane
In view of my oprofile results http://archives.postgresql.org/pgsql-hackers/2006-06/msg00859.php I'm thinking we need some major surgery on the way that the stats collection mechanism works. It strikes me that we are using a single communication mechanism to handle what are really two distinct

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-17 Thread PFC
It strikes me that we are using a single communication mechanism to handle what are really two distinct kinds of data: Interesting. I recently read a paper on how to get rid of locks for this kind of pattern. * For the Command String - Problem : need to display the currently

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-17 Thread Tom Lane
PFC [EMAIL PROTECTED] writes: So, the proposal : On executing a command, Backend stores the command string, then overwrites the counter with (counter + 1) and with the timestamp of command start. Periodically, like every N seconds, a separate process reads the counter,

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-17 Thread Greg Stark
PFC [EMAIL PROTECTED] writes: - Will only be of use if the command is taking a long, long time. So, it need not be realtime ; no problem if the data comes with a little delay, or not at all if the command executes quickly. I would dispute this point. Picture a system running a

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-17 Thread Douglas McNaught
Greg Stark [EMAIL PROTECTED] writes: PFC [EMAIL PROTECTED] writes: - Will only be of use if the command is taking a long, long time. So, it need not be realtime ; no problem if the data comes with a little delay, or not at all if the command executes quickly. I would dispute