Bruce Momjian wrote:
I have talked to Jan over the phone, and he has convinced me that UDP is
the proper way to communicate stats to the collector, rather than my
shared memory idea.
The advantages of his UDP approach is that the collector can sleep on
the UDP socket rather than having the
I have talked to Jan over the phone, and he has convinced me that UDP is
the proper way to communicate stats to the collector, rather than my
shared memory idea.
The advantages of his UDP approach is that the collector can sleep on
the UDP socket rather than having the collector poll the shared
Bruce Momjian [EMAIL PROTECTED] writes:
Only shared memory gives us near-zero cost for write/read. 99% of
backends will not be using stats, so it has to be cheap.
Not with a circular buffer it's not cheap, because you need interlocking
on writes. Your claim that you can get away without that
Bruce Momjian [EMAIL PROTECTED] writes:
Only shared memory gives us near-zero cost for write/read. 99% of
backends will not be using stats, so it has to be cheap.
Not with a circular buffer it's not cheap, because you need interlocking
on writes. Your claim that you can get away
I have a new statistics collection proposal.
I suggest three shared memory areas:
One per backend to hold the query string and other per-backend stats
One global area to hold accumulated stats for all backends
One global circular buffer to hold per-table/object stats
Bruce Momjian wrote:
Bruce Momjian [EMAIL PROTECTED] writes:
Only shared memory gives us near-zero cost for write/read. 99% of
backends will not be using stats, so it has to be cheap.
Not with a circular buffer it's not cheap, because you need interlocking
on writes. Your claim
I figured it could just wake up every few seconds and check. It will
remember the loop counter and current pointer, and read any new
information. I was thinking of a 20k buffer, which could cover about 4k
events.
Here I wonder what your EVENT is. With an Oid as identifier
On Fri, Mar 16, 2001 at 05:25:24PM -0500, Jan Wieck wrote:
Jan Wieck wrote:
...
Just to get some evidence at hand - could some owners of
different platforms compile and run the attached little C
source please?
...
Seems Tom is (unfortunately) right. The pipe blocks
Jan Wieck [EMAIL PROTECTED] writes:
Just to get some evidence at hand - could some owners of
different platforms compile and run the attached little C
source please?
(The program tests how much data can be stuffed into a pipe
or a Sys-V message queue before the
At 13:49 16/03/01 -0500, Jan Wieck wrote:
Similar problem as with shared memory - size. If a long
running backend of a multithousand table database needs to
send access stats per table - and had accessed them all up to
now - it'll be alot of wasted bandwidth.
Not if you
Philip Warner wrote:
At 13:49 16/03/01 -0500, Jan Wieck wrote:
Similar problem as with shared memory - size. If a long
running backend of a multithousand table database needs to
send access stats per table - and had accessed them all up to
now - it'll be alot of
On Sat, Mar 17, 2001 at 09:33:03AM -0500, Jan Wieck wrote:
The general problem remains. We only have one central
collector with a limited receive capacity. The more load is
on the machine, the smaller it's capacity gets. The more
complex the DB schemas get
Samuel Sieb [EMAIL PROTECTED] writes:
Just as another suggestion, what about sending the data to a different
computer, so instead of tying up the database server with processing the
statistics, you have another computer that has some free time to do the
processing.
Some drawbacks are that
... and a lot more load on the CPU. Same-machine "network" connections
are much cheaper (on most kernels, anyway) than real network
connections.
I think all of this discussion is vast overkill. No one has yet
demonstrated that it's not sufficient to have *one* collector process
and a
Bruce Momjian [EMAIL PROTECTED] writes:
The only open issue is per-table stuff, and I would like to see some
circular buffer implemented to handle that, with a collection process
that has access to shared memory.
That will get us into locking/contention issues. OTOH, frequent trips
to the
Bruce Momjian [EMAIL PROTECTED] writes:
The only open issue is per-table stuff, and I would like to see some
circular buffer implemented to handle that, with a collection process
that has access to shared memory.
That will get us into locking/contention issues. OTOH, frequent trips
to
Bruce Momjian [EMAIL PROTECTED] writes:
Even better, have an SQL table updated with the per-table stats
periodically.
That will be horribly expensive, if it's a real table.
But per-table stats aren't something that people will look at often,
right? They can sit in the collector's memory
But per-table stats aren't something that people will look at often,
right? They can sit in the collector's memory for quite a while. See
people wanting to look at per-backend stuff frequently, and that is why
I thought share memory should be good, and a global area for aggregate
Tom Lane wrote:
Samuel Sieb [EMAIL PROTECTED] writes:
Just as another suggestion, what about sending the data to a different
computer, so instead of tying up the database server with processing the
statistics, you have another computer that has some free time to do the
processing.
Jan Wieck [EMAIL PROTECTED] writes:
Uh - not much time to spend if the statistics should at least
be half accurate. And it would become worse in SMP systems.
So that was a nifty idea, but I think it'd cause much more
statistic losses than I assumed at first.
Back to
At 17:10 15/03/01 -0800, Alfred Perlstein wrote:
Which is why the backends should not do anything other than maintain the
raw data. If there is atomic data than can cause inconsistency, then a
dropped UDP packet will do the same.
The UDP packet (a COPY) can contain a consistant snapshot of
Alfred Perlstein wrote:
* Jan Wieck [EMAIL PROTECTED] [010316 08:08] wrote:
Philip Warner wrote:
But I prefer the UDP/Collector model anyway; it gives use greater
flexibility + the ability to keep stats past backend termination, and,as
you say, removes any possible locking
Tom Lane wrote:
Jan Wieck [EMAIL PROTECTED] writes:
Uh - not much time to spend if the statistics should at least
be half accurate. And it would become worse in SMP systems.
So that was a nifty idea, but I think it'd cause much more
statistic losses than I assumed
Jan Wieck [EMAIL PROTECTED] writes:
Does a pipe guarantee that a buffer, written with one atomic
write(2), never can get intermixed with other data on the
readers end?
Yes. The HPUX man page for write(2) sez:
o Write requests of {PIPE_BUF} bytes or less will not
Tom Lane wrote:
Jan Wieck [EMAIL PROTECTED] writes:
Does a pipe guarantee that a buffer, written with one atomic
write(2), never can get intermixed with other data on the
readers end?
Yes. The HPUX man page for write(2) sez:
o Write requests of {PIPE_BUF}
Tom Lane wrote:
Now this would put a pretty tight time constraint on the collector:
fall more than 4K behind, you start losing data. I am not sure if
a UDP socket would provide more buffering or not; anyone know?
Looks like Linux has something around 16-32K of buffer space
for UDP
Jan Wieck wrote:
Tom Lane wrote:
Now this would put a pretty tight time constraint on the collector:
fall more than 4K behind, you start losing data. I am not sure if
a UDP socket would provide more buffering or not; anyone know?
Looks like Linux has something around 16-32K of
Jan Wieck [EMAIL PROTECTED] writes:
Just to get some evidence at hand - could some owners of
different platforms compile and run the attached little C
source please?
HPUX 10.20:
Pipe buffer is 8192 bytes
Sys-V message queue buffer is 16384 bytes
Just to get some evidence at hand - could some owners of
different platforms compile and run the attached little C
source please?
$ uname -srm
FreeBSD 4.1.1-STABLE
$ ./jan
Pipe buffer is 16384 bytes
Sys-V message queue buffer is 2048 bytes
$ uname -srm
NetBSD 1.5 alpha
* Jan Wieck [EMAIL PROTECTED] [010316 16:35]:
Jan Wieck wrote:
Tom Lane wrote:
Now this would put a pretty tight time constraint on the collector:
fall more than 4K behind, you start losing data. I am not sure if
a UDP socket would provide more buffering or not; anyone know?
* Larry Rosenman [EMAIL PROTECTED] [010316 20:47]:
* Jan Wieck [EMAIL PROTECTED] [010316 16:35]:
$ ./queuetest
Pipe buffer is 32768 bytes
Sys-V message queue buffer is 4096 bytes
$ uname -a
UnixWare lerami 5 7.1.1 i386 x86at SCO UNIX_SVR5
$
I think some of these are configurable...
Jan Wieck [EMAIL PROTECTED] writes:
What about a collector deamon, fired up by the postmaster and
receiving UDP packets from the backends. Under heavy load, it
might miss some statistic messages, well, but that's not as
bad as having locks causing backends to loose
Tom Lane wrote:
Jan Wieck [EMAIL PROTECTED] writes:
What about a collector deamon, fired up by the postmaster and
receiving UDP packets from the backends. Under heavy load, it
might miss some statistic messages, well, but that's not as
bad as having locks causing
* Philip Warner [EMAIL PROTECTED] [010315 16:14] wrote:
At 06:57 15/03/01 -0500, Jan Wieck wrote:
And shared memory has all the interlocking problems we want
to avoid.
I suspect that if we keep per-backend data in a separate area, then we
don;t need locking since there is only
At 06:57 15/03/01 -0500, Jan Wieck wrote:
And shared memory has all the interlocking problems we want
to avoid.
I suspect that if we keep per-backend data in a separate area, then we
don;t need locking since there is only one writer. It does not matter if a
reader gets an inconsistent
At 16:17 15/03/01 -0800, Alfred Perlstein wrote:
Lost data is probably better than incorrect data. Either use locks
or a copying mechanism. People will depend on the data returned
making sense.
But with per-backend data, there is only ever *one* writer to a given set
of counters. Everyone
* Philip Warner [EMAIL PROTECTED] [010315 16:46] wrote:
At 16:17 15/03/01 -0800, Alfred Perlstein wrote:
Lost data is probably better than incorrect data. Either use locks
or a copying mechanism. People will depend on the data returned
making sense.
But with per-backend data, there
* Philip Warner [EMAIL PROTECTED] [010315 17:08] wrote:
At 16:55 15/03/01 -0800, Alfred Perlstein wrote:
* Philip Warner [EMAIL PROTECTED] [010315 16:46] wrote:
At 16:17 15/03/01 -0800, Alfred Perlstein wrote:
Lost data is probably better than incorrect data. Either use locks
or a
At 16:55 15/03/01 -0800, Alfred Perlstein wrote:
* Philip Warner [EMAIL PROTECTED] [010315 16:46] wrote:
At 16:17 15/03/01 -0800, Alfred Perlstein wrote:
Lost data is probably better than incorrect data. Either use locks
or a copying mechanism. People will depend on the data returned
* Philip Warner [EMAIL PROTECTED] [010312 18:56] wrote:
At 13:34 12/03/01 -0800, Alfred Perlstein wrote:
Is it possible
to have a spinlock over it so that an external utility can take a snapshot
of it with the spinlock held?
I'd suggest that locking the stats area might be a bad idea;
This ought to always give a consistant snapshot of the file to
whomever opens it.
I think Tom has previously stated that there are technical reasons not to
do IO in signal handlers, and I have philosophical problems with
performance monitors that ask 50 backends to do file IO. I really
This ought to always give a consistant snapshot of the file to
whomever opens it.
I think Tom has previously stated that there are technical reasons not to
do IO in signal handlers, and I have philosophical problems with
performance monitors that ask 50 backends to do file IO. I really do
* Philip Warner [EMAIL PROTECTED] [010313 06:42] wrote:
This ought to always give a consistant snapshot of the file to
whomever opens it.
I think Tom has previously stated that there are technical reasons not to
do IO in signal handlers, and I have philosophical problems with
At 13:34 12/03/01 -0800, Alfred Perlstein wrote:
Is it possible
to have a spinlock over it so that an external utility can take a snapshot
of it with the spinlock held?
I'd suggest that locking the stats area might be a bad idea; there is only
one writer for each backend-specific chunk,
I was going to implement the signal handler like we do with Cancel,
where the signal sets a flag and we check the status of the flag in
various _safe_ places.
Can anyone think of a better way to get information out of a backend?
--
Bruce Momjian|
* Bruce Momjian [EMAIL PROTECTED] [010312 12:12] wrote:
I was going to implement the signal handler like we do with Cancel,
where the signal sets a flag and we check the status of the flag in
various _safe_ places.
Can anyone think of a better way to get information out of a backend?
Why
At 13:34 12/03/01 -0800, Alfred Perlstein wrote:
Is it possible
to have a spinlock over it so that an external utility can take a snapshot
of it with the spinlock held?
I'd suggest that locking the stats area might be a bad idea; there is only
one writer for each backend-specific chunk, and it
47 matches
Mail list logo
