On 01/21/2014 02:48 PM, Simon Riggs wrote:
I agree with people saying that stddev is better than nothing at all,
so I am inclined to commit this, in spite of the above. Any objections
to commit?
I have not been following terribly closely, but if it includes stddev
then yes, please do, many
On Tue, Jan 21, 2014 at 11:48 AM, Simon Riggs si...@2ndquadrant.com wrote:
I agree with people saying that stddev is better than nothing at all,
so I am inclined to commit this, in spite of the above.
I could live with stddev. But we really ought to be investing in
making pg_stat_statements
On 21 January 2014 21:19, Peter Geoghegan p...@heroku.com wrote:
On Tue, Jan 21, 2014 at 11:48 AM, Simon Riggs si...@2ndquadrant.com wrote:
I agree with people saying that stddev is better than nothing at all,
so I am inclined to commit this, in spite of the above.
I could live with stddev.
On 14 November 2013, Kondo Mitsumasa wrote:
Subject: Re: [HACKERS] Add min and max execute statement time in
pg_stat_statement
Oh! Sorry...
I forgot to attach my latest patch.
* Is the patch in a patch format which has context?
No
* Does it apply cleanly to the current git master?
Yes
(2013/10/21 20:17), KONDO Mitsumasa wrote:
(2013/10/18 22:21), Andrew Dunstan wrote:
If we're going to extend pg_stat_statements, even more than min and max
I'd like to see the standard deviation in execution time.
OK. I do! I am making some other patches, please wait more!
I add stddev_time
Oh! Sorry...
I forgot to attach my latest patch.
Regards,
--
Mitsumasa KONDO
NTT Open Source Software Center
diff --git a/contrib/pg_stat_statements/pg_stat_statements--1.1--1.2.sql b/contrib/pg_stat_statements/pg_stat_statements--1.1--1.2.sql
new file mode 100644
index 000..929d623
---
On Thu, Nov 14, 2013 at 7:11 AM, Peter Geoghegan p...@heroku.com wrote:
On Wed, Oct 23, 2013 at 8:52 PM, Alvaro Herrera
alvhe...@2ndquadrant.com wrote:
Hmm, now if we had portable atomic addition, so that we could spare the
spinlock ...
That certainly seems like an interesting possibility.
(2013/11/14 7:11), Peter Geoghegan wrote:
On Wed, Oct 23, 2013 at 8:52 PM, Alvaro Herrera
alvhe...@2ndquadrant.com wrote:
Hmm, now if we had portable atomic addition, so that we could spare the
spinlock ...
And adding a histogram or
min/max for something like execution time isn't an approach
On Thu, Nov 14, 2013 at 9:09 AM, Fujii Masao masao.fu...@gmail.com wrote:
I think that pg_stat_statements should be made to do this kind of
thing by a third party tool that aggregates snapshots of deltas.
Time-series data, including (approximate) *local* minima and maxima
should be built from
On Thu, Nov 14, 2013 at 6:28 PM, KONDO Mitsumasa
kondo.mitsum...@lab.ntt.co.jp wrote:
It is confirmation just to make sure, does this patch mean my patch? I
agree with you about not adding another lock implementation. It will becomes
overhead.
Yes, I referred to your patch. I don't want to go
(2013/11/15 11:31), Peter Geoghegan wrote:
On Thu, Nov 14, 2013 at 6:28 PM, KONDO Mitsumasa
kondo.mitsum...@lab.ntt.co.jp wrote:
It is confirmation just to make sure, does this patch mean my patch? I
agree with you about not adding another lock implementation. It will becomes
overhead.
Yes, I
(2013/11/15 2:09), Fujii Masao wrote:
Agreed.
Could you tell me your agreed reason? I am sorry that I suspect you doesn't
understand this disccusion enough:-(
Regards,
--
Mitsumasa KONDO
NTT Open Source Software Ceter
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To
On Wed, Oct 23, 2013 at 8:52 PM, Alvaro Herrera
alvhe...@2ndquadrant.com wrote:
Hmm, now if we had portable atomic addition, so that we could spare the
spinlock ...
That certainly seems like an interesting possibility.
I think that pg_stat_statements should be made to do this kind of
thing by
On 10/23/2013 07:51 PM, Peter Geoghegan wrote:
On Wed, Oct 23, 2013 at 4:46 PM, Josh Berkus j...@agliodbs.com wrote:
So you're suggesting that instead of storing the aggregates as we
currently do, we store a buffer of the last N queries (in normal form)
and their stats? And then aggregate
On Thu, Oct 24, 2013 at 6:54 AM, Andrew Dunstan and...@dunslane.net wrote:
I'll be quite happy if we can get around the query text length limit. I have
greatly increased the buffer size at quite a few clients, in one case where
they run some pretty large auto-generated queries and have memory
On 10/23/2013 08:52 PM, Alvaro Herrera wrote:
Peter Geoghegan escribió:
I am interested in making it store richer statistics,
provided we're very careful about the costs. Every time those counters
are incremented, a spinlock is held.
Hmm, now if we had portable atomic addition, so that we
All,
Bringing this down to Earth: yes, it would be useful to have min and
max for pg_stat_statement, and even if we add more stats to
pg_stat_statement, it would be useful to have those two. So can we
approve this patch on that basis?
For my part, I generally use the 9-part percentiles for
Josh,
* Josh Berkus (j...@agliodbs.com) wrote:
On the other hand, it's still true that a high STDDEV indicates a high
variance in the response times of a particular query, whereas a low one
indicates that most are close to the average. While precision math
might not work if we don't have the
On Wed, Oct 23, 2013 at 9:20 AM, Stephen Frost sfr...@snowman.net wrote:
Josh,
* Josh Berkus (j...@agliodbs.com) wrote:
On the other hand, it's still true that a high STDDEV indicates a high
variance in the response times of a particular query, whereas a low one
indicates that most are
* Jeff Janes (jeff.ja...@gmail.com) wrote:
On Wed, Oct 23, 2013 at 9:20 AM, Stephen Frost sfr...@snowman.net wrote:
* Josh Berkus (j...@agliodbs.com) wrote:
On the other hand, it's still true that a high STDDEV indicates a high
variance in the response times of a particular query, whereas
On Wed, Oct 23, 2013 at 1:07 PM, Jeff Janes jeff.ja...@gmail.com wrote:
How does max not answer is this query ever really slow?? But good point,
if we have a max, then I think a time-stamp for when that max was obtained
would also be very useful.
I'm concerned about the cost of all of this.
On Tue, Oct 22, 2013 at 11:16:19AM -0700, Jeff Janes wrote:
On Mon, Oct 21, 2013 at 4:29 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Hm. It's been a long time since college statistics, but doesn't the
entire concept of standard deviation depend on the assumption that the
underlying distribution
On Wed, Oct 23, 2013 at 2:46 PM, Martijn van Oosterhout
klep...@svana.org wrote:
Well, the standard deviation is the square root of the variance, which
is the second moment of the distribution. The first moment being the
mean. No matter what distribution it is, these are useful numbers.
I'm
On 24/10/13 10:34, Marc Mamin wrote:
Oscillating plan changes may fit multimodal but I don't feel that's
typical. My experience has been it's either an extremely rare plan
difference or it's a shift from one plan to another over time.
After all, all of avg, min, max and stdev are only
On Wed, Oct 23, 2013 at 2:57 PM, Gavin Flower
gavinflo...@archidevsys.co.nz wrote:
Looks definitely bimodal in the log version, very clear!
Yes, I feel that having a 32 log binary binned histogram (as Alvaro Herrera
suggested) would be very useful.
I'm having a hard time imagining how you'd
On 24/10/13 11:26, Peter Geoghegan wrote:
On Wed, Oct 23, 2013 at 2:57 PM, Gavin Flower
gavinflo...@archidevsys.co.nz wrote:
Looks definitely bimodal in the log version, very clear!
Yes, I feel that having a 32 log binary binned histogram (as Alvaro Herrera
suggested) would be very useful.
On Wed, Oct 23, 2013 at 3:26 PM, Peter Geoghegan p...@heroku.com wrote:
On Wed, Oct 23, 2013 at 2:57 PM, Gavin Flower
gavinflo...@archidevsys.co.nz wrote:
Looks definitely bimodal in the log version, very clear!
Yes, I feel that having a 32 log binary binned histogram (as Alvaro
Herrera
On 24/10/13 12:00, Gavin Flower wrote:
On 24/10/13 11:26, Peter Geoghegan wrote:
On Wed, Oct 23, 2013 at 2:57 PM, Gavin Flower
gavinflo...@archidevsys.co.nz wrote:
Looks definitely bimodal in the log version, very clear!
Yes, I feel that having a 32 log binary binned histogram (as Alvaro
On Wed, Oct 23, 2013 at 4:00 PM, Gavin Flower gavinflo...@archidevsys.co.nz
wrote:
On 24/10/13 11:26, Peter Geoghegan wrote:
On Wed, Oct 23, 2013 at 2:57 PM, Gavin Flower
gavinflo...@archidevsys.co.nz** wrote:
Looks definitely bimodal in the log version, very clear!
Yes, I feel that
On Wed, Oct 23, 2013 at 4:14 PM, Jeff Janes jeff.ja...@gmail.com wrote:
The last bucket would be limited to 8ms x = 16 ms. If you find something
16ms, then you have to rescale *before* you increment any of the buckets.
Once you do, there is now room to hold it.
How is that laid out in shared
* Martijn van Oosterhout (klep...@svana.org) wrote:
If I had to guess a distribution for query runtimes I'd go for Poisson,
which would mean you'd expect the mean to equal the variance. Don't
have enough experience with such measurements to say whether that is
reasonable.
I was thinking the
On 24/10/13 12:14, Jeff Janes wrote:
On Wed, Oct 23, 2013 at 4:00 PM, Gavin Flower
gavinflo...@archidevsys.co.nz mailto:gavinflo...@archidevsys.co.nz
wrote:
On 24/10/13 11:26, Peter Geoghegan wrote:
On Wed, Oct 23, 2013 at 2:57 PM, Gavin Flower
* Peter Geoghegan (p...@heroku.com) wrote:
On Wed, Oct 23, 2013 at 4:14 PM, Jeff Janes jeff.ja...@gmail.com wrote:
The last bucket would be limited to 8ms x = 16 ms. If you find something
16ms, then you have to rescale *before* you increment any of the buckets.
Once you do, there is now
On 10/23/2013 01:26 PM, Peter Geoghegan wrote:
So fixing that problem would go a long
way towards resolving these concerns. It would also probably have the
benefit of making it possible for query texts to be arbitrarily long -
we'd be storing them in files (with a shared memory buffer). I get
On Wed, Oct 23, 2013 at 4:34 PM, Stephen Frost sfr...@snowman.net wrote:
Any chance we could accumulate into the histogram in local memory and
only push to the shared memory on an infrequent or at least less
frequent basis? Apologies, I've not looked into the pg_stat_statements
bits all that
On 24/10/13 12:24, Peter Geoghegan wrote:
On Wed, Oct 23, 2013 at 4:14 PM, Jeff Janes jeff.ja...@gmail.com wrote:
The last bucket would be limited to 8ms x = 16 ms. If you find something
16ms, then you have to rescale *before* you increment any of the buckets.
Once you do, there is now room
On Wed, Oct 23, 2013 at 4:46 PM, Josh Berkus j...@agliodbs.com wrote:
So you're suggesting that instead of storing the aggregates as we
currently do, we store a buffer of the last N queries (in normal form)
and their stats? And then aggregate when the user asks for it?
No, I'm not. I'm
On 24/10/13 12:46, Josh Berkus wrote:
On 10/23/2013 01:26 PM, Peter Geoghegan wrote:
So fixing that problem would go a long
way towards resolving these concerns. It would also probably have the
benefit of making it possible for query texts to be arbitrarily long -
we'd be storing them in files
On Wed, Oct 23, 2013 at 4:48 PM, Gavin Flower
gavinflo...@archidevsys.co.nz wrote:
32 int64 buckets is only 256 bytes, so a thousand histograms would be less
than a quarter of a MB. Any machine that busy, would likely have many GB's
of RAM. I have 32 GB on my development machine.
Who wants
On Wed, Oct 23, 2013 at 4:51 PM, Peter Geoghegan p...@heroku.com wrote:
No, I'm not. I'm suggesting storing the query texts externally, in a
file. They usually use 1024 bytes of shared memory per entry,
regardless of how long the query text is.
I should add that I think that that's about the
On 24/10/13 12:58, Peter Geoghegan wrote:
On Wed, Oct 23, 2013 at 4:48 PM, Gavin Flower
gavinflo...@archidevsys.co.nz wrote:
32 int64 buckets is only 256 bytes, so a thousand histograms would be less
than a quarter of a MB. Any machine that busy, would likely have many GB's
of RAM. I have 32
On Wed, Oct 23, 2013 at 5:15 PM, Gavin Flower
gavinflo...@archidevsys.co.nz wrote:
Anyhow, I was suggesting the faclity be off by default - I see no point in
enabling where people don't need it, even if the resource RAM, processor,
whatever, were minimal.
As long as any new field in the
Peter Geoghegan escribió:
I am interested in making it store richer statistics,
provided we're very careful about the costs. Every time those counters
are incremented, a spinlock is held.
Hmm, now if we had portable atomic addition, so that we could spare the
spinlock ...
--
Álvaro Herrera
Tom Lane t...@sss.pgh.pa.us writes:
Hm. It's been a long time since college statistics, but doesn't the
entire concept of standard deviation depend on the assumption that the
underlying distribution is more-or-less normal (Gaussian)? Is there a
I just had a quick chat with a statistician
On Tue, Oct 22, 2013 at 2:56 AM, Dimitri Fontaine
dimi...@2ndquadrant.fr wrote:
Tom Lane t...@sss.pgh.pa.us writes:
Hm. It's been a long time since college statistics, but doesn't the
entire concept of standard deviation depend on the assumption that the
underlying distribution is
On 22/10/13 22:56, Dimitri Fontaine wrote:
Tom Lane t...@sss.pgh.pa.us writes:
Hm. It's been a long time since college statistics, but doesn't the
entire concept of standard deviation depend on the assumption that the
underlying distribution is more-or-less normal (Gaussian)? Is there a
I
All,
* Dimitri Fontaine (dimi...@2ndquadrant.fr) wrote:
In our case, what I keep experiencing with tuning queries is that we
have like 99% of them running under acceptable threshold and 1% of them
taking more and more time.
This is usually described (at least where I come from) as 'rare
On Mon, Oct 21, 2013 at 4:29 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Andrew Dunstan and...@dunslane.net writes:
This is why I suggested the standard deviation, and why I find it would
be more useful than just min and max. A couple of outliers will set the
min and max to possibly extreme
On Mon, Oct 21, 2013 at 1:36 PM, Robert Haas robertmh...@gmail.com wrote:
On Mon, Oct 21, 2013 at 4:01 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Gavin Flower gavinflo...@archidevsys.co.nz writes:
If we're going to extend pg_stat_statements, even more than min and
max
I'd like to see the
Hi All,
(2013/10/22 22:26), Stephen Frost wrote:
* Dimitri Fontaine (dimi...@2ndquadrant.fr) wrote:
In our case, what I keep experiencing with tuning queries is that we
have like 99% of them running under acceptable threshold and 1% of them
taking more and more time.
This is usually
(2013/10/18 22:21), Andrew Dunstan wrote:
If we're going to extend pg_stat_statements, even more than min and max
I'd like to see the standard deviation in execution time.
OK. I do! I am making some other patches, please wait more!
Regards,
--
Mitsumasa KONDO
NTT Open Source Software Center.;
On 22/10/13 00:17, KONDO Mitsumasa wrote:
(2013/10/18 22:21), Andrew Dunstan wrote:
If we're going to extend pg_stat_statements, even more than min and max
I'd like to see the standard deviation in execution time.
OK. I do! I am making some other patches, please wait more!
Regards,
--
Gavin Flower gavinflo...@archidevsys.co.nz writes:
If we're going to extend pg_stat_statements, even more than min and max
I'd like to see the standard deviation in execution time.
How about the 'median', often a lot more useful than the 'arithmetic
mean' (which most people call the
On Mon, Oct 21, 2013 at 4:01 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Gavin Flower gavinflo...@archidevsys.co.nz writes:
If we're going to extend pg_stat_statements, even more than min and max
I'd like to see the standard deviation in execution time.
How about the 'median', often a lot more
On Mon, Oct 21, 2013 at 1:36 PM, Robert Haas robertmh...@gmail.com wrote:
Yeah, and I worry about min and max not being very usable - once they
get pushed out to extreme values, there's nothing to drag them back
toward normality except resetting the stats, and that's not something
we want to
On 22/10/13 09:01, Tom Lane wrote:
Gavin Flower gavinflo...@archidevsys.co.nz writes:
If we're going to extend pg_stat_statements, even more than min and max
I'd like to see the standard deviation in execution time.
How about the 'median', often a lot more useful than the 'arithmetic
mean'
Gavin Flower wrote:
One way it could be done, but even this would consume far too much
storage and processing power (hence totally impractical), would be
to 'simply' store a counter for each value found and increment it
for each occurence...
An histogram? Sounds like a huge lot of code
On 10/21/2013 04:43 PM, Peter Geoghegan wrote:
On Mon, Oct 21, 2013 at 1:36 PM, Robert Haas robertmh...@gmail.com wrote:
Yeah, and I worry about min and max not being very usable - once they
get pushed out to extreme values, there's nothing to drag them back
toward normality except resetting
Robert Haas robertmh...@gmail.com writes:
Yeah, and I worry about min and max not being very usable - once they
get pushed out to extreme values, there's nothing to drag them back
toward normality except resetting the stats, and that's not something
we want to encourage people to do
Andrew Dunstan and...@dunslane.net writes:
This is why I suggested the standard deviation, and why I find it would
be more useful than just min and max. A couple of outliers will set the
min and max to possibly extreme values but hardly perturb the standard
deviation over a large number of
On 10/21/2013 07:29 PM, Tom Lane wrote:
Andrew Dunstan and...@dunslane.net writes:
This is why I suggested the standard deviation, and why I find it would
be more useful than just min and max. A couple of outliers will set the
min and max to possibly extreme values but hardly perturb the
On Mon, Oct 21, 2013 at 4:29 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Hm. It's been a long time since college statistics, but doesn't the
entire concept of standard deviation depend on the assumption that the
underlying distribution is more-or-less normal (Gaussian)?
I don't see how. The
On Tue, Oct 22, 2013 at 1:09 AM, Alvaro Herrera
alvhe...@2ndquadrant.com wrote:
Gavin Flower wrote:
One way it could be done, but even this would consume far too much
storage and processing power (hence totally impractical), would be
to 'simply' store a counter for each value found and
On 22/10/13 13:26, Ants Aasma wrote:
On Tue, Oct 22, 2013 at 1:09 AM, Alvaro Herrera
alvhe...@2ndquadrant.com wrote:
Gavin Flower wrote:
One way it could be done, but even this would consume far too much
storage and processing power (hence totally impractical), would be
to 'simply' store a
On 22/10/13 13:26, Ants Aasma wrote:
On Tue, Oct 22, 2013 at 1:09 AM, Alvaro Herrera
alvhe...@2ndquadrant.com wrote:
Gavin Flower wrote:
One way it could be done, but even this would consume far too much
storage and processing power (hence totally impractical), would be
to 'simply' store a
On Tue, Oct 22, 2013 at 4:00 AM, Gavin Flower
gavinflo...@archidevsys.co.nz wrote:
I have a proof of concept patch somewhere that does exactly this. I
used logarithmic bin widths. With 8 log10 bins you can tell the
fraction of queries running at each order of magnitude from less than
1ms to
On 10/18/2013 04:02 AM, KONDO Mitsumasa wrote:
I submit patch adding min and max execute statement time in pg_stat_statement
in
next CF.
pg_stat_statement have execution time, but it is average execution time and
does
not provide detail information very much. So I add min and max execute
101 - 167 of 167 matches
Mail list logo
