Hello Andres,
pgbench postgres -M prepared -c 72 -j 72 -P 5 -T 3600 -R4 -L100
progress: 365.0 s, 18392.1 tps, lat 94.857 ms stddev 23.917, lag 90.980 ms,
106244 skipped
100k skipped transactions at a rate limit of 40k? That doesn't seem right.
Argh. It's just because I used -P5.
On 2015/10/22 18:20, Fabien COELHO wrote:
>>
>>> progress: 365.0 s, 18392.1 tps, lat 94.857 ms stddev 23.917, lag 90.980
>>> ms, 106244 skipped
>>>
>>> 100k skipped transactions at a rate limit of 40k? That doesn't seem right.
>>
>> Argh. It's just because I used -P5. It's a bit confusing that the
Argh. It's just because I used -P5. It's a bit confusing that the other
options are per second, and this is per interval...
I agree, but I'm unsure of a fix, beyond what is already done which is to
show units next to the figures...
ISTM that people expect "tps" for performance, even on
On 2014-08-14 15:01:53 +0200, Fabien COELHO wrote:
>
> Add --limit to limit latency under throttling
>
> Under throttling, transactions are scheduled for execution at certain times.
> Transactions may be far behind schedule and the system may catch up with the
> load later. This option allows to
On 2015-10-20 20:55:46 +0200, Andres Freund wrote:
> On 2014-08-14 15:01:53 +0200, Fabien COELHO wrote:
> >
> > Add --limit to limit latency under throttling
> >
> > Under throttling, transactions are scheduled for execution at certain times.
> > Transactions may be far behind schedule and the
On 10/09/2014 10:39 PM, Fabien COELHO wrote:
One thing bothers me with the log format. Here's an example:
0 81 4621 0 1412881037 912698 3005
0 82 6173 0 1412881037 914578 4304
0 83 skipped 0 1412881037 914578 5217
0 83 skipped 0 1412881037 914578 5099
0 83 4722 0 1412881037 916203
On 10/13/14, 1:54 PM, Heikki Linnakangas wrote:
Greg Smith, I'd still appreciate it if you could take a look at this,
to check how this will work for pgbench-tools.
I'll do a QA pass on the committed version looking for issues, and
update the toolchain I publish to be compatible with it along
On 10/05/2014 10:43 AM, Fabien COELHO wrote:
Hello Heikki,
Here are new patches, again the first one is just refactoring, and the second
one contains this feature. I'm planning to commit the first one shortly, and
the second one later after people have had a chance to look at it.
I looked
Hello Heikki,
Here are new patches, again the first one is just refactoring, and the second
one contains this feature. I'm planning to commit the first one shortly, and
the second one later after people have had a chance to look at it.
I looked at it. It looks ok, but for a few spurious
On 09/15/2014 08:46 PM, Fabien COELHO wrote:
I'm not sure I like the idea of printing a percentage. It might be
unclear what the denominator was if somebody feels the urge to work
back to the actual number of skipped transactions. I mean, I guess
it's probably just the value you passed to
On 09/13/2014 11:25 AM, Fabien COELHO wrote:
[about logging...]
Here is an attempt at updating the log features, including the aggregate
and sampling stuff, with skipped transactions under throttling.
I moved the logging stuff into a function which is called when a
transaction is skipped or
On Mon, Sep 15, 2014 at 6:34 AM, Heikki Linnakangas
hlinnakan...@vmware.com wrote:
Please have a look. I have not looked at the docs changes yet.
One thing that needs some thinking and changing is the progress reporting.
It currently looks like this:
progress: 1.0 s, 4863.0 tps, lat 3.491 ms
I'm not sure I like the idea of printing a percentage. It might be
unclear what the denominator was if somebody feels the urge to work
back to the actual number of skipped transactions. I mean, I guess
it's probably just the value you passed to -R, so maybe that's easy
enough, but then why
On Sat, Sep 13, 2014 at 4:25 AM, Fabien COELHO coe...@cri.ensmp.fr wrote:
[about logging...]
Here is an attempt at updating the log features, including the aggregate and
sampling stuff, with skipped transactions under throttling.
I moved the logging stuff into a function which is called when
[about logging...]
Here is an attempt at updating the log features, including the aggregate
and sampling stuff, with skipped transactions under throttling.
I moved the logging stuff into a function which is called when a
transaction is skipped or finished.
From a log file format
On 09/11/2014 03:36 PM, Fabien COELHO wrote:
Hello Heikki,
Now that I've finished the detour and committed and backpatched the changes
to the way latency is calculated, we can get back to this patch. It needs to
be rebased.
Before rebasing, I think that there are a few small problems with
On 09/11/2014 05:16 PM, Fabien COELHO wrote:
How should skipped transactions should be taken into account in the log file
output, with and without aggregation? I assume we'll want to have some trace
of skipped transactions in the logs.
The problem with this point is that how to report
(1) ISTM that the + 0.5 which remains in the PoissonRand computation comes
from the previous integer approach and is not needed here. If I'm not
mistaken the formula should be plain:
-log(uniform) * center
No. The +0.5 is to round the result to the nearest integer, instead of
The output would look something like this (modified from the manual's example
by hand, so the numbers don't add up):
0 199 2241 0 1175850568 995598 1020
0 200 2465 0 1175850568 998079 1010
0 201 skipped 1175850569 608 3011
0 202 skipped 1175850569 608 2400
0 203 skipped 1175850569 608 1000
0
On 9/10/14, 10:57 AM, Fabien COELHO wrote:
Indeed. I think that people do not like it to change. I remember that
I suggested to change timestamps to .yy instead of the
unreadable yyy, and be told not to, because some people have
tool which process the output so the format MUST NOT
On Fri, Sep 12, 2014 at 2:27 PM, Gregory Smith gregsmithpg...@gmail.com wrote:
If there's a good case that the whole format needs to be changed anyway,
like adding a new field, then we might as well switch to fractional epoch
timestamps too now though. When I added timestamps to the latency
On 09/12/2014 08:59 PM, Fabien COELHO wrote:
The output would look something like this (modified from the manual's example
by hand, so the numbers don't add up):
0 199 2241 0 1175850568 995598 1020
0 200 2465 0 1175850568 998079 1010
0 201 skipped 1175850569 608 3011
0 202 skipped 1175850569
On 09/10/2014 05:47 PM, Mitsumasa KONDO wrote:
Hi,
I find typo in your patch. Please confirm.
@line 239
- agg-sum2_lag = 0;
+ agg-sum_lag = 0;
Ah thanks, cood catch!
And back patch is welcome for me.
I've committed and backpatched this, as well as a patch to refactor the
way the
On 08/30/2014 07:16 PM, Fabien COELHO wrote:
+ if (latency_limit)
+ printf(number of transactions above the %.1f ms latency limit:
INT64_FORMAT \n,
+ latency_limit / 1000.0, latency_late);
+
Any reason not to report a percentage here?
Yes: I
Hello Heikki,
Now that I've finished the detour and committed and backpatched the changes
to the way latency is calculated, we can get back to this patch. It needs to
be rebased.
Before rebasing, I think that there are a few small problems with the
modification applied to switch from an
(3) I wish that the maximum implied multiplier could be explicitely
documented in the source code. From pg_rand48 source code, I think
that it is 33.27106466687737
Small possibly buggy code attached, to show how I computed the above
figure.
--
Fabien.#include math.h
#include stdio.h
Hello Heikki
Now that I've finished the detour and committed and backpatched the changes
to the way latency is calculated, we can get back to this patch. It needs to
be rebased.
Here is the rebase, which seems ok.
See also the small issues raised aboud getPoissonRand in another email.
--
How should skipped transactions should be taken into account in the log file
output, with and without aggregation? I assume we'll want to have some trace
of skipped transactions in the logs.
The problem with this point is that how to report something not done is
unclear, especially as the
On 09/09/2014 03:35 PM, Fabien COELHO wrote:
Hello Heikki,
I think we have to reconsider what we're reporting in 9.4, when --rate
is enabled, even though it's already very late in the release cycle.
It's a bad idea to change the definition of latency between 9.4 and 9.5,
so let's get it right
Hi,
I find typo in your patch. Please confirm.
@line 239
- agg-sum2_lag = 0;
+ agg-sum_lag = 0;
And back patch is welcome for me.
Best Regards,
--
Mitsumasa KONDO
Hello Heikki,
I looked closer at the this, and per Jan's comments, realized that we don't
log the lag time in the per-transaction log file. I think that's a serious
omission; when --rate is used, the schedule lag time is important information
to make sense of the result. I think we have to
On 09/10/2014 05:57 PM, Fabien COELHO wrote:
Hello Heikki,
I looked closer at the this, and per Jan's comments, realized that we don't
log the lag time in the per-transaction log file. I think that's a serious
omission; when --rate is used, the schedule lag time is important information
to
On 09/10/2014 11:28 AM, Heikki Linnakangas wrote:
On 09/10/2014 05:57 PM, Fabien COELHO wrote:
Hello Heikki,
I looked closer at the this, and per Jan's comments, realized that we don't
log the lag time in the per-transaction log file. I think that's a serious
omission; when --rate is used,
On 09/05/2014 06:38 PM, Jan Wieck wrote:
On 09/05/2014 10:12 AM, Fabien COELHO wrote:
Note that despite pg appaling latency performance, in may stay well over
the 90% limit, or even 100%: when things are going well a lot of
transaction run in about ms, while when things are going bad
On 09/09/2014 01:49 PM, Heikki Linnakangas wrote:
I think we have to reconsider what we're reporting in 9.4, when --rate
is enabled, even though it's already very late in the release cycle.
It's a bad idea to change the definition of latency between 9.4 and 9.5,
so let's get it right in 9.4.
Hello Heikki,
I think we have to reconsider what we're reporting in 9.4, when --rate
is enabled, even though it's already very late in the release cycle.
It's a bad idea to change the definition of latency between 9.4 and 9.5,
so let's get it right in 9.4.
Indeed.
As per the attached patch.
On 08/27/2014 04:08 AM, Heikki Linnakangas wrote:
That model might make some sense if you think e.g. of a web application,
where the web server has a timeout for how long it waits to get a
database connection from a pool, but once a query is started, the
transaction is considered a succeess no
That model might make some sense if you think e.g. of a web application,
where the web server has a timeout for how long it waits to get a
database connection from a pool, but once a query is started, the
transaction is considered a succeess no matter how long it takes. The
latency limit would
On 09/05/2014 10:12 AM, Fabien COELHO wrote:
Note that despite pg appaling latency performance, in may stay well over
the 90% limit, or even 100%: when things are going well a lot of
transaction run in about ms, while when things are going bad transactions
would take a long time (although
Hi,
I generally want to say that having a feature like this feels *very*
helpful to me. Lots of pg development hasn't really paid attention to
anything but the final pgbench results...
On 2014-08-29 19:48:43 +0200, Fabien COELHO wrote:
+ if (latency_limit)
+ printf(number of
+ if (latency_limit)
+ printf(number of transactions above the %.1f ms latency limit:
INT64_FORMAT \n,
+ latency_limit / 1000.0, latency_late);
+
Any reason not to report a percentage here?
Yes: I did not thought of it.
Here is a v7, with a
On 08/27/2014 08:05 PM, Fabien COELHO wrote:
[...]
Yeah, something like that. I don't think it would be necessary to set
statement_timeout, you can inject that in your script or postgresql.conf if
you want. I don't think aborting a transaction that's already started is
necessary either. You
Hello Heikki,
This now begs the question:
In --rate mode, shouldn't the reported transaction latency also be calculated
from the *scheduled* start time, not the time the transaction actually
started? Otherwise we're using two different definitions of latency, one
for the purpose of the
Hello Heikki,
[...] I would be fine with both.
After giving it some thought, ISTM better to choose consistency over
intuition, and have latency under throttling always defined wrt the
scheduled start time and not the actual start time, even if having a
latency of 1 ms for an OLTP load
On 08/27/2014 03:47 AM, Rukh Meski wrote:
Hi Fabien,
On Tue, Aug 26, 2014 at 04:07 AM, Fabien COELHO coe...@cri.ensmp.fr wrote:
Please find attached a new version which fixes these two points.
Indeed it does. Marking the patch ready for a committer.
I find the definition of the latency
Hello Heikki,
I find the definition of the latency limit a bit strange. It's a limit on how
late a transaction can *start* compared to it's scheduled starting time, not
how long a query is allowed to last.
Yes. This is what can be done easily with pgbench under throttling. Note
that if
On 08/27/2014 12:41 PM, Fabien COELHO wrote:
Hello Heikki,
I find the definition of the latency limit a bit strange. It's a limit on how
late a transaction can *start* compared to it's scheduled starting time, not
how long a query is allowed to last.
Yes. This is what can be done easily
Hello Heikki,
[...]
With a latency limit on when the query should finish, as opposed to how
late it can start, it's a lot easier to give a number. For example, your
requirements might state that a user must always get a response to a click on
a web page in 200 ms, so you set the limit to
On 08/27/2014 02:37 PM, Fabien COELHO wrote:
As for an actual latency limit under throttling, this is significantly
more tricky and invasive to implement... ISTM that it would mean:
- if the tx is not stated an the latency is already consummed, SKIP++.
- if the tx is after its schedule
As for an actual latency limit under throttling, this is significantly
more tricky and invasive to implement... ISTM that it would mean:
[...]
Yeah, something like that. I don't think it would be necessary to set
statement_timeout, you can inject that in your script or postgresql.conf if
On 08/27/2014 06:08 PM, Fabien COELHO wrote:
I've submitted this simple lag limit version because being able to
measure quickly and simply (un)responsiveness seems like a good idea,
especially given the current state of things.
Ok, fair enough. I don't think doing a latency limit would be
[...]
Yeah, something like that. I don't think it would be necessary to set
statement_timeout, you can inject that in your script or postgresql.conf if
you want. I don't think aborting a transaction that's already started is
necessary either. You could count it as LATE, but let it finish
Hi Fabien,
On Sun, Aug 24, 2014 at 9:16 AM, Fabien COELHO coe...@cri.ensmp.fr wrote:
Find attached a new version:
- fix dropped percent computation in the final report
- simplify progress report code
I have reviewed this patch.
Is the patch in a patch format which has context?
Yes.
Does
Hello Rukh,
I have reviewed this patch.
Thanks!
[...] I get: pgbench: invalid option -- L
Which appears to be caused by the fact that the call to getopt_long()
has not been updated to reflect the new parameter.
Indeed, I only tested/used it with the --limit= syntax.
Also this part:
+
Marking Waiting for Author until these small issues have been fixed.
I've put it back to Needs review. Feel free to set it to Ready if it
is ok for you.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
Hi Fabien,
On Tue, Aug 26, 2014 at 04:07 AM, Fabien COELHO coe...@cri.ensmp.fr wrote:
Please find attached a new version which fixes these two points.
Indeed it does. Marking the patch ready for a committer.
Thanks,
♜
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
Add --limit to limit latency under throttling
Under throttling, transactions are scheduled for execution at certain times.
Transactions may be far behind schedule and the system may catch up with the
load later. This option allows to change this behavior by skipping
transactions which are
Add --limit to limit latency under throttling
Under throttling, transactions are scheduled for execution at certain
times. Transactions may be far behind schedule and the system may catch up
with the load later. This option allows to change this behavior by
skipping transactions which are
58 matches
Mail list logo
