It might be done in the other direction, though - the "writer" thread
might collect current results at the end of the interval.
Yep, you can indeed accumulate per thread and sum on the end of the
interval, but a lock is still needed if you want exact figures.
ISTM that it is what is done fo
Hi,
On 05/02/15 15:30, Fabien COELHO wrote:
Hello,
The counters are updated when the transaction is finished anyway?
Yes, but the thread does not know it's time to write the results until
it completes the first transaction after the interval ends ...
Let's say the very first query in threa
Hello,
The counters are updated when the transaction is finished anyway?
Yes, but the thread does not know it's time to write the results until
it completes the first transaction after the interval ends ...
Let's say the very first query in thread #1 takes a minute for some
reason, while the
On 29.3.2015 10:58, Fabien COELHO wrote:
>>> So my overall conclusion is:
>>>
>>> (1) The simple thread-shared file approach would save pgbench from
>>> post-processing merge-sort heavy code, for a reasonable cost.
>>
>> No it wouldn't - you're missing the fact that the proposed approach
>> (share
I also implemented a quick and dirty version for a merge log based on
sharing a file handle (append mode + sprintf + fputs).
I tried the "append + per-thread 2KB buffered sprintf + fputs when full",
with the same number of runs. The logs are out of order by chunks, the
overhead seems higher w
Hello Tomas,
The results are as follow:
* 1 thread 33 runs median tps (average is consistent):
- no logging:22062
- separate logging: 19360 (-12.2%)
- merged logging:19326 (-12.4%, not significant from previous)
Interesting. What hardware is this?
Dell PowerEdge R720 wit
On 28.3.2015 11:21, Fabien COELHO wrote:
>
> Hello Tomas,
>
>> I do agree that fprintf is not cheap, actually when profiling pgbench
>> it's often the #1 item, but the impact on the measurements is actually
>> quite small. For example with a small database (scale 10) and read-only
>> 30-second ru
Hello Tomas,
I do agree that fprintf is not cheap, actually when profiling pgbench
it's often the #1 item, but the impact on the measurements is actually
quite small. For example with a small database (scale 10) and read-only
30-second runs (single client), I get this:
no logging: 18672 1879
On March 23, 2015 8:00:04 PM GMT+01:00, Fabien COELHO
wrote:
>
>> Guys, I don't see this theoretical discussion going anywhere. I think
>> it's time to simply implement this and evaluate it on a bigger
>> machine.
>
>Sure. I was kind of hoping that someone else would implement it,
>because
>I'm
Guys, I don't see this theoretical discussion going anywhere. I think
it's time to simply implement this and evaluate it on a bigger
machine.
Sure. I was kind of hoping that someone else would implement it, because
I'm a reviewer on this one, and I do not have the bigger machine at hand
eith
Hi,
Guys, I don't see this theoretical discussion going anywhere. I think
it's time to simply implement this and evaluate it on a bigger
machine. It can't take very long to implement tosimply just write to one
file instead of the multiple files as now. The posix guaranteed fprintf
locking should a
Hello,
Yes but for a third thread (each on a physical core) it will be 1/40 +
1/40 and so on up to roughly 40/40 for 40 cores.
That is why I proposed a formula which depends on the number of threads.
[...] But they aren't constant only close. It may or not show up in this
case but I've noti
Hi,
On Sat, Mar 21, 2015 at 8:42 PM, Fabien COELHO wrote:
>
> Hello Didier,
>
>>> If fprintf takes p = 0.025 (1/40) of the time, then with 2 threads the
>>> collision probability would be about 1/40 and the delayed thread would be
>>> waiting for half this time on average, so the performance impa
Well, fprintf() doesn't have to acquire the lock for the entirety of
it's operation - just for the access to the stream buffer.
Yep. If it is implemented by appending stuff to the stream as the format
is processed, this would still mean the whole time of its operation.
Hence preprocessing t
Hello Didier,
If fprintf takes p = 0.025 (1/40) of the time, then with 2 threads the
collision probability would be about 1/40 and the delayed thread would be
waiting for half this time on average, so the performance impact due to
fprintf locking would be negligeable (1/80 delay occured in 1/40
Hello Andres,
With your worst-case figure and some rounding, it seems to look like:
#threadscollision probabilityperformance impact
2 1/401/3200
4 1/7 1/533
8 0.7 < 0.01 (about
Hi,
On Sat, Mar 21, 2015 at 10:37 AM, Fabien COELHO wrote:
>> no logging: 18672 18792 18667 18518 18613 18547
>> with logging: 18170 18093 18162 18273 18307 18234
>>
>> So on average, that's 18634 vs. 18206, i.e. less than 2.5% difference.
>> And with more expensive transactions (larger scale,
On 2015-03-17 11:50:28 -0400, Robert Haas wrote:
> On Tue, Mar 17, 2015 at 11:27 AM, Fabien COELHO wrote:
> > The fprintf we are talking about occurs at most once per pgbench
> > transaction, possibly much less when aggregation is activated, and this
> > transaction involves networks exchanges and
On 2015-03-21 10:37:05 +0100, Fabien COELHO wrote:
>
> Hello Tomas,
> Let us take this as a worst-case figure and try some maths.
>
> If fprintf takes p = 0.025 (1/40) of the time, then with 2 threads the
> collision probability would be about 1/40 and the delayed thread would be
> waiting for ha
Hello Tomas,
My point is that if there are many threads and tremendous TPS, the
*detailed* per-transaction log (aka simple log) is probably a bad
choice anyway, and the aggregated version is the way to go.
I disagree with this reasoning. Can you provide numbers supporting it?
I'm not sure w
Hi,
On 20.3.2015 13:43, Fabien COELHO wrote:
>
> Hello Robert,
>
>>> The fprintf we are talking about occurs at most once per pgbench
>>> transaction, possibly much less when aggregation is activated,
>>> and this transaction involves networks exchanges and possibly
>>> disk writes on the serve
Hello Robert,
The fprintf we are talking about occurs at most once per pgbench
transaction, possibly much less when aggregation is activated, and this
transaction involves networks exchanges and possibly disk writes on the
server.
random() was occurring four times per transaction rather than
Hello,
I agree, but I think it is due to the multi process thread emulation: if you
have real threads, you can do a simple fprintf, possibly with some mutex,
and you're done. There is really nothing to do to implement this
feature.
I think that's probably not a good idea, because fprintf() mi
On Tue, Mar 17, 2015 at 11:27 AM, Fabien COELHO wrote:
> The fprintf we are talking about occurs at most once per pgbench
> transaction, possibly much less when aggregation is activated, and this
> transaction involves networks exchanges and possibly disk writes on the
> server.
random() was occu
On Sun, Mar 15, 2015 at 3:35 PM, Fabien COELHO wrote:
>> Firstly, the fact that pgbench produces one file per thread is awkward.
>
> I agree, but I think it is due to the multi process thread emulation: if you
> have real threads, you can do a simple fprintf, possibly with some mutex,
> and you're
I agree that inter-process stuff should be avoided. This is not what
I had in mind. I was thinking of "fprintf" on the same file handler
by different threads.
That still involves some sort of 'implicit' locking, no?
I fully agree that it would need locking, wether it is implicit if fprintf
On 15.3.2015 20:35, Fabien COELHO wrote:
>
>> Firstly, the fact that pgbench produces one file per thread is awkward.
>
> I agree, but I think it is due to the multi process thread emulation: if
> you have real threads, you can do a simple fprintf, possibly with some
> mutex, and you're done. The
Firstly, the fact that pgbench produces one file per thread is awkward.
I agree, but I think it is due to the multi process thread emulation: if
you have real threads, you can do a simple fprintf, possibly with some
mutex, and you're done. There is really nothing to do to implement this
fea
On 15.3.2015 11:22, Fabien COELHO wrote:
>
> I've looked at the patch. Although I think that such a feature is
> somehow desirable... I have two issues with it: ISTM that
>
> (1) it does not belong to pgbench as such
>
> (2) even if, the implementation is not right
>
> About (1):
>
> I think th
Sun, 15 Mar 2015 11:22:01 +0100 (CET)
Hello Tomas,
attached is a patch implementing merging of pgbench logs. These logs are
written by each thread, so with N threads you get N files with names
pgbench_log.PID
pgbench_log.PID.1
...
pgbench_log.PID.N
Before analyzing these logs, thes
Hi there,
attached is a patch implementing merging of pgbench logs. These logs are
written by each thread, so with N threads you get N files with names
pgbench_log.PID
pgbench_log.PID.1
...
pgbench_log.PID.N
Before analyzing these logs, these files need to be combined. I usually
31 matches
Mail list logo