Hello Dave,
I am running pgbench with the following
pgbench -h localhost -c 100 -j 100 -t 2 -S -s 1000 pgbench -U pgbench
--protocol=simple
Without pgbouncer I get around 5k TPS
with pgbouncer I get around 15k TPS
Looking at the code connection initiation time should not be part of the
>
> I've recently run into something I am having difficulty understanding.
>
> I am running pgbench with the following
> pgbench -h localhost -c 100 -j 100 -t 2 -S -s 1000 pgbench -U pgbench
> --protocol=simple
>
> Without pgbouncer I get around 5k TPS
> with pgbouncer I get around 15k TPS
>
>
Dave Cramer
www.postgres.rocks
On Tue, 16 May 2023 at 07:27, Andres Freund wrote:
> Hi,
>
> I am trying to run a few benchmarks measuring the effects of patch to
> make GetSnapshotData() faster in the face of larger numbers of
> established connections.
>
> Before the patch connection
Hello Thomas,
I must say that I'm not a big fan of the macro-based all-in-capitals API
for threads because it exposes some platform specific uglyness (eg
THREAD_FUNC_CC) and it does not look much like clean C code when used. I
liked the previous partial pthread implementation better, even if
On Sat, Mar 13, 2021 at 9:08 PM Fabien COELHO wrote:
> I must say that I'm not a big fan of the macro-based all-in-capitals API
> for threads because it exposes some platform specific uglyness (eg
> THREAD_FUNC_CC) and it does not look much like clean C code when used. I
> liked the previous
Hello Thomas,
David Rowley kindly tested this for me on Windows and told me how to
fix one of the macros that had incorrect error checking on that OS.
So here's a new version. I'm planning to commit 0001 and 0002 soon,
if there are no objections. 0003 needs some more review.
I made a few
Thomas Munro writes:
> On Sat, Mar 13, 2021 at 4:59 PM Tom Lane wrote:
>> Looks reasonable by eyeball. If you'd push it, I can launch
>> a gaur run right away.
> Done.
gaur's gotten through "make" and "make check" cleanly. Unfortunately
I expect it will fail at the pg_amcheck test before it
On Sat, Mar 13, 2021 at 4:59 PM Tom Lane wrote:
> Thomas Munro writes:
> > Thanks. This seems to work for me on a Mac. I confirmed with nm that
> > we don't define or reference any pthread_XXX symbols with
> > --disable-thread-safety, and we do otherwise, and the pgbench tests
> > pass either
Thomas Munro writes:
> Thanks. This seems to work for me on a Mac. I confirmed with nm that
> we don't define or reference any pthread_XXX symbols with
> --disable-thread-safety, and we do otherwise, and the pgbench tests
> pass either way.
Looks reasonable by eyeball. If you'd push it, I can
On Sat, Mar 13, 2021 at 4:09 PM Tom Lane wrote:
> OK, cool. I don't think it's hard, just do
>
> if test "$enable_thread_safety" = yes; then
> AC_REPLACE_FUNCS(pthread_barrier_wait)
> fi
>
> Probably this check should be likewise conditional:
>
> AC_SEARCH_LIBS(pthread_barrier_wait, pthread)
Thomas Munro writes:
> On Sat, Mar 13, 2021 at 3:47 PM Tom Lane wrote:
>> Was any thought given to being able to opt out of this patchset
>> to support that configure option?
> Oops. The pgbench code was tested under --disable-thread-safety, but
> it didn't occur to me that the
On Sat, Mar 13, 2021 at 3:47 PM Tom Lane wrote:
> Checking the man pages, it seems that this ancient HPUX version
> is using some pre-POSIX API spec in which pthread_cond_init takes a
> pthread_condattr_t rather than a pointer to pthread_condattr_t.
> Similarly for pthread_mutex_init.
Wow.
>
Thomas Munro writes:
> On Mon, Mar 8, 2021 at 3:18 PM Thomas Munro wrote:
>> David Rowley kindly tested this for me on Windows and told me how to
>> fix one of the macros that had incorrect error checking on that OS.
>> So here's a new version. I'm planning to commit 0001 and 0002 soon,
>> if
On Mon, Mar 8, 2021 at 3:18 PM Thomas Munro wrote:
> David Rowley kindly tested this for me on Windows and told me how to
> fix one of the macros that had incorrect error checking on that OS.
> So here's a new version. I'm planning to commit 0001 and 0002 soon,
> if there are no objections.
On Fri, Mar 5, 2021 at 6:22 PM Thomas Munro wrote:
> On Thu, Mar 4, 2021 at 10:44 PM Thomas Munro wrote:
> > v10-0002-pgbench-Refactor-the-way-we-do-thread-portabilit.patch
>
> Here's a better version of that part. I don't yet know if it actually
> works on Windows...
David Rowley kindly
On Thu, Mar 4, 2021 at 10:44 PM Thomas Munro wrote:
> v10-0002-pgbench-Refactor-the-way-we-do-thread-portabilit.patch
Here's a better version of that part. I don't yet know if it actually
works on Windows...
From 3aa63dfc086ab1f687ed668091a6bda8bf270fa7 Mon Sep 17 00:00:00 2001
From: Thomas
On Wed, Mar 3, 2021 at 6:23 PM Thomas Munro wrote:
> On Sun, Jan 31, 2021 at 1:18 AM Fabien COELHO wrote:
> > I think it would be much more consistent to move all the thread complement
> > stuff there directly: Currently (v8) the windows implementation is in
> > pgbench and the MacOS
On Sun, Jan 31, 2021 at 1:18 AM Fabien COELHO wrote:
> I think it would be much more consistent to move all the thread complement
> stuff there directly: Currently (v8) the windows implementation is in
> pgbench and the MacOS implementation in port, which is quite messy.
Hmm. Well this is
Hello Thomas,
3 . Decide if it's sane for the Windows-based emulation to be in here
too, or if it should stay in pgbench.c. Or alternatively, if we're
emulating pthread stuff on Windows, why not also put the other pthread
emulation stuff from pgbench.c into a "ports" file; that seems
On Sat, Jan 9, 2021 at 8:13 AM Thomas Munro wrote:
> On Sun, Jan 3, 2021 at 9:49 AM Fabien COELHO wrote:
> > > Just for fun, the attached 0002 patch is a quick prototype of a
> > > replacement for that stuff that seems to work OK on a Mac here. (I'm
> > > not sure if the Windows part makes
On Sun, Jan 3, 2021 at 9:49 AM Fabien COELHO wrote:
> > Just for fun, the attached 0002 patch is a quick prototype of a
> > replacement for that stuff that seems to work OK on a Mac here. (I'm
> > not sure if the Windows part makes sense or works.)
>
> Thanks! That will definitely help because I
It looks like macOS doesn't have pthread barriers (via cfbot 2021, now
with more operating systems):
Indeed:-(
I'll look into that.
Just for fun, the attached 0002 patch is a quick prototype of a
replacement for that stuff that seems to work OK on a Mac here. (I'm
not sure if the Windows
It looks like macOS doesn't have pthread barriers (via cfbot 2021, now
with more operating systems):
Indeed:-(
I'll look into that.
--
Fabien.
On Sun, Nov 15, 2020 at 4:53 AM Fabien COELHO wrote:
> In the attached version, I just comment out the call and add an
> explanation about why it is commented out. If pg overall version
> requirements are changed on windows, then it could be reinstated.
It looks like macOS doesn't have pthread
I think the issue really is that, independent of PG lock contention,
it'll take a while to establish all connections, and that starting to
benchmark with only some connections established will create pretty
pointless numbers.
Yes. This is why I think that if we have some way to synchronize
Hi,
On 2020-11-17 00:09:34 +0300, Marina Polyakova wrote:
> Sorry I'm not familiar with the internal architecture of snapshots, locks
> etc. in postgres, but I wanted to ask - what exact kind of patch for fair
> lwlocks do you want to offer to the community? I applied the 6-th version of
> the
Hi,
On 2020-11-14 20:07:38 +0300, Alexander Korotkov wrote:
> Hmm... Let's see the big picture. You've recently committed a
> patchset, which greatly improved the performance of GetSnapshotData().
> And you're making further improvements in this direction. But you're
> getting trouble in
Hello!
On 2020-11-14 20:07, Alexander Korotkov wrote:
Hmm... Let's see the big picture. You've recently committed a
patchset, which greatly improved the performance of GetSnapshotData().
And you're making further improvements in this direction. But you're
getting trouble in measuring the
Hi!
On Thu, Feb 27, 2020 at 9:01 PM Andres Freund wrote:
> I am trying to run a few benchmarks measuring the effects of patch to
> make GetSnapshotData() faster in the face of larger numbers of
> established connections.
>
> Before the patch connection establishment often is very slow due to
>
Hello Marina,
1) It looks like pgbench will no longer support Windows XP due to the
function DeleteSynchronizationBarrier. From
https://docs.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-deletesynchronizationbarrier
Minimum supported client: Windows 8 [desktop apps only]
Minimum
Hello!
On 2020-11-13 08:44, kuroda.hay...@fujitsu.com wrote:
Dear Fabien,
and this will wait till its time comes. In the mean time, I think that
you
should put the patch status as you see fit, independently of the other
patch: it seems unlikely that they would be committed together, and
ith me.
Hayato Kuroda
FUJITSU LIMITED
-Original Message-
From: Fabien COELHO
Sent: Wednesday, November 11, 2020 9:24 PM
To: Kuroda, Hayato/黒田 隼人
Cc: Andres Freund ; Daniel Gustafsson ;
pgsql-hack...@postgresql.org
Subject: RE: pgbench: option delaying queries till connections establishme
Hello,
I can remove the line, but I strongly believe that reporting performance
figures if some client connection failed thus the bench could not run as
prescribed is a bad behavior. The good news is that it is probably quite
unlikely. So I'd prefer to keep it and possibly submit a patch to
Dear Fabien,
> Usually I run many pgbench through scripts, so I'm probably not there to
> check a lone stderr failure at the beginning if performance figures are
> actually reported.
> I can remove the line, but I strongly believe that reporting performance
> figures if some client connection
Hello,
Indeed. I took your next patch with an added explanation. I'm unclear
whether proceeding makes much sense though, that is some thread would run
the test and other would have aborted. Hmmm.
Your comment looks good, thanks. In the previous version pgbench starts
benchmarking even if
Dear Fabien,
> Indeed. I scanned the file but did not find other places that needed
> updating.
> Yes.
> Not sure either. I'm not for having too many braces anyway, so I removed
> them.
I checked your fixes and I think it's OK.
Finally, please move the doc fixes to patch B in order to
Hello,
Please complete fixes for the documentation. At least the following sentence
should be fixed:
```
The last two lines report the number of transactions per second, figured with
and without counting the time to start database sessions.
```
Indeed. I scanned the file but did not find
Dear Fabien;
> The current implementation is too simple. If nthreads >= 2 and connection
> fails in the one thread,
> the other one will wait forever.
I attached the very preliminary patch for solving the problem.
Even if threads fail to connect, the others can go through the barrier.
But I
sql.org
Subject: Re: pgbench: option delaying queries till connections establishment?
>> * First patch reworks time measurements in pgbench.
>> * Second patch adds a barrier before starting the bench
>>
>> It applies on top of the previous one. The initial imbalance du
* First patch reworks time measurements in pgbench.
* Second patch adds a barrier before starting the bench
It applies on top of the previous one. The initial imbalance due to
thread creation times is smoothed.
The usecs patch fails to apply to HEAD, can you please submit a rebased version
> On 17 May 2020, at 11:55, Fabien COELHO wrote:
> I have split the patch.
>
> * First patch reworks time measurements in pgbench.
> * Second patch adds a barrier before starting the bench
>
> It applies on top of the previous one. The initial imbalance due to thread
> creation times is
Hello,
I've merged all time-related stuff (time_t, instr_time, int64) to use a
unique type (pg_time_usec_t) and set of functions/macros, which simplifies
the code somehow.
Hm. I'm not convinced it's a good idea for pgbench to do its own thing
here.
I really think that the refactoring part
Hallo Andres,
Slight aside: Have you ever looked at moving pgbench to non-blocking
connection establishment? It seems weird to use non-blocking everywhere
but connection establishment.
Attached an attempt at doing that, mostly done for fun. It seems to be a
little slower on a local socket.
Hello Andres,
I've changed the performance calculations depending on -C or not. Ramp-up
effects are smoothed.
I've merged all time-related stuff (time_t, instr_time, int64) to use a
unique type (pg_time_usec_t) and set of functions/macros, which simplifies
the code somehow.
Hm. I'm not
Hello Andres,
Slight aside: Have you ever looked at moving pgbench to non-blocking
connection establishment? It seems weird to use non-blocking everywhere
but connection establishment.
Nope. If there is some interest, why not. The reason for not doing it is
that the typical use case is just
Hi,
On 2020-03-01 22:16:06 +0100, Fabien COELHO wrote:
>
> Hello Andres,
>
> > FWIW, leaving windows, error handling, and other annoyances aside, this
> > can be implemented fairly simply. See below.
>
> Attached an attempt at improving things.
Awesome!
> I've put 2 barriers: one so that all
Hello Andres,
FWIW, leaving windows, error handling, and other annoyances aside, this
can be implemented fairly simply. See below.
Attached an attempt at improving things.
I've put 2 barriers: one so that all threads are up, one when all
connections are setup and the bench is ready to go.
Hi,
On 2020-02-29 15:29:19 +0100, Fabien COELHO wrote:
> Pgbench is more or less designed to run a long hopefully steady-state
> benchmark, so that the initial connection setup is always negligeable. Not
> complying with this hypothesis quite often leads to weird results.
I don't think this is a
Hello Kyotaro-san,
I think this also shows that "including/excluding connections
establishing" as well as some of the other stats reported pretty
bogus. In the 'before' case a substantial numer of the connections had
not yet been established until the end of the test run!
I see it useful.
Hello Andres,
Therefore I'd like to make pgbench wait till it has established all
connections, before they run queries.
Does anybody else see this as being useful?
Yes, I think that having this behavior available would make sense.
If so, should this be done unconditionally?
Dunno. I
At Thu, 27 Feb 2020 10:51:29 -0800, Andres Freund wrote in
> Hi,
>
> On 2020-02-27 10:01:00 -0800, Andres Freund wrote:
> > If so, should this be done unconditionally? A new option? Included in an
> > existing one somehow?
>
> FWIW, leaving windows, error handling, and other annoyances aside,
Hi,
On 2020-02-27 10:01:00 -0800, Andres Freund wrote:
> If so, should this be done unconditionally? A new option? Included in an
> existing one somehow?
FWIW, leaving windows, error handling, and other annoyances aside, this
can be implemented fairly simply. See below.
As an example of the
Hi,
I am trying to run a few benchmarks measuring the effects of patch to
make GetSnapshotData() faster in the face of larger numbers of
established connections.
Before the patch connection establishment often is very slow due to
contention. The first few connections are fast, but after that it
53 matches
Mail list logo
