I ran the new Pgpool-1.2.2 and it was a bit faster on
the TCP but still slower than on UNIX socket. I used
the same script as before.
TCP Socket (Pgpool 1.2.0)
--
2.39 sec
TCP Socket (Pgpool 1.2.2)
--
0.80 sec
0.80 sec
0.79 sec
UNIX Socket (Pgpool 1.2.2)
I ran the new Pgpool-1.2.2 and it was a bit faster on
the TCP but still slower than on UNIX socket. I used
the same script as before.
TCP Socket (Pgpool 1.2.0)
--
2.39 sec
TCP Socket (Pgpool 1.2.2)
--
0.80 sec
0.80 sec
0.79 sec
UNIX Socket (Pgpool 1.2.2)
---
0.026 sec
Sorry for the confusion because Rod asked a question
and I answered too quickly. This is what I mean.
15x Slower:
---
Client --TCP-- PgPool --UNIX-- PostgreSQL
Client --TCP-- PgPool --TCP-- PostgreSQL
5x Faster:
--
Client --UNIX-- PgPool --UNIX-- PostgreSQL
Hi Bruce,
Sorry for the confusion because Rod asked a question
and I answered too quickly. This is what I mean.
15x Slower:
---
Client --TCP-- PgPool --UNIX-- PostgreSQL
Client --TCP-- PgPool --TCP-- PostgreSQL
5x Faster:
--
Client --UNIX-- PgPool --UNIX--
Yes, I realize it's a bit old but I just wanted to
make a small point that forking is slower. It's funny
you should ask because thread creation on Linux has in
fact improved over process creation much more in 2.4
kernel.
Benchmark at IBM shows Linux 2.4 thread creation is
30x faster than process
Tatsuo,
I did some benchmark on my Linux box (AMD 1.2Ghz,
256MB, Fedora Core 1 Linux 2.4.20-8) using Pgpool 1.2
and PostgreSQL 7.4. I ran the benchmark script
repeatedly (10+ times each).
I get 5x faster using Pgpool on UNIX socket, which is
encouraging. This shows pre-fork does speed things up.
Ok, I did some benchmark on my Linux box (AMD 1.2Ghz,
256MB, Fedora Core 1 Linux 2.4.20-8) using Pgpool 1.2
and PostgreSQL 7.4. I ran the benchmark script
repeatedly (10+ times each).
I get 5x faster using Pgpool on UNIX socket, which is
encouraging. This shows pre-fork does speed things up
even
sdv == sdv mailer [EMAIL PROTECTED] writes:
sdv Yes, I realize it's a bit old but I just wanted to make a
sdv small point that forking is slower. It's funny you should ask
sdv because thread creation on Linux has in fact improved over
sdv process creation much more in 2.4 kernel.
However, when I tried TCP socket, Pgpool was actually
slower by 15x !! Perhaps you can clarify why the TCP
socket is so much slower?
How did you have pgpool configured to connect to the database? Domain
socket or tcpip?
---(end of broadcast)---
Pgpool connects to PostgreSQL on UNIX socket. I also
ran on TCP socket but there is no significant
difference if I recall correctly due to the inherent
nature of connection pooling or pre-fork technology.
;-)
--- Rod Taylor [EMAIL PROTECTED] wrote:
However, when I tried TCP socket, Pgpool was
sdv mailer wrote:
Pgpool connects to PostgreSQL on UNIX socket. I also
ran on TCP socket but there is no significant
difference if I recall correctly due to the inherent
nature of connection pooling or pre-fork technology.
;-)
I am confused. First you said TCP was slower, but now you say
Hi Bruce,
Sorry for the confusion because Rod asked a question
and I answered too quickly. This is what I mean.
15x Slower:
---
Client --TCP-- PgPool --UNIX-- PostgreSQL
Client --TCP-- PgPool --TCP-- PostgreSQL
5x Faster:
--
Client --UNIX-- PgPool --UNIX-- PostgreSQL
Client
sdv mailer wrote:
Hi Bruce,
Sorry for the confusion because Rod asked a question
and I answered too quickly. This is what I mean.
15x Slower:
---
Client --TCP-- PgPool --UNIX-- PostgreSQL
Client --TCP-- PgPool --TCP-- PostgreSQL
5x Faster:
--
Client --UNIX-- PgPool
sdv mailer wrote:
The speedup (UNIX) and slowdown (TCP) are both
compared against normal direct connections from Client
to PostgreSQL. This means with Pgpool (UNIX) it is 5x
faster than normal connections to PostgreSQL. It is
also 15x slower with Pgpool (TCP) compared to normal
connections
sdv mailer wrote:
Hi Bruce,
Sorry for the confusion because Rod asked a question
and I answered too quickly. This is what I mean.
15x Slower:
---
Client --TCP-- PgPool --UNIX-- PostgreSQL
Client --TCP-- PgPool --TCP-- PostgreSQL
5x Faster:
--
Client --UNIX-- PgPool --UNIX--
No SSL. No authentication either. Just friendly
handshakes.
__
Do you Yahoo!?
Win a $20,000 Career Makeover at Yahoo! HotJobs
http://hotjobs.sweepstakes.yahoo.com/careermakeover
---(end of
On May 6, 2004, at 12:19 PM, sdv mailer wrote:
15x Slower:
---
Client --TCP-- PgPool --UNIX-- PostgreSQL
Client --TCP-- PgPool --TCP-- PostgreSQL
5x Faster:
--
Client --UNIX-- PgPool --UNIX-- PostgreSQL
Client --UNIX-- PgPool --TCP-- PostgreSQL
If the problem were in the TCP stack
I compared against both TCP and UNIX direct
connections. No SSL, no authentication. See benchmark
results posted below again:
Direct
--
0.144 sec. Client --UNIX-- PG
0.152 sec. Client --TCP-- PG
5x Faster
-
0.027 sec. Client --UNIX-- Pgpool --UNIX-- PG
0.028 sec. Client --UNIX--
Nope. I commented out that block of code at 372 and no
difference.
__
Do you Yahoo!?
Win a $20,000 Career Makeover at Yahoo! HotJobs
http://hotjobs.sweepstakes.yahoo.com/careermakeover
---(end of
Quick overview of the code for differences in TCP-on-the-frontend code
is a call to setsockopt(..., TCP_NODELAY, ...) if the connection to the
frontend is a TCP socket. Could this be producing pseudo-fragmentation,
resulting in over-the-top context switches? Looks like
pool_process_query()
sdv mailer [EMAIL PROTECTED] writes:
The point is pre-forking can *potentially* speed up
connections by 5x as shown in this simplistic
non-conclusive benchmark.
I think this benchmark proves no such thing.
The thing that pgpool is doing is not preforking connections at all, but
re-using prior
James Robinson [EMAIL PROTECTED] writes:
Quick overview of the code for differences in TCP-on-the-frontend code
is a call to setsockopt(..., TCP_NODELAY, ...) if the connection to the
frontend is a TCP socket. Could this be producing pseudo-fragmentation,
resulting in over-the-top context
Tom,
You're correct about the test measuring a hot backend
and not forking. How much exactly is the bulk of the
startup done by cache initialization relative to the
forking? What would be the impact on Win32 knowing
that process creation is twice as slow than on Linux?
On May 6, 2004, at 1:06 PM, sdv mailer wrote:
I compared against both TCP and UNIX direct
connections. No SSL, no authentication. See benchmark
results posted below again:
I recall your script only connected 20 times - that is not enough to
filter out noise in those numbers. Please run it again,
Tom Lane writes:
... too much flushing ...
I agree. I'll bet replacing the pool_write_and_flush() calls in
BinaryRow()
and AsciiRow() with just pool_write(), followed by removing the
fflush() calls at the bottom of those two methods should go a long
way towards fixing things, since the
On Thu, 6 May 2004, Tom Lane wrote:
sdv mailer [EMAIL PROTECTED] writes:
The point is pre-forking can *potentially* speed up
connections by 5x as shown in this simplistic
non-conclusive benchmark.
I think this benchmark proves no such thing.
The thing that pgpool is doing is not
I don't think I can volunteer on this end as I am
already actively volunteering for another open
project. I was hoping someone could take up on this
since one of the last threads mentionned we don't have
something substantial to present for 7.5 if June 1 is
dateline for code freeze. Pre-fork came
sdv mailer said:
Forking is expensive on many systems. Linux is a bit
better but still expensive compared to threads. On
Windows, creating process is much more expensive than
on Linux. Check this benchmark:
http://cs.nmu.edu/~randy/Research/Papers/Scheduler/understanding.html
Forking
sdv mailer wrote:
We used to run persistent connection until the DB
servers got maxed out because of too many idle
connections sucking up all the memory. Web servers run
different loads than database servers and persistent
connections are notorious for crashing your DB.
And this translates
And preforking makes this different, how ? Perhaps having a pool of
processes ready to be handed a query to a specific database, where you
configure N connections to db1, M to db2 etc. still means lots of resource
usage. In effect a preforked database server *is* an idle connection, just
On Mon, May 03, 2004 at 11:59:45PM -0700, sdv mailer wrote:
Connection pooling (eg. SQLRelay) didn't work either
because we needed to connect to hundreds of DB servers
from each web server. Imagine having 200+ open
connections on the web server and how many more of
these connections remain
I know the issue of pre-fork PostgreSQL has been discussed previously.
Someone mentionned pre-fork can be implemented when schemas become available
Any chance of that happening for 7.5?
0 chance unless you have a patch ready now.
---(end of
I'm talking about connecting to multiple database
servers on separate machines. Schemas don't apply
here.
How much work would it take to make a pre-fork smart
enough to open different databases on incoming
connection? How much of it can be modeled after
Apache?
Pre-fork does not equal to idle connections! Pre-fork
scales with database load where as persistent
connections scales with webserver load. A web server
that is heavily loaded but not necessarily performing
a lot of database activity will spawn hundreds of idle
database connections using
Rod Taylor [EMAIL PROTECTED] writes:
Cutting that count down to 10 idlers in total by having PostgreSQL
prefork a specific database would make a significant difference.
Well it would be 10 for each database. Since as has been pointed out before
loading the database is most of the delay.
If
Or, you run several seperate Apache webservers. The ones that serve static
content or don't need database connections don't run with the ones that do.
And just like each idle Apache process uses memory and other resources,
each idle PostgreSQL connection does to. So managing the number of
sdv mailer wrote:
I'm talking about connecting to multiple database
servers on separate machines. Schemas don't apply
here.
How much work would it take to make a pre-fork smart
enough to open different databases on incoming
connection? How much of it can be modeled after
Apache?
I've not used it
On Wed, 2004-05-05 at 11:57, Greg Stark wrote:
Rod Taylor [EMAIL PROTECTED] writes:
Cutting that count down to 10 idlers in total by having PostgreSQL
prefork a specific database would make a significant difference.
Well it would be 10 for each database. Since as has been pointed out
sdv mailer wrote:
[snip]
Pre-fork will give MySQL one less argument to throw at
PostgreSQL.
I think optimizing is this area will speed up the
general case for everyone rather than optimizing a
feature that affects 10% of the users. On top of that,
it will make a strong marketing case because
On Wednesday 05 May 2004 07:24 am, Rod Taylor wrote:
And preforking makes this different, how ? Perhaps having a pool of
processes ready to be handed a query to a specific database, where you
configure N connections to db1, M to db2 etc. still means lots of
resource usage. In effect a
On Wed, May 05, 2004 at 02:12:58PM -0400, Rod Taylor wrote:
Most of it has been. It's the duty cycle. As stated in another email,
only about 20% of the work a script does is database related -- which
occurs all at one time. Even when all Apache backends are active, a
large number of
On Wed, 5 May 2004, sdv mailer wrote:
Forking is quite fast on Linux but creating a new
process is still 10x more expensive than creating a
thread and is even worse on Win32 platform. CPU load
goes up because the OS needs to allocate/deallocate
memory making it difficult to get a steady
* Steve Atkins ([EMAIL PROTECTED]) wrote:
On Wed, May 05, 2004 at 02:12:58PM -0400, Rod Taylor wrote:
Most of it has been. It's the duty cycle. As stated in another email,
only about 20% of the work a script does is database related -- which
occurs all at one time. Even when all Apache
The fact that windows has a heavy process / lightweight thread design
means little to me, since I'll likely never deploy a production postgresql
server on it that needs to handle any serious load.
Yes but Solaris also has a heavy process / lightweight thread design.
J
And, of course, most development environments (perl, php, java etc)
have their own language specific connection pooling solutions.
Yes, the one for php is what I was thinking of when I made my statement.
They work on a per backend basis as Apache does not allow for the type
of communication
On Wed, 5 May 2004, Rod Taylor wrote:
And, of course, most development environments (perl, php, java etc)
have their own language specific connection pooling solutions.
Yes, the one for php is what I was thinking of when I made my statement.
They work on a per backend basis as Apache does
I cannot tell if mod_pg_pool works across Apache forked backends or is
still bound to a single process. They state it is intended for sharing
connections across modules, so it is probably still backend specific.
Have you looked at sqlrealy.sourceforge.net? IT looks like it might do
Rod Taylor wrote:
As a temporary step these types of things help. But it's still doesn't
really fix the problem of Apache not using real connection pooling.
Rod,
In principle, this should not be enormously hard to do - at least for
Unix where the methods of handing off file handles between
I've already tried pooling (SQLRelay) and persistent
connection (PHP). They may work for other people but
they do not work for me. I have already separated
static from database driven codes but you can never
balance web server load with database server load.
Pre-fork scales with database load and
Forking is expensive on many systems. Linux is a bit
better but still expensive compared to threads. On
Windows, creating process is much more expensive than
on Linux. Check this benchmark:
http://cs.nmu.edu/~randy/Research/Papers/Scheduler/understanding.html
Forking shouldn't be taken lightly
I'll pretend I didn't see that last comment on
Windows. I wouldn't want to disappoint the users who
are eagerly expecting the Win32 port to complete
including myself. ;-)
Having said that, I think it's more the reason to get
a working pre-fork for Win32. Don't you think so?
--- scott.marlowe
We used to run persistent connection until the DB
servers got maxed out because of too many idle
connections sucking up all the memory. Web servers run
different loads than database servers and persistent
connections are notorious for crashing your DB.
Connection pooling (eg. SQLRelay) didn't
sdv mailer wrote:
Instead, there's a big need to
create a new connection on
every query and with PostgreSQL needing to fork on
every incoming connection
can be quite slow.
Really? My general experience has beent that forking/connection setup
times are very good with PgSQL. Do not assume your
Paul Ramsey [EMAIL PROTECTED] writes:
... So the operational benefit of adding the complexity of a
pre-fork system is not very high.
In particular, most of the connection startup overhead work cannot be
performed until we've identified which database to connect to (since
it largely consists of
Forking consumes a large amount of CPU when you have
many simultaneous connections and adds up to the
latency. Particularly MySQL users may think
PostgreSQL's connection time is much slower because
these users tend to perform relatively simple queries.
In my case, connection pooling and
sdv mailer [EMAIL PROTECTED] writes:
Forking consumes a large amount of CPU when you have
many simultaneous connections and adds up to the
latency. Particularly MySQL users may think
PostgreSQL's connection time is much slower because
these users tend to perform relatively simple queries.
56 matches
Mail list logo
