On Jul 13, 2008, at 3:03 PM, Manu George wrote:
Pretty cool stuff Dave. Why is the performance better for high TPS for
ejbd over http/jetty?
Don't know why we're 3x faster on the 2 client + 20 threads side of
things. It might simply be that we have an advantage in that we can
completely control the client side. I saw the biggest boost in
performance when I made the client buffer up nearly all it's data
before writing any of it. I also made it so the client can have at
most one open connection per server URL and all requests file through
it.
For the record you can actually get more connections per client by
slightly appending to your server url as so:
ejbd://localhost:4201/a
ejbd://localhost:4201/b
Everything after the "/" is currently ignored, but will allow you to
get another connection to the same server if you think you really need
it. I actually saw pretty great performance with just one connection
shared among 10 client threads, but maybe you want a second connection
open for very long running requests as to not block up the shorter
requests.
As the no of clients increases does the TPS
decrease for ejbd over raw socket?
We're still using regular java io so we have to dedicate a thread to
each open connection. So we're limited by the number of threads that
can realistically be running without too much context switching. On a
dual core you can run quite a few, on quad core you can run an
impressive number, on an Azul box ... sky is the limit practically.
However if you do exceed your set thread pool size, new clients are
simply not going to get a connection till someone else gives theirs up
or it times out after 3 seconds of inactivity (the timeout will be
configurable). Works great if you have dozens clients (or more if you
have more cores or CPUs) who just want to wail on the server
continuously.
Jetty uses NIO and can have one thread dedicated to reading data from
open connections (using NIO Selectors). It'll buffer up data till
there's enough for a whole request *then* it will send it into the
thread pool for work. So the number of clients you can have is not
strictly tied to the number of threads. So if you had a huge burst in
client count and requests it could likely handle that just fine.
Works great if the number of clients can fluctuate erratically from a
few dozen to several thousand on a regular basis.
-David
On Mon, Jul 14, 2008 at 3:18 AM, David Blevins
<[EMAIL PROTECTED]> wrote:
We've had more than one report of windows users running into
performance
issues with the way we manage sockets between the client and server.
Typically they run out of ports, etc. A bit back Dain got The
Grinder[1]
running and showed me how good the performance was when we dropped
in Jetty
and ran our ejbd protocol over http. On my machine with a simple
ejb, 2
processes, 10 threads each (i.e. 20 threads totoal), I was getting
about
2500 TPS (transactions per second) over jetty+http+ejbd. That was
several
times our raw ejbd performance which would start at around 600-700
TPS and
taper off after a few hours to next to nothing.
I looked into how much of the Jetty code we'd need to run that
setup all the
time and it was around 240K so I decided to give improving our raw
socket +
ejbd setup another go and read up on NIO and all that business. I
added
some keep-alive logic, buffering, eliminated some flushing, and
introduced
another thread to check for and close connections being kept alive
in the
"waiting for request" state too long (3 second cut off currently).
Here's the performance we're getting now:
http://people.apache.org/~dblevins/ejbd-client-performance.png
The number to look at is the mean, which is 7300 TPS. I kicked
this off
yesterday and it's been running pretty consistently at that number
for about
21 hours now. It actually starts at about 7200 TPS and goes up just
slightly, perhaps because of vm hotspotting, though not too sure.
Anyway this should eliminate any performance issues someone might
have seen
before with heavily active remote clients. It's not yet using
NIO. I'd
like to try it out, but might keep that for next release.
There are still good performance related reasons to use the ejbd over
http/jetty setup. Here's the basic trade off:
- ejbd over raw socket:
- high TPS with lower number of active clients (hundred or so vms)
- low TPS with larger number of active clients (more than a
hundred or so)
- ejbd over http/jetty:
- low TPS with lower number of active clients (less than a hundred)
- high TPS with larger number of active clients (hundreds or
thousands)
Note that client doesn't mean threads, it means actual VMs. You
can have
several client threads going with pretty much no impact.
-David
[1] http://grinder.sourceforge.net/
