We've had more than one report of windows users running into
performance issues with the way we manage sockets between the client
and server. Typically they run out of ports, etc. A bit back Dain
got The Grinder[1] running and showed me how good the performance was
when we dropped in Jetty and ran our ejbd protocol over http. On my
machine with a simple ejb, 2 processes, 10 threads each (i.e. 20
threads totoal), I was getting about 2500 TPS (transactions per
second) over jetty+http+ejbd. That was several times our raw ejbd
performance which would start at around 600-700 TPS and taper off
after a few hours to next to nothing.
I looked into how much of the Jetty code we'd need to run that setup
all the time and it was around 240K so I decided to give improving our
raw socket + ejbd setup another go and read up on NIO and all that
business. I added some keep-alive logic, buffering, eliminated some
flushing, and introduced another thread to check for and close
connections being kept alive in the "waiting for request" state too
long (3 second cut off currently).
Here's the performance we're getting now:
http://people.apache.org/~dblevins/ejbd-client-performance.png
The number to look at is the mean, which is 7300 TPS. I kicked this
off yesterday and it's been running pretty consistently at that number
for about 21 hours now. It actually starts at about 7200 TPS and goes
up just slightly, perhaps because of vm hotspotting, though not too
sure.
Anyway this should eliminate any performance issues someone might have
seen before with heavily active remote clients. It's not yet using
NIO. I'd like to try it out, but might keep that for next release.
There are still good performance related reasons to use the ejbd over
http/jetty setup. Here's the basic trade off:
- ejbd over raw socket:
- high TPS with lower number of active clients (hundred or so vms)
- low TPS with larger number of active clients (more than a
hundred or so)
- ejbd over http/jetty:
- low TPS with lower number of active clients (less than a hundred)
- high TPS with larger number of active clients (hundreds or
thousands)
Note that client doesn't mean threads, it means actual VMs. You can
have several client threads going with pretty much no impact.
-David
[1] http://grinder.sourceforge.net/
