On 12/12/09 3:54 PM, Michael Gentry wrote:
I created a brand new DC in each thread, got the system time, ran the
query (and possibly committed a change), then got the system time
again to compute the difference.
OK. So you even though you aren't directly counting the time spent creating
contexts, since your test is threaded you are implicitly counting that time.
Say (I don't know) that one ORM locks around creating new contexts, then you
are incorporating that into your timing results.
The problem with a test like this is that it doesn't always reproduce how this
works in real life. Would you be creating lots of new contexts concurrently in
your real application?
So the numbers are the time it takes
the thread to complete, for the most part (I kick them off at the same
time -- at least as closely as I can).
You should be recording and reporting the start and end times of each thread in
order to understand if locking is causing some to wait. If you graphed them on
a little gantt style chart, then that would become apparent. If you've ever
used Firebug's network analysis, you'll know what I mean. When did the thread
start? When did you get a new context returned? How long did the fetch take?
Then the commit?
My guess is it is
locking/concurrency, but I didn't have time to delve into it any
deeper (was the end of the day). When I ran RetrieveIndividual20 with
5 threads instead of 20 the results were MUCH closer to the Hibernate
values (sometimes better). (Keep in mind both ORMs had a limit of 5
open DB channels.)
There are so many things going on here, it is hard to isolate the differences.
Perhaps you need to fire up a profiler and see where it is spending most of its
time. Is there a big impact going from equal or less than the number of db
channels to one more?
* There is so much variance between runs that something is odd here. What
are you doing between runs to clear the cache? How are you dealing with
variability at the database? Is the database running on another server which
is unloaded and isn't itself interfering with the results?
I ran them several times and each time gave similar numbers. The ones
in the spreadsheet are just from the last run I did, but they were all
similar from run-to-run.
Ah, I thought these were the results of lots of runs. But you are showing the
time from each thread in a single run. In that case the wild variance is
definitely the result of come sort of locking. Can you compare CPU usage
between the two ORMs? That would give you somewhat of an idea whether the time
is spent in a lock or doing something.
The DB was on a different server, which is
also another reason I did several runs. I expect a few variances, but
the general pattern was the same. I don't have the QueryLogger
results here, but the DB times were significantly faster, so it really
was on the local Java side.
* The huge variability makes me think that memory pressures might be
important. Are you seeing the JVM garbage collect during the middle of some
runs? Can you run jstat and see if that is interfering.
Didn't have time to get into GC. The machine I was on had about 5 GB
of RAM free and I had allocated plenty to the JVM. Also, fetching and
updating 20 records in the DB shouldn't be that memory intensive.
That isn't really the point. No matter how much RAM you have Java may perform
GC, depending on your test application, how much stack/heap you gave to the
JVM, how long the application has been running, etc.
* Are you able to pick the worst case (update) and time the various parts:
create context, database UPDATE, commit finished... In your tests are you
timing creating the context or is that outside the timing loop?
I was creating the DC outside of the timing.
Sort of, but not really. If you want to eliminate that from your measurement,
you need to spawn 20 threads, have them all create contexts, fetch the data and
then sleep. Then all at once you can wake them to perform an update
concurrently and you'll be measuring only the update.
None of this is much use to tell your team how well Cayenne might work in your
application in the real world since the tests don't really resemble how most
ORMs are used. However if you are trying to isolate a specific thing to
measure, it might be useful.
Regards
Ari Maniatis
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
