Re: Thread deadlock in CacheEventQueue class

[Matthew Cooke]

Aaron,
As far as I know the deadlock was not recoverable and did not occur when 
iterating the local memory cache but rather on put operations triggered 
by a master/remotecache update.
The problem once it occurred stopped all updates occurring I think it 
was blocking the local puts triggered from the mastercache by holding a 
lock.
I agree automated tests are difficult, I tried, it required high load 
from a stress tool difficult to setup in an automated test.

On an unrelated note, we tried the threadpool timeout code 1.1.3dev 
yesterday. It seemed to work in testing but once we staged to production 
it turned out that deletions were no longer getting propagated from the 
remote/mastercache resulting in content never updating on the client 
machines. I tried various config combinations, with threadpool, without 
threadpool and with/without removeOnPut enabled. I got the same error in 
every case on the mastercache (failure to enqueue remove events) and had 
to roll back to the version prior to thread pools. Unfortunately during 
testing we had only checked the initial items were received, we never 
tested items were updated - so this was partially my own fault.
Having retested the code now on test servers i definitely can't get the 
remove events working in the 1.1.3dev build.

If you haven't tested that deletion are propagating with the new 
threadpool code i suggest you do so, if you have tested this could you 
send me a copy of your working cache.ccf and remote.cache.ccf in case i 
missed something.

Many Thanks,
Matt.
Aaron Smuts wrote:
I suspect that it was the remove / put problem that I solved in 1.1.2.
It was the only known problem of that sort.  It may be due to iteration
but I thought this was resolved.  Iteration is very problematic on an
active collection.  There is no really good way to do it.  We basically
copy the keys in fail fast mode.  Were you seeing unrecoverable deadlock
or just temporary lags?
I'll try some more strenuous testing.  The big problem is that it is
very difficult to create automated tests for the remote and lateral
caches.  I mainly rely on the tester script.  I have a random function
that will beat the hell out of the cache. . . . .  

Aaron

-----Original Message-----
From: Matthew Cooke [mailto:[EMAIL PROTECTED]
Sent: Tuesday, January 11, 2005 8:52 AM
To: turbine-jcs-user@jakarta.apache.org
Subject: RE: Thread deadlock in CacheEventQueue class
We had major problems when we used the remotecache configuration in
push
mode rather than delete mode (removeOnPut).
At the time we strongly suspected a deadlock caused by the local puts
placing cached items back into the local memory caches when they
recieved async put events from the remotecache.
Could the deadlock we were seeing when using remotePuts have been the
same thing? We are quite keen to try remotePuts again as this
functionality would be very useful to us (would allow iteration over
whole caches on individual machines). As the deadlock was only
occuring
under high load we don't have a reliable way to test.
Any advice appreciated!
---------------------------------------------------------------------
To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]