Thanks Chuck that gave me some helpful pointers to look at. Looking thru
all of the thread dumps I found one thread that was slightly different than
the others. I am not sure why a cooperating object store is being added.
"http-10042-Processor46" nid=59971 state=WAITING
- waiting on <0xcb1792> (a com.webobjects.foundation.NSRecursiveLock)
- locked <0xcb1792> (a com.webobjects.foundation.NSRecursiveLock)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at
com.webobjects.foundation.NSRecursiveLock.lock(NSRecursiveLock.java:72)
at
com.webobjects.eoaccess.EODatabaseContext.lock(EODatabaseContext.java:1973)
at
com.webobjects.eocontrol.EOObjectStoreCoordinator.addCooperatingObjectStore(
EOObjectStoreCoordinator.java:130)
at
com.webobjects.eoaccess.EODatabaseChannel.setCurrentEditingContext(EODatabas
eChannel.java:166)
at
com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditi
ngContext(EODatabaseChannel.java:788)
at
com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecificatio
n(EODatabaseChannel.java:215)
at
com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEdit
ingContext(EODatabaseContext.java:3205)
at
com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification(EODa
tabaseContext.java:3346)
at
com.webobjects.eocontrol.EOObjectStoreCoordinator.objectsWithFetchSpecificat
ion(EOObjectStoreCoordinator.java:539)
at
com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEd
itingContext.java:4114)
at
com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEd
itingContext.java:4500)
at
com.webobjects.eoaccess.EOUtilities.objectsWithQualifierFormat(EOUtilities.j
ava:145)
I also found this line of code I don¹t think it is being called at this
time, but it seems like perhaps this should have a try/catch/finally block
and a lock()/unlock(). Thoughts?
EOObjectStoreCoordinator.defaultCoordinator().invalidateAllObjects();
I couldn¹t find any reference to any locking bugs being fixed for 5.4.2.
Dov Rosenberg
On 8/27/08 7:46 PM, "Chuck Hill" <[EMAIL PROTECTED]> wrote:
> Hi Dov,
>
>
> On Aug 27, 2008, at 4:18 PM, Dov Rosenberg wrote:
>
>> We have a WO (5.4.1)
>
> I'd really very seriously think about moving to 5.4.2 to see if this helps.
>
>
>> app that is deployed as a servlet in Tomcat 5.5 (Java 1.5). We do not use
>> Project Wonder or multiple ObjectStoreCoordinators. We have experienced
>> intermittent hanging issues under load. When we look at the thread dumps I
>> always see things like
>>
>> "http-10042-Processor111" nid=60350 state=WAITING
>> - waiting on <0xcb1792> (a com.webobjects.foundation.NSRecursiveLock)
>> - locked <0xcb1792> (a com.webobjects.foundation.NSRecursiveLock)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Unknown Source)
>> at
>> com.webobjects.foundation.NSRecursiveLock.lock(NSRecursiveLock.java:72)
>> at
>> com.webobjects.eocontrol.EOObjectStoreCoordinator.lock(EOObjectStoreCoordinat
>> or.java:466)
>> at
>> com.webobjects.eocontrol.EOEditingContext.lockObjectStore(EOEditingContext.ja
>> va:4735)
>> at
>> com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEdi
>> tingContext.java:4112)
>> at
>> com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEdi
>> tingContext.java:4500)
>> at
>> com.webobjects.eoaccess.EOUtilities.objectsMatchingValues(EOUtilities.java:19
>> 3)
>> at
>> com.webobjects.eoaccess.EOUtilities.objectsMatchingKeyAndValue(EOUtilities.ja
>> va:168)
>> ...
>
> Those threads are symptoms, not the problem. The problem is that there is a
> hanging lock on EOObjectStoreCoordinator.
>
>
>> It seems the root of every thread always has NSRecursiveLock.lock() as part
>> of the thread dump. It doesn¹t seem to matter if the call was via EOUtilities
>> or thru a fetch.
>
> The real key line in the trace is
>
> at
> com.webobjects.eocontrol.EOObjectStoreCoordinator.lock(EOObjectStoreCoordinato
> r.java:466)
>
>
>> In a recent thread dump there were 286 threads listed of which 251 (all HTTP
>> threads) had references to NSRecursiveLock.lock().
>
> You might want to reduce the number of threads and listeners you create so you
> find the problem sooner. The impact on your users may be less or at least
> less frustrating.
>
> A more interesting thing is what did the other threads show?
>
>
>
>> All of the threads were marked as state=WAITING (none were RUNNABLE). It
>> seems that all of the threads were waiting for something and thus could not
>> do anything but there was no Java deadlocks being thrown.
>
> Yes, they were waiting for some thread to unlock the OSC.
>
>
>> Questions:
>>
>> * Does this indicate that we have an issue with multiple threads within the
>> same JVM? We have ConcurrentRequest handling turned on.
> No.
>> *
>> * Should we investigate using multiple ObjectStoreCoordinators?
>
> This might increase the time before the entire instance goes dead, but it
> won't address the root problem.
>
>
>> *
>> * Is this a red herring and should I look elsewhere for the problem?
>> *
>
> You have found the symptom, now you need to find the problem:
>
>
> * app running out of memory and not unlocking the OSC
> * your code locking the OSC and not unlocking it in a finally block
> * bug in WO 5.4.1 (hypothetical, I don't know of on) locking the OSC and not
> unlocking it in a finally block
> * deadlock or very long running transaction at the database level that is
> preventing and EOF operation from completing in a reasonable amount of time.
>
>
> Chuck
>
>
>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list ([email protected])
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com
This email sent to [EMAIL PROTECTED]
