All
I believe I have found a defect with XmlBeans which can cause a deadlock
to occur under a specific situation.
The situation can occur when calling a setter on an Object which
synchronizes on Locale A, to set an Object which synchronizes on Locale
B at the same time as calling a setter on an Object which synchronizes
on Locale B to set an Object which synchronizes on Locale A.
The code which XmlBeans generates follows this flow:
1. MyObjectUnderLocaleA.setMyElement(MyObjectUnderLocaleB)
a) Synchronize on monitor()
b) Delegates to XmlObjectBase.set(XmlObject src)
i.
Identifies that MyObjectUnderLocaleA.monitor() !=
MyObjectUnderLocaleB.monitor()
ii.
GlobalLock.acquire
iii. Synchronize
on monitor()
iv.
Synchronize on MyObjectUnderLocaleB.monitor()
So if the reverse is happening concurrently, Thread 1 will have a lock
on Locale A and the GlobalLock, and Thread 2 will have a lock on Locale
B and wait to get the GlobalLock - hence the deadlock situation.
It may be that this is arising from a strange/undesired use of XmlBeans
- we have a HashMap which caches template XmlBean objects, and when we
need an instance of the template, we were calling the copy() method and
returning that. I believe this means the template and our instance share
the same locale. Once the number of users ramps up, there are a lot of
users sharing the same locale and consequently introducing a lot of
contention.
We have worked around the problem for now by performing an
XmlObject.Factory.parse(...) and so each returned object is then in its
own Locale.
If someone can confirm this, can it be raised as a defect? If it's not a
defect, then is there a way to do a deep copy that will create the new
object in a new Locale?
Thanks
Paul
-----Original Message-----
From: Paul Hepworth
Sent: 27 October 2008 14:28
To: 'dev@xmlbeans.apache.org'
Subject: RE: XmlBeans and Synchronization
Hi Radu
Thanks for the explanation. In theory it all seems sound. Only our
application is getting itself locked somehow.
I've done a thread-dump from Weblogic, and have multiple threads waiting
on XmlObjectBase.java:1944 which is fine and to be expected as per your
explanation.
I then have another thread which is stuck on XmlObjectBase.java:1949 -
so this is the one that has the GlobalLock, but is in turn waiting on
another lock which isn't getting released at all.
I have several other threads that are blocked on XmlObjectBase.java:116
performing a copy which I believe are fine and just waiting for a lock.
I then have a few threads performing simple gets that are blocked
waiting for locks on Locale.
I have no other threads that are doing any processing (I've left the app
in this state for 10 or so minutes and all the threads are just locked
up)
As I can't find much info on the net, could you explain a bit more about
the Local please?
What I'd like to know is the following:
- Is there a new Local per instance of a document or a shared one
for all instances of the same type? i.e. if I call
MyDocument.Factory.newInstance(), does this get a new Local each time,
or 1 shared one for all MyDocument instances?
- We have a template service that reads and parses XML stored in
the DB, this is cached for performance reasons, and when we request a
template, we call the copy() method. Does this mean that each copied
instance shares the same Locale? If that's the case, this will greatly
impact the performance of our app.
- Basically... what determines the Locale?
Additionally, you mention turning off synchronisation... I've attempted
this in the code using XmlBeans 2.3.0 and run into the problems
mentioned here: https://issues.apache.org/jira/browse/XMLBEANS-328 it
should also be noted that we were having these errors in our performance
test environment originally and applied the patch mentioned. This
stopped the errors described, however we were still running into the
deadlock situation.
Is this something which has been fixed in 2.4? Is it a safe option to
turn off synchronisation?
Any help greatly appreciated!
Thanks
Paul
Hi Paul,
I can tell you what the synchronization strategy is. Each XmlObject
belongs to an xml document,
and each xml document belongs to what XmlBeans calls a Locale object
(nothing to do with the
Java Locale). This is the synchronization domain for XmlBeans. When
doing a set, there are
two XmlObjects involved: the source (the parameter of the "set" method)
and the target (the
object you're calling "set" on). XMLBeans requires that they both be
synchronized. Now, if
the synchronization domain for the two is the same (the same Locale
object), then there's
no problem (see check at XmlObjectBase.java:1934 if you have the 2.3.0
source). But if the
synchronization domain is not the same, then it means that two locks
must be acquired and
so there is a possibility for deadlock. In order to avoid the deadlock,
XMLBeans has the "GlobalLock".
This global lock is held only for as long as it is necessary to acquire
the two XmlObject
locks and then it is immediately released. This guards against the
deadlock, but doesn't guard
against the possibility that one thread grabs the global lock and then
blocks on waiting for
another thread to finish with one of the XmlObject locks it is trying to
acquire. Eventually,
it will grab the lock it is waiting for and then release the Global
lock, but while it is
waiting, all the other threads in the whole system are prevented from
initiating a set that
requires two locks, which can be a problem.
So I think first of all, it would be interesting for you to look at the
stack traces for all
the threads to confirm my scenario above. Then, as far as solution, the
only reliable one
that I know of is not use the "setMyValue" method and replace it with
"addNewMyValue" whenever
possible because it doesn't require synchronization (and the overall
performance is better).
In XMLBeans 2.4.0, I think, removing synchronization altogether from the
XMLBeans layer would
also solve the problem.
Radu
________________________________
From: Paul Hepworth [mailto:[EMAIL PROTECTED]
Sent: Friday, October 24, 2008 5:00 AM
To: [EMAIL PROTECTED]; dev@xmlbeans.apache.org
Subject: XmlBeans and Synchronization
Hi (included dev list as this may be too in depth for the user
list)
I'm using XmlBeans 2.3.0 in a Weblogic 9.2 J2EE web application
(JVM 1.5.0_10). We use XmlBeans
to compile our schema and these effectively become our domain objects.
These are looked up
from the DB, stored in the Http Session, manipulated from the UI and
passed to the DB again.
The application appears to run fine with a single user, however
when we test it with multiple
users using LoadRunner, we hit issues.
What we are seeing is that we're getting stuck threads with the
following stack:
at java.lang.Object.wait(Native Method)
- waiting on <0xd4a816e8> (a
org.apache.xmlbeans.impl.common.Mutex)
at java.lang.Object.wait(Object.java:474)
at org.apache.xmlbeans.impl.common.Mutex.acquire(Mutex.java:33)
- locked <0xd4a816e8> (a org.apache.xmlbeans.impl.common.Mutex)
at
org.apache.xmlbeans.impl.common.GlobalLock.acquire(GlobalLock.java:27)
at
org.apache.xmlbeans.impl.values.XmlObjectBase.set(XmlObjectBase.java:194
4)
at ...MyClassImpl.setMyValue(...)
Can anyone shed any light on why this may happen?
More specifically, does anyone know what the synchronisation
strategy is in XmlBeans. From
my investigation, the GlobalLock uses a static Mutex, which means that
if an instance needs
to take out more than 1 lock, it attempts to acquire a GlobalLock and
will subsequently lock
out all other threads in the JVM from making any changes to an
XmlObject.
If that's the case, it seems very strange that making an update to
1 instance of an XmlObject
would lock out all other threads form updating any other instance of an
XmlObject. I would
have thought that the synchronisation would have solely been around the
instance itself.
We have also seen the same issues that have been recorded here:
https://issues.apache.org/jira/browse/XMLBEANS-328
Any help on this would be much appreciated as it's causing major
issues!!
Thanks
Paul
This message should be regarded as confidential. If you have
received this email in error
please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in
hard copy by an authorised
signatory. The contents of this email may relate to dealings with other
companies within the
Detica Group plc group of companies.
Detica Limited is registered in England under No: 1337451.
Registered offices: Surrey Research Park, Guildford, Surrey, GU2
7YP, England.
