Re: [infinispan-dev] JGroupsDistSync and ISPN-83
Apparently I did not understand semantics of RWL. I thought that writer
can obtain a write lock even though read lock has not been released. But
no, after all read locks have been released only then a writer can
obtain a lock.
On 11-05-16 11:18 PM, Sanne Grinovero wrote:
Same result here - where you expecting something different?
Cheers,
Sanne
2011/5/16 Erik Salteresal...@bnivideo.com:
EDIT: Originally posted in the wrong thread. I blame Outlook.
I guess I qualify as others, since I'm looking at similar issues.
Got read lock
java.util.concurrent.TimeoutException: Thread-1 could not obtain exclusive
processing lock after 3 seconds. Locks in question are
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock@16aa37a6[Read
locks = 1] and
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock@12b7eea[Unlocked]
This is on the latest master.
Erik
-Original Message-
From: infinispan-dev-boun...@lists.jboss.org
[mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Vladimir
Blagojevic
Sent: Monday, May 16, 2011 4:25 PM
To: infinispan -Dev List
Cc: Manik Surtani
Subject: Re: [infinispan-dev] JGroupsDistSync and ISPN-83
Manik (and others),
Can you run this code on your laptops and let me know what happened!
Vladimir
public static void main (String [] arg) throws Exception {
final JGroupsDistSync ds = new JGroupsDistSync();
ds.acquireProcessingLock(false, 3, TimeUnit.SECONDS);
System.out.println(Got read lock);
Thread t = new Thread(){
public void run (){
try {
ds.acquireProcessingLock(true, 3, TimeUnit.SECONDS);
System.out.println(Got write lock);
} catch (TimeoutException e) {
System.out.println(e);
}
}
};
t.start();
}
On 11-05-13 4:53 PM, Manik Surtani wrote:
Yes, please have a look. If we are relying on lock upgrades then that's
really bad. I am aware of the inability to (safely) upgrade a RWL and I'm
pretty sure we don't try, but the dist sync codebase has evolved a lot and
could do with some careful analysis.
Sent from my mobile phone
On 12 May 2011, at 09:24, Vladimir Blagojevicvblag...@redhat.comwrote:
On 11-05-11 11:23 AM, Dan Berindei wrote:
If ReentrantReadWriteLock would allow upgrades then you would get a
deadlock when two threads both hold the read lock and try to upgrade
to a write lock at the same time.
There's always a trade-off...
I'm not familiar with the code, but are you sure the read lock is
being held by the same thread that is trying to acquire the write
lock?
Not sure and it sounds counter intuitive that a thread holding a read
lock from cluster invocation is doing state generation for state
transfer as well. But this lock business is fishy and I plan to get
to the bottom of it...
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
The information contained in this message is legally privileged and
confidential, and is intended for the individual or entity to whom it is
addressed (or their designee). If this message is read by anyone other than
the intended recipient, please be advised that distribution of this message,
in any form, is strictly prohibited. If you have received this message in
error, please notify the sender immediately and delete or destroy all copies
of this message.
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] JGroupsDistSync and ISPN-83
Manik (and others),
Can you run this code on your laptops and let me know what happened!
Vladimir
public static void main (String [] arg) throws Exception {
final JGroupsDistSync ds = new JGroupsDistSync();
ds.acquireProcessingLock(false, 3, TimeUnit.SECONDS);
System.out.println(Got read lock);
Thread t = new Thread(){
public void run (){
try {
ds.acquireProcessingLock(true, 3, TimeUnit.SECONDS);
System.out.println(Got write lock);
} catch (TimeoutException e) {
System.out.println(e);
}
}
};
t.start();
}
On 11-05-13 4:53 PM, Manik Surtani wrote:
Yes, please have a look. If we are relying on lock upgrades then that's
really bad. I am aware of the inability to (safely) upgrade a RWL and I'm
pretty sure we don't try, but the dist sync codebase has evolved a lot and
could do with some careful analysis.
Sent from my mobile phone
On 12 May 2011, at 09:24, Vladimir Blagojevicvblag...@redhat.com wrote:
On 11-05-11 11:23 AM, Dan Berindei wrote:
If ReentrantReadWriteLock would allow upgrades then you would get a
deadlock when two threads both hold the read lock and try to upgrade
to a write lock at the same time.
There's always a trade-off...
I'm not familiar with the code, but are you sure the read lock is
being held by the same thread that is trying to acquire the write
lock?
Not sure and it sounds counter intuitive that a thread holding a read
lock from cluster invocation is doing state generation for state
transfer as well. But this lock business is fishy and I plan to get to
the bottom of it...
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] JGroupsDistSync and ISPN-83
Same result here - where you expecting something different?
Cheers,
Sanne
2011/5/16 Erik Salter esal...@bnivideo.com:
EDIT: Originally posted in the wrong thread. I blame Outlook.
I guess I qualify as others, since I'm looking at similar issues.
Got read lock
java.util.concurrent.TimeoutException: Thread-1 could not obtain exclusive
processing lock after 3 seconds. Locks in question are
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock@16aa37a6[Read
locks = 1] and
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock@12b7eea[Unlocked]
This is on the latest master.
Erik
-Original Message-
From: infinispan-dev-boun...@lists.jboss.org
[mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Vladimir
Blagojevic
Sent: Monday, May 16, 2011 4:25 PM
To: infinispan -Dev List
Cc: Manik Surtani
Subject: Re: [infinispan-dev] JGroupsDistSync and ISPN-83
Manik (and others),
Can you run this code on your laptops and let me know what happened!
Vladimir
public static void main (String [] arg) throws Exception {
final JGroupsDistSync ds = new JGroupsDistSync();
ds.acquireProcessingLock(false, 3, TimeUnit.SECONDS);
System.out.println(Got read lock);
Thread t = new Thread(){
public void run (){
try {
ds.acquireProcessingLock(true, 3, TimeUnit.SECONDS);
System.out.println(Got write lock);
} catch (TimeoutException e) {
System.out.println(e);
}
}
};
t.start();
}
On 11-05-13 4:53 PM, Manik Surtani wrote:
Yes, please have a look. If we are relying on lock upgrades then that's
really bad. I am aware of the inability to (safely) upgrade a RWL and I'm
pretty sure we don't try, but the dist sync codebase has evolved a lot and
could do with some careful analysis.
Sent from my mobile phone
On 12 May 2011, at 09:24, Vladimir Blagojevicvblag...@redhat.com wrote:
On 11-05-11 11:23 AM, Dan Berindei wrote:
If ReentrantReadWriteLock would allow upgrades then you would get a
deadlock when two threads both hold the read lock and try to upgrade
to a write lock at the same time.
There's always a trade-off...
I'm not familiar with the code, but are you sure the read lock is
being held by the same thread that is trying to acquire the write
lock?
Not sure and it sounds counter intuitive that a thread holding a read
lock from cluster invocation is doing state generation for state
transfer as well. But this lock business is fishy and I plan to get
to the bottom of it...
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
The information contained in this message is legally privileged and
confidential, and is intended for the individual or entity to whom it is
addressed (or their designee). If this message is read by anyone other than
the intended recipient, please be advised that distribution of this message,
in any form, is strictly prohibited. If you have received this message in
error, please notify the sender immediately and delete or destroy all copies
of this message.
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] JGroupsDistSync and ISPN-83
I agree Manik. Codebase has evolved and the solution I have for this problem involves detecting such a lockup happening again but this time we will capture which threads locked RWL and why thus leading to a final resolution. I will include the fix in the next CR and follow up in later releases (if necessary). Vladimir On 11-05-13 4:53 PM, Manik Surtani wrote: Yes, please have a look. If we are relying on lock upgrades then that's really bad. I am aware of the inability to (safely) upgrade a RWL and I'm pretty sure we don't try, but the dist sync codebase has evolved a lot and could do with some careful analysis. Sent from my mobile phone ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] JGroupsDistSync and ISPN-83
Yes, please have a look. If we are relying on lock upgrades then that's really bad. I am aware of the inability to (safely) upgrade a RWL and I'm pretty sure we don't try, but the dist sync codebase has evolved a lot and could do with some careful analysis. Sent from my mobile phone On 12 May 2011, at 09:24, Vladimir Blagojevic vblag...@redhat.com wrote: On 11-05-11 11:23 AM, Dan Berindei wrote: If ReentrantReadWriteLock would allow upgrades then you would get a deadlock when two threads both hold the read lock and try to upgrade to a write lock at the same time. There's always a trade-off... I'm not familiar with the code, but are you sure the read lock is being held by the same thread that is trying to acquire the write lock? Not sure and it sounds counter intuitive that a thread holding a read lock from cluster invocation is doing state generation for state transfer as well. But this lock business is fishy and I plan to get to the bottom of it... ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] JGroupsDistSync and ISPN-83
On 11-05-11 11:23 AM, Dan Berindei wrote: If ReentrantReadWriteLock would allow upgrades then you would get a deadlock when two threads both hold the read lock and try to upgrade to a write lock at the same time. There's always a trade-off... I'm not familiar with the code, but are you sure the read lock is being held by the same thread that is trying to acquire the write lock? Not sure and it sounds counter intuitive that a thread holding a read lock from cluster invocation is doing state generation for state transfer as well. But this lock business is fishy and I plan to get to the bottom of it... ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] JGroupsDistSync and ISPN-83
The more I research ReentrantReadWriteLock the more shocked I am. Not only that a thread wanting to acquire write lock first has to release read lock, but we can block forever even if we release the read lock if we have acquired that read lock reentrantly. Each call to unlock just reduces the hold count, and the lock is only actually released when the hold count hits zero. Surreal! People have already debated this issue: http://stackoverflow.com/questions/464784/java-reentrantreadwritelocks-how-to-safely-acquire-write-lock In conlusion we have to seriously revisit all our uses of ReentrantReadWriteLock! Vladimir On 11-05-10 11:31 AM, Vladimir Blagojevic wrote: Hi, Would you please confirm my understanding and reading of javadoc for ReentrantReadWriteLock under section for lock downgrading. It says: Reentrancy also allows downgrading from the write lock to a read lock, by acquiring the write lock, then the read lock and then releasing the write lock. However, upgrading from a read lock to the write lock is not possible. ReentrantReadWriteLock javadoc code example with class CacheData also shows how a thread owning a read lock first has to release it in order to obtain a write lock! So a thread owning a read lock first has to release a read lock in order to obtain a write lock? This is very symptomatic in logs of ISPN-83 such as this one https://issues.jboss.org/secure/attachment/12341409/server1.log Recall how FLUSH stops all invocations on cluster and therefore all read lock acquisitions in JGroupsDistSync ultimately enabling smooth write lock acquisitions for state transfer and what not. In conclusion this leads me wondering if the culprit of all this FLUSH mess is rooted in read/write lock semantics from above? Regards, Vladimir ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] JGroupsDistSync and ISPN-83
On Wed, May 11, 2011 at 11:25 AM, Vladimir Blagojevic vblag...@redhat.com wrote: The more I research ReentrantReadWriteLock the more shocked I am. Not only that a thread wanting to acquire write lock first has to release read lock, but we can block forever even if we release the read lock if we have acquired that read lock reentrantly. Each call to unlock just reduces the hold count, and the lock is only actually released when the hold count hits zero. Surreal! If ReentrantReadWriteLock would allow upgrades then you would get a deadlock when two threads both hold the read lock and try to upgrade to a write lock at the same time. There's always a trade-off... I'm not familiar with the code, but are you sure the read lock is being held by the same thread that is trying to acquire the write lock? Dan People have already debated this issue: http://stackoverflow.com/questions/464784/java-reentrantreadwritelocks-how-to-safely-acquire-write-lock In conlusion we have to seriously revisit all our uses of ReentrantReadWriteLock! Vladimir On 11-05-10 11:31 AM, Vladimir Blagojevic wrote: Hi, Would you please confirm my understanding and reading of javadoc for ReentrantReadWriteLock under section for lock downgrading. It says: Reentrancy also allows downgrading from the write lock to a read lock, by acquiring the write lock, then the read lock and then releasing the write lock. However, upgrading from a read lock to the write lock is not possible. ReentrantReadWriteLock javadoc code example with class CacheData also shows how a thread owning a read lock first has to release it in order to obtain a write lock! So a thread owning a read lock first has to release a read lock in order to obtain a write lock? This is very symptomatic in logs of ISPN-83 such as this one https://issues.jboss.org/secure/attachment/12341409/server1.log Recall how FLUSH stops all invocations on cluster and therefore all read lock acquisitions in JGroupsDistSync ultimately enabling smooth write lock acquisitions for state transfer and what not. In conclusion this leads me wondering if the culprit of all this FLUSH mess is rooted in read/write lock semantics from above? Regards, Vladimir ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
