[
https://issues.apache.org/jira/browse/YARN-9163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17904451#comment-17904451
]
William Montaz edited comment on YARN-9163 at 12/10/24 10:56 AM:
-----------------------------------------------------------------
This ticket should be reopened as this is a real issue and is {*}still present
in community version 3.3.0{*}. We have a busy cluster (up to 3-4k containers
per second) and we call refreshQueues quite frequently to apply dynamic
reconfiguration (every 5min). We encountered the problem twice in a 2 months
period and it correspond exactly to what is described.
Moreover, this is not a bug of a given JDK -> ReentrantReadWriteLock, even with
unfair policy, will favor a writer thread to get the lock if it is the next to
get it in the wait queue.
This situation can be reproduced in plain java code, where the ReadLock is
held, but cannot be get by another reader thread because a writer thread is
already in queue and is the next in the queue.
{code:java}
import java.util.concurrent.locks.ReentrantLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
public class Main {
ReentrantLock otherLock = new ReentrantLock();
void log(String s) {
System.out.printf("%s: %s%n", Thread.currentThread().getName(), s);
}
public static void main(String[] args) throws Exception {
new Main().runTest();
}
public void runTest() throws Exception {
ReentrantReadWriteLock rwlock = new ReentrantReadWriteLock(false);
// obtain read lock
log("get readlock");
rwlock.readLock().lock(); //should sucess to get the readLock
new Thread(this.new ReadLockThread(rwlock), "TryRead").start(); //will
get write lock and 2 sec later try to get read
new Thread(this.new WriteLockThread(rwlock), "TryWrite").start();
//will try to get write lock and be queued before previous read thread
log("try to get other lock");
otherLock.lock(); //should not succeed as this lock is taken by the
read thread, but the read thread is blocked by the write thread in queue
rwlock.readLock().unlock();
}
class WriteLockThread implements Runnable {
private ReentrantReadWriteLock rwlock;
public WriteLockThread(ReentrantReadWriteLock rwlock) {
this.rwlock = rwlock;
}
public void run() {
try {
log("try get writelock");
rwlock.writeLock().lock(); //should fail to get the writeLock
since the readLock already hold by another thread
log("can get writelock");
} finally {
rwlock.writeLock().unlock();
}
}
}
class ReadLockThread implements Runnable {
private ReentrantReadWriteLock rwlock;
public ReadLockThread(ReentrantReadWriteLock rwlock) {
this.rwlock = rwlock;
}
public void run() {
try {
log("try get write lock");
otherLock.lock();
log("try get read lock");
Thread.sleep(2000); // introduce latency to allow a writer
thread to be placed in queue before this one
rwlock.readLock().lock();
log("can get readlock");
} catch (InterruptedException e) {
throw new RuntimeException(e);
} finally {
log("unlock readlock");
rwlock.readLock().unlock();
}
}
}
}
{code}
was (Author: willymontaz):
This ticket should be reopened as this is a real issue and is {*}still present
in community version 3.3.0{*}. We have a busy cluster (up to 3-4k containers
per second) and we call refreshQueues quite frequently to apply dynamic
reconfiguration (every 5sec). We encountered the problem twice in a 2 months
period and it correspond exactly to what is described.
Moreover, this is not a bug of a given JDK -> ReentrantReadWriteLock, even with
unfair policy, will favor a writer thread to get the lock if it is the next to
get it in the wait queue.
This situation can be reproduced in plain java code, where the ReadLock is
held, but cannot be get by another reader thread because a writer thread is
already in queue and is the next in the queue.
{code:java}
import java.util.concurrent.locks.ReentrantLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
public class Main {
ReentrantLock otherLock = new ReentrantLock();
void log(String s) {
System.out.printf("%s: %s%n", Thread.currentThread().getName(), s);
}
public static void main(String[] args) throws Exception {
new Main().runTest();
}
public void runTest() throws Exception {
ReentrantReadWriteLock rwlock = new ReentrantReadWriteLock(false);
// obtain read lock
log("get readlock");
rwlock.readLock().lock(); //should sucess to get the readLock
new Thread(this.new ReadLockThread(rwlock), "TryRead").start(); //will
get write lock and 2 sec later try to get read
new Thread(this.new WriteLockThread(rwlock), "TryWrite").start();
//will try to get write lock and be queued before previous read thread
log("try to get other lock");
otherLock.lock(); //should not succeed as this lock is taken by the
read thread, but the read thread is blocked by the write thread in queue
rwlock.readLock().unlock();
}
class WriteLockThread implements Runnable {
private ReentrantReadWriteLock rwlock;
public WriteLockThread(ReentrantReadWriteLock rwlock) {
this.rwlock = rwlock;
}
public void run() {
try {
log("try get writelock");
rwlock.writeLock().lock(); //should fail to get the writeLock
since the readLock already hold by another thread
log("can get writelock");
} finally {
rwlock.writeLock().unlock();
}
}
}
class ReadLockThread implements Runnable {
private ReentrantReadWriteLock rwlock;
public ReadLockThread(ReentrantReadWriteLock rwlock) {
this.rwlock = rwlock;
}
public void run() {
try {
log("try get write lock");
otherLock.lock();
log("try get read lock");
Thread.sleep(2000); // introduce latency to allow a writer
thread to be placed in queue before this one
rwlock.readLock().lock();
log("can get readlock");
} catch (InterruptedException e) {
throw new RuntimeException(e);
} finally {
log("unlock readlock");
rwlock.readLock().unlock();
}
}
}
}
{code}
> Deadlock when use yarn rmadmin -refreshQueues
> ---------------------------------------------
>
> Key: YARN-9163
> URL: https://issues.apache.org/jira/browse/YARN-9163
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 3.1.1
> Reporter: Hu Ziqian
> Assignee: Hu Ziqian
> Priority: Blocker
> Attachments: YARN-9163.001.patch, rm.jstack.ziqian.log
>
>
> We have a cluster with 4000+ node and 10w+ app per-day in our production
> environment. When we use CLI: yarn rmadmin -refreshQueues, the active rm's
> process is stuck and ha doesn't happen, which means all the cluster stops
> service and we can only fix it by reboot active rm. We can reproduce on our
> production cluster every time but can't reproduce in our test environment
> which only has 100+ nodes and few apps. Both of our production and test
> environment use CapacityScheduler which open asyncSchedule function and
> preemption
> Analyzing the jstack of active rm, we found a dead lock in it:
> thread one（ refreshqueue thread）：
> * take write lock of capacity scheduler
> * take write lock of preemptionManager
> * wait read lock of root queue
> thread two (asyncScheduleThread)
> * take read lock of root queue
> * wait write lock of PreemptionManager
> thread three (ipc handler on 8030 which deal the allocate )
> * wait write lock of root queue
> These three thread work with a dead lock.
>
> The deadlock happens because of a "bug" of ReadWriteLock: writeLock request
> blocks future readLock despite policy
> unfair([https://bugs.openjdk.java.net/browse/JDK-6893626).] In order to solve
> this problem, we change the logic of refreshqueue thread, get a queue info
> copy first and avoid the thread to take write lock of preemptionManager and
> read lock of root queue at the same time.
>
> We test our new code in our production environment and the refresh queue
> command works well.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
