Thanks, I will do more test.
Maybe the deadlock hapened like this? Please point it out if it's wrong.
1，One handler is handling put op, and reclaimMemStoreMemory, but the memory is
isAboveHighWaterMark, so this handler locked the memstoreflusher until global
mem is lower:
public synchronized void reclaimMemStoreMemory() {
if (isAboveHighWaterMark()) {
lock.lock();
try {
while (isAboveHighWaterMark() && !server.isStopped()) {
wakeupFlushThread();
try {
// we should be able to wait forever, but we've seen a bug where
// we miss a notify, so put a 5 second bound on it at least.
flushOccurred.await(5, TimeUnit.SECONDS);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
}
} finally {
lock.unlock();
2, flushforGlobalPressure is trigered, but to flush the memstore, it needed to
lock the memstoreflusher:
private boolean flushRegion(final HRegion region, final boolean
emergencyFlush) {
synchronized (this.regionsInQueue) {
FlushRegionEntry fqe = this.regionsInQueue.remove(region);
if (fqe != null && emergencyFlush) {
// Need to remove from region from delay queue. When NOT an
// emergencyFlush, then item was removed via a flushQueue.poll.
flushQueue.remove(fqe);
}
lock.lock();
}
3, because lock is locked by the ipchandler of put op, the flushRegion will
never get the lock and flush will never happen.
4, no flush, memory stay in AboveHighWaterMark state, and never unlock, so,
deadlock happend.
Is it right?
Zhou Shuaifeng(Frank)
-----邮件原件-----
发件人: [email protected] [mailto:[email protected]] 代表 Jean-Daniel Cryans
发送时间: 2011年4月29日 3:09
收件人: [email protected]
主题: Re: found one deadlock on hbase?
Like I said in the previous thread you made about this issue, it seems
that the YourKit profiler is doing something unexpected from the HBase
POV. Can you try running without it and see if it still happens?
J-D
2011/4/28 Zhoushuaifeng <[email protected]>:
> Thanks, version is 0.90.1
>
> Zhou Shuaifeng(Frank)
>
> -----邮件原件-----
> 发件人: [email protected] [mailto:[email protected]] 代表 Stack
> 发送时间: 2011年4月28日 13:10
> 收件人: [email protected]
> 抄送: Yanlijun
> 主题: Re: found one deadlock on hbase?
>
> Must be a deadlock if the dumb JVM can figure it out. What version of
> hbase please so I can dig into source code?
> Thanks,
> St.Ack
>
>
