[jira] [Commented] (HBASE-16304) HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock when trying to obtain write lock on updatesLock
[
https://issues.apache.org/jira/browse/HBASE-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893038#comment-15893038
]
stack commented on HBASE-16304:
---
The comments don't answer his question which was:
bq. Why this is called only in doDelta ... dropMemstoreContents();
> HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock
> when trying to obtain write lock on updatesLock
> --
>
> Key: HBASE-16304
> URL: https://issues.apache.org/jira/browse/HBASE-16304
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.2
>Reporter: mingmin xu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3
>
> Attachments: 16304.branch-1.2.v4.txt, 16304.branch-1.2.v5.txt,
> 16304.branch-1.2.v5.txt, 16304.branch-1.v1.txt, 16304.v1.txt, 16304.v3.txt,
> 16304.v4.txt, 16304.v4.txt, 16304.v5.txt, 16304.v6.txt, 16304.v7.txt
>
>
> here is my jvm stack:
> {code}
> 2016-07-29 16:36:56
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode):
> "Timer for 'HBase' metrics system" daemon prio=10 tid=0x7f205cf38000
> nid=0xafa5 in Object.wait() [0x7f203b353000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00063503c790> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> "Attach Listener" daemon prio=10 tid=0x7f205d017800 nid=0x1300 waiting on
> condition [0x]
>java.lang.Thread.State: RUNNABLE
> "IPC Parameter Sending Thread #2" daemon prio=10 tid=0x7f205c7c4000
> nid=0x4f1a waiting on condition [0x7f20362e1000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066f996718> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-1" prio=10
> tid=0x7f2054ec8000 nid=0x832d waiting on condition [0x7f2039a18000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-0" prio=10
> tid=0x7f20542ca800 nid=0x5a5d waiting on condition [0x7f2033bba000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "hadoop-datanode-0042.corp.cootek.com,16020,1469690065288_ChoreService_2"
> daemon prio=10 tid=0x7f205d0d4000
[jira] [Commented] (HBASE-16304) HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock when trying to obtain write lock on updatesLock
[
https://issues.apache.org/jira/browse/HBASE-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890490#comment-15890490
]
Ted Yu commented on HBASE-16304:
This was brought up by HBASE-17712
I addressed Ram's comment in 16304.v7.txt.
Here was the comment in HRegion:
{code}
// dropMemstoreContentsForSeqId() would acquire write lock of updatesLock
// We perform this operation outside of the read lock of updatesLock to avoid
dead lock
// See HBASE-16304
{code}
> HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock
> when trying to obtain write lock on updatesLock
> --
>
> Key: HBASE-16304
> URL: https://issues.apache.org/jira/browse/HBASE-16304
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.2
>Reporter: mingmin xu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3
>
> Attachments: 16304.branch-1.2.v4.txt, 16304.branch-1.2.v5.txt,
> 16304.branch-1.2.v5.txt, 16304.branch-1.v1.txt, 16304.v1.txt, 16304.v3.txt,
> 16304.v4.txt, 16304.v4.txt, 16304.v5.txt, 16304.v6.txt, 16304.v7.txt
>
>
> here is my jvm stack:
> {code}
> 2016-07-29 16:36:56
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode):
> "Timer for 'HBase' metrics system" daemon prio=10 tid=0x7f205cf38000
> nid=0xafa5 in Object.wait() [0x7f203b353000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00063503c790> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> "Attach Listener" daemon prio=10 tid=0x7f205d017800 nid=0x1300 waiting on
> condition [0x]
>java.lang.Thread.State: RUNNABLE
> "IPC Parameter Sending Thread #2" daemon prio=10 tid=0x7f205c7c4000
> nid=0x4f1a waiting on condition [0x7f20362e1000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066f996718> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-1" prio=10
> tid=0x7f2054ec8000 nid=0x832d waiting on condition [0x7f2039a18000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-0" prio=10
> tid=0x7f20542ca800 nid=0x5a5d waiting on condition [0x7f2033bba000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
>
[jira] [Commented] (HBASE-16304) HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock when trying to obtain write lock on updatesLock
[
https://issues.apache.org/jira/browse/HBASE-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435980#comment-15435980
]
Hudson commented on HBASE-16304:
FAILURE: Integrated in Jenkins build HBase-1.4 #367 (See
[https://builds.apache.org/job/HBase-1.4/367/])
HBASE-16304 HRegion#RegionScannerImpl#handleFileNotFoundException may (tedyu:
rev f9490aaf43e1df0caead47cb03127cd61206ba00)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock
> when trying to obtain write lock on updatesLock
> --
>
> Key: HBASE-16304
> URL: https://issues.apache.org/jira/browse/HBASE-16304
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.2
>Reporter: mingmin xu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3
>
> Attachments: 16304.branch-1.2.v4.txt, 16304.branch-1.2.v5.txt,
> 16304.branch-1.2.v5.txt, 16304.branch-1.v1.txt, 16304.v1.txt, 16304.v3.txt,
> 16304.v4.txt, 16304.v4.txt, 16304.v5.txt, 16304.v6.txt, 16304.v7.txt
>
>
> here is my jvm stack:
> {code}
> 2016-07-29 16:36:56
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode):
> "Timer for 'HBase' metrics system" daemon prio=10 tid=0x7f205cf38000
> nid=0xafa5 in Object.wait() [0x7f203b353000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00063503c790> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> "Attach Listener" daemon prio=10 tid=0x7f205d017800 nid=0x1300 waiting on
> condition [0x]
>java.lang.Thread.State: RUNNABLE
> "IPC Parameter Sending Thread #2" daemon prio=10 tid=0x7f205c7c4000
> nid=0x4f1a waiting on condition [0x7f20362e1000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066f996718> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-1" prio=10
> tid=0x7f2054ec8000 nid=0x832d waiting on condition [0x7f2039a18000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-0" prio=10
> tid=0x7f20542ca800 nid=0x5a5d waiting on condition [0x7f2033bba000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
>
[jira] [Commented] (HBASE-16304) HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock when trying to obtain write lock on updatesLock
[
https://issues.apache.org/jira/browse/HBASE-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435806#comment-15435806
]
Hudson commented on HBASE-16304:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #14 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/14/])
HBASE-16304 HRegion#RegionScannerImpl#handleFileNotFoundException may (tedyu:
rev 029131ddfedb7ef878dd6531988cbadd2b2da6a8)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock
> when trying to obtain write lock on updatesLock
> --
>
> Key: HBASE-16304
> URL: https://issues.apache.org/jira/browse/HBASE-16304
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.2
>Reporter: mingmin xu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3
>
> Attachments: 16304.branch-1.2.v4.txt, 16304.branch-1.2.v5.txt,
> 16304.branch-1.2.v5.txt, 16304.branch-1.v1.txt, 16304.v1.txt, 16304.v3.txt,
> 16304.v4.txt, 16304.v4.txt, 16304.v5.txt, 16304.v6.txt, 16304.v7.txt
>
>
> here is my jvm stack:
> {code}
> 2016-07-29 16:36:56
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode):
> "Timer for 'HBase' metrics system" daemon prio=10 tid=0x7f205cf38000
> nid=0xafa5 in Object.wait() [0x7f203b353000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00063503c790> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> "Attach Listener" daemon prio=10 tid=0x7f205d017800 nid=0x1300 waiting on
> condition [0x]
>java.lang.Thread.State: RUNNABLE
> "IPC Parameter Sending Thread #2" daemon prio=10 tid=0x7f205c7c4000
> nid=0x4f1a waiting on condition [0x7f20362e1000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066f996718> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-1" prio=10
> tid=0x7f2054ec8000 nid=0x832d waiting on condition [0x7f2039a18000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-0" prio=10
> tid=0x7f20542ca800 nid=0x5a5d waiting on condition [0x7f2033bba000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
>
[jira] [Commented] (HBASE-16304) HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock when trying to obtain write lock on updatesLock
[
https://issues.apache.org/jira/browse/HBASE-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435721#comment-15435721
]
Hudson commented on HBASE-16304:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #11 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/11/])
HBASE-16304 HRegion#RegionScannerImpl#handleFileNotFoundException may (tedyu:
rev 029131ddfedb7ef878dd6531988cbadd2b2da6a8)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock
> when trying to obtain write lock on updatesLock
> --
>
> Key: HBASE-16304
> URL: https://issues.apache.org/jira/browse/HBASE-16304
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.2
>Reporter: mingmin xu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3
>
> Attachments: 16304.branch-1.2.v4.txt, 16304.branch-1.2.v5.txt,
> 16304.branch-1.2.v5.txt, 16304.branch-1.v1.txt, 16304.v1.txt, 16304.v3.txt,
> 16304.v4.txt, 16304.v4.txt, 16304.v5.txt, 16304.v6.txt, 16304.v7.txt
>
>
> here is my jvm stack:
> {code}
> 2016-07-29 16:36:56
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode):
> "Timer for 'HBase' metrics system" daemon prio=10 tid=0x7f205cf38000
> nid=0xafa5 in Object.wait() [0x7f203b353000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00063503c790> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> "Attach Listener" daemon prio=10 tid=0x7f205d017800 nid=0x1300 waiting on
> condition [0x]
>java.lang.Thread.State: RUNNABLE
> "IPC Parameter Sending Thread #2" daemon prio=10 tid=0x7f205c7c4000
> nid=0x4f1a waiting on condition [0x7f20362e1000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066f996718> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-1" prio=10
> tid=0x7f2054ec8000 nid=0x832d waiting on condition [0x7f2039a18000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-0" prio=10
> tid=0x7f20542ca800 nid=0x5a5d waiting on condition [0x7f2033bba000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
>
[jira] [Commented] (HBASE-16304) HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock when trying to obtain write lock on updatesLock
[
https://issues.apache.org/jira/browse/HBASE-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435622#comment-15435622
]
Hudson commented on HBASE-16304:
FAILURE: Integrated in Jenkins build HBase-1.3 #827 (See
[https://builds.apache.org/job/HBase-1.3/827/])
HBASE-16304 HRegion#RegionScannerImpl#handleFileNotFoundException may (tedyu:
rev 70daa23ea79b21b9ae337bcf962ada26ac95c1a2)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock
> when trying to obtain write lock on updatesLock
> --
>
> Key: HBASE-16304
> URL: https://issues.apache.org/jira/browse/HBASE-16304
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.2
>Reporter: mingmin xu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3
>
> Attachments: 16304.branch-1.2.v4.txt, 16304.branch-1.2.v5.txt,
> 16304.branch-1.2.v5.txt, 16304.branch-1.v1.txt, 16304.v1.txt, 16304.v3.txt,
> 16304.v4.txt, 16304.v4.txt, 16304.v5.txt, 16304.v6.txt, 16304.v7.txt
>
>
> here is my jvm stack:
> {code}
> 2016-07-29 16:36:56
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode):
> "Timer for 'HBase' metrics system" daemon prio=10 tid=0x7f205cf38000
> nid=0xafa5 in Object.wait() [0x7f203b353000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00063503c790> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> "Attach Listener" daemon prio=10 tid=0x7f205d017800 nid=0x1300 waiting on
> condition [0x]
>java.lang.Thread.State: RUNNABLE
> "IPC Parameter Sending Thread #2" daemon prio=10 tid=0x7f205c7c4000
> nid=0x4f1a waiting on condition [0x7f20362e1000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066f996718> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-1" prio=10
> tid=0x7f2054ec8000 nid=0x832d waiting on condition [0x7f2039a18000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-0" prio=10
> tid=0x7f20542ca800 nid=0x5a5d waiting on condition [0x7f2033bba000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
>
[jira] [Commented] (HBASE-16304) HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock when trying to obtain write lock on updatesLock
[
https://issues.apache.org/jira/browse/HBASE-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435463#comment-15435463
]
Hudson commented on HBASE-16304:
FAILURE: Integrated in Jenkins build HBase-1.3-IT #801 (See
[https://builds.apache.org/job/HBase-1.3-IT/801/])
HBASE-16304 HRegion#RegionScannerImpl#handleFileNotFoundException may (tedyu:
rev 70daa23ea79b21b9ae337bcf962ada26ac95c1a2)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock
> when trying to obtain write lock on updatesLock
> --
>
> Key: HBASE-16304
> URL: https://issues.apache.org/jira/browse/HBASE-16304
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.2
>Reporter: mingmin xu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3
>
> Attachments: 16304.branch-1.2.v4.txt, 16304.branch-1.2.v5.txt,
> 16304.branch-1.2.v5.txt, 16304.branch-1.v1.txt, 16304.v1.txt, 16304.v3.txt,
> 16304.v4.txt, 16304.v4.txt, 16304.v5.txt, 16304.v6.txt, 16304.v7.txt
>
>
> here is my jvm stack:
> {code}
> 2016-07-29 16:36:56
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode):
> "Timer for 'HBase' metrics system" daemon prio=10 tid=0x7f205cf38000
> nid=0xafa5 in Object.wait() [0x7f203b353000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00063503c790> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> "Attach Listener" daemon prio=10 tid=0x7f205d017800 nid=0x1300 waiting on
> condition [0x]
>java.lang.Thread.State: RUNNABLE
> "IPC Parameter Sending Thread #2" daemon prio=10 tid=0x7f205c7c4000
> nid=0x4f1a waiting on condition [0x7f20362e1000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066f996718> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-1" prio=10
> tid=0x7f2054ec8000 nid=0x832d waiting on condition [0x7f2039a18000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-0" prio=10
> tid=0x7f20542ca800 nid=0x5a5d waiting on condition [0x7f2033bba000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
>
[jira] [Commented] (HBASE-16304) HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock when trying to obtain write lock on updatesLock
[
https://issues.apache.org/jira/browse/HBASE-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435373#comment-15435373
]
Hudson commented on HBASE-16304:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1474 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1474/])
HBASE-16304 HRegion#RegionScannerImpl#handleFileNotFoundException may (tedyu:
rev bf7015d3204818fdc88ef505e0a06cac4ea2774b)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock
> when trying to obtain write lock on updatesLock
> --
>
> Key: HBASE-16304
> URL: https://issues.apache.org/jira/browse/HBASE-16304
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.2
>Reporter: mingmin xu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: 16304.branch-1.2.v4.txt, 16304.branch-1.2.v5.txt,
> 16304.branch-1.2.v5.txt, 16304.branch-1.v1.txt, 16304.v1.txt, 16304.v3.txt,
> 16304.v4.txt, 16304.v4.txt, 16304.v5.txt, 16304.v6.txt, 16304.v7.txt
>
>
> here is my jvm stack:
> {code}
> 2016-07-29 16:36:56
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode):
> "Timer for 'HBase' metrics system" daemon prio=10 tid=0x7f205cf38000
> nid=0xafa5 in Object.wait() [0x7f203b353000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00063503c790> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> "Attach Listener" daemon prio=10 tid=0x7f205d017800 nid=0x1300 waiting on
> condition [0x]
>java.lang.Thread.State: RUNNABLE
> "IPC Parameter Sending Thread #2" daemon prio=10 tid=0x7f205c7c4000
> nid=0x4f1a waiting on condition [0x7f20362e1000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066f996718> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-1" prio=10
> tid=0x7f2054ec8000 nid=0x832d waiting on condition [0x7f2039a18000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-0" prio=10
> tid=0x7f20542ca800 nid=0x5a5d waiting on condition [0x7f2033bba000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
>
[jira] [Commented] (HBASE-16304) HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock when trying to obtain write lock on updatesLock
[
https://issues.apache.org/jira/browse/HBASE-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434524#comment-15434524
]
Ted Yu commented on HBASE-16304:
Planning to integrate later today if there is no further review comment.
> HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock
> when trying to obtain write lock on updatesLock
> --
>
> Key: HBASE-16304
> URL: https://issues.apache.org/jira/browse/HBASE-16304
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.2
>Reporter: mingmin xu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: 16304.branch-1.2.v4.txt, 16304.branch-1.2.v5.txt,
> 16304.branch-1.2.v5.txt, 16304.branch-1.v1.txt, 16304.v1.txt, 16304.v3.txt,
> 16304.v4.txt, 16304.v4.txt, 16304.v5.txt, 16304.v6.txt, 16304.v7.txt
>
>
> here is my jvm stack:
> {code}
> 2016-07-29 16:36:56
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode):
> "Timer for 'HBase' metrics system" daemon prio=10 tid=0x7f205cf38000
> nid=0xafa5 in Object.wait() [0x7f203b353000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00063503c790> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> "Attach Listener" daemon prio=10 tid=0x7f205d017800 nid=0x1300 waiting on
> condition [0x]
>java.lang.Thread.State: RUNNABLE
> "IPC Parameter Sending Thread #2" daemon prio=10 tid=0x7f205c7c4000
> nid=0x4f1a waiting on condition [0x7f20362e1000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066f996718> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-1" prio=10
> tid=0x7f2054ec8000 nid=0x832d waiting on condition [0x7f2039a18000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-0" prio=10
> tid=0x7f20542ca800 nid=0x5a5d waiting on condition [0x7f2033bba000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00066ffb5950> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "hadoop-datanode-0042.corp.cootek.com,16020,1469690065288_ChoreService_2"
> daemon prio=10 tid=0x7f205d0d4000 nid=0x72af waiting on condition
> [0x7f203b151000]
