[
https://issues.apache.org/jira/browse/HBASE-5972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271479#comment-13271479
]
Nicolas Spiegelberg commented on HBASE-5972:
--------------------------------------------
RS lost its znode at around 9:02. .. but kept dragging on until 9:22 am to
finally die.
2012-05-01 09:02:47,617 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 32 on 60020: exiting
2012-05-01 09:02:47,616 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 289 on 60020: exiting
2012-05-01 09:02:47,617 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 51 on 60020: exiting
2012-05-01 09:03:38,395 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...
2012-05-01 09:04:38,395 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...
2012-05-01 09:05:38,396 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...2012-05-01 09:06:38,396 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...2012-05-01 09:07:38,396 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...
2012-05-01 09:08:38,396 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...
2012-05-01 09:09:38,396 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...2012-05-01 09:10:38,422 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...2012-05-01 09:11:38,472 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...2012-05-01 09:12:38,472 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...2012-05-01 09:13:38,473 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...2012-05-01 09:14:38,473 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...2012-05-01 09:15:38,482 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...2012-05-01 09:16:38,482 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...2012-05-01 09:17:38,482 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...2012-05-01 09:18:38,483 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...2012-05-01 09:19:38,483 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...2012-05-01 09:20:38,483 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...2012-05-01 09:21:38,483 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...2012-05-01 09:22:38,483 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
Compaction Thread to finish...
2012-05-01 09:22:40,378 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/TITAN019-ASH3-HBASE/MailBox_cell19_v1/a
296dd72364afee146190f5a751606eb/.tmp/ca501bfd82474bd39a94f5aff0e08709 File does
not exist. [Lease. Holder: DFSClient_-1458957666, pendingcreates: 1] at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1830)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1821)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1665)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:600)
at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:737)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1065)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1061)
at java.security.AccessController.doPrivileged(Native Method)
> RegionServer Hangs on Compaction upon NameNode Death
> ----------------------------------------------------
>
> Key: HBASE-5972
> URL: https://issues.apache.org/jira/browse/HBASE-5972
> Project: HBase
> Issue Type: Bug
> Reporter: Nicolas Spiegelberg
>
> Noticed 20-minute hangs when doing a namenode kill test on our production
> shadow cluster. All regionservers terminated except one, which got stuck with
> the following. There might be some race conditions/deadlocks on compaction
> thread shutdown worth investigating.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
