[
https://issues.apache.org/jira/browse/HBASE-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668713#action_12668713
]
Andrew Purtell commented on HBASE-1163:
---------------------------------------
Every node in the cluster has an IPC connection open to /10.30.94.32:60020 (the
HRS hosting ROOT) and all have the exact same stack trace as thread #309 in the
comment above.
In the other HRS stack traces I see that many are hung on RPC to the master out
of CompactSplitThread now. RegionHistorian.add -> HTable.commit ->
HTable.flushCommits -> [...] TableServers.locateRegionInMeta ->
$Proxy.getClosestRowBefore
and the master is not returning from that RPC to getClosestRowBefore.
> Master root scanner hung, clients blocked indefinitely waiting for
> getStartKeys()
> ---------------------------------------------------------------------------------
>
> Key: HBASE-1163
> URL: https://issues.apache.org/jira/browse/HBASE-1163
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.19.0
> Reporter: Andrew Purtell
> Priority: Critical
>
> Mapreduce tasks based on TIF won't start. Clients trying to find regions by
> start key block indefinitely (Heritrix hbase writer eventually times out
> archiver).
> Master seems hung in root scan. I've dumped thread stacks 10 times in 10
> minutes and the same HBaseClient$Call object appears in the trace. See below:
> Thread 21 (RegionManager.rootScanner):
> State: WAITING
> Blocked count: 500
> Waited count: 621
> Waiting on org.apache.hadoop.hbase.ipc.hbaseclient$c...@55a2896d
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:695)
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:321)
> $Proxy2.next(Unknown Source)
>
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:161)
> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:55)
>
> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:80)
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
> org.apache.hadoop.hbase.Chore.run(Chore.java:65)
> I only see messages from the MetaScanner scanner in the master log, nothing
> from RootScanner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
