HMaster sometimes hangs during initialization due to missing notify call
------------------------------------------------------------------------
Key: HBASE-3493
URL: https://issues.apache.org/jira/browse/HBASE-3493
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.0
Environment: Linux, HBase 0.90
Reporter: Bruno Dumon
Attachments: assignment-manager-regions-notify-patch.txt
During HMaster.finishInitialization(), assignRootAndMeta() is called, which at
some point does:
{code}
this.assignmentManager.waitForAssignment(HRegionInfo.FIRST_META_REGIONINFO);
{code}
And waitForAssignment does this:
{code}
synchronized(regions) {
while(!regions.containsKey(regionInfo)) {
regions.wait();
}
}
{code}
However, I could not find any call to regions.notify(), so this could wait
forever.
I have not noticed this problem on a real cluster, only when using
HBaseTestingUtility on a slow and low-memory Hudson server (I can reproduce it
on my local machine by creating some background load). Adding a notify() call
when regions.put() is called seems to fix the problem. Will attach patch.
For reference, this was based on seeing the following from jstack:
{noformat}
"Master:0;lat:44776" prio=10 tid=0x08d5b400 nid=0x381a in Object.wait()
[0x9c76d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa5196fb8> (a java.util.TreeMap)
at java.lang.Object.wait(Object.java:485)
at
org.apache.hadoop.hbase.master.AssignmentManager.waitForAssignment(AssignmentManager.java:1152)
- locked <0xa5196fb8> (a java.util.TreeMap)
at
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:440)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:382)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:277)
at java.lang.Thread.run(Thread.java:619)
{noformat}
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
