[
https://issues.apache.org/jira/browse/HADOOP-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548427
]
Jim Kellerman commented on HADOOP-2338:
---------------------------------------
What happened:
Master is starting up and successfully assigns root and meta regions and
processes the responses.
As the master is scanning the meta, it is finding stale serverinfo, splitting
the old logs and assigning regions.
The first open report comes in and is queued up for the main thread. It tries
to process the request but can't
because the meta region has not been completely scanned. The main thread then
starts starving the rest of
the master by putting the request back on the queue and taking it off.
Needless to say, this slows down processing of the meta region considerably and
prevents the master from responding to
heartbeat messages.
The master completes its first scan of the meta table, and starts its second
pass, enough time has elapsed
that the master thinks that the first server it assigned the region to either
didn't get the message or died, so
the master assigns the region to another server (the one that can't get a reply
back to the master for 67 seconds).
The master finally processes the original open response and updates the meta.
A while later, the master assigns the same region again to yet a third server.
It updates the meta again with the third server's
information.
It is quite understandable that in this chaos, the region would no longer
appear in the assignAttempts map causing the
NPE in the master.
Recommendations:
- Do not assign any user regions until all the meta regions have been scanned
once.
- If we assume that message delivery is reliable, we don't need the
assignAttempts map because if we don't hear
back from the server we assign a region to, its lease will expire and we
could reassign the region at that time.
> [hbase] NPE in master server
> ----------------------------
>
> Key: HADOOP-2338
> URL: https://issues.apache.org/jira/browse/HADOOP-2338
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.16.0
>
> Attachments: master.log.gz
>
>
> Master gets an NPE after receiving multiple responses from the same server
> telling the master it has opened a region.
> {code}
> 2007-12-02 20:31:37,515 DEBUG hbase.HRegion - Next sequence id for region
> postlog,img254/577/02suecia024richardburnson0.jpg,1196619667879 is 73377537
> 2007-12-02 20:31:37,517 INFO hbase.HRegion - region
> postlog,img254/577/02suecia024richardburnson0.jpg,1196619667879 available
> 2007-12-02 20:31:39,200 WARN hbase.HRegionServer - Processing message
> (Retry: 0)
> java.io.IOException: java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hbase.HMaster.processMsgs(HMaster.java :1484)
> at org.apache.hadoop.hbase.HMaster.regionServerReport(HMaster.java:1423)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java
> :25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0 (Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java
> :27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException
> (RemoteExceptionHandler.java:48)
> at org.apache.hadoop.hbase.HRegionServer.run(HRegionServer.java:759)
> at java.lang.Thread.run(Thread.java:619)
> case HMsg.MSG_REPORT_PROCESS_OPEN:
> synchronized ( this.assignAttempts) {
> // Region server has acknowledged request to open region.
> // Extend region open time by 1/2 max region open time.
> **1484** assignAttempts.put(region.getRegionName (),
> Long.valueOf(assignAttempts.get(
> region.getRegionName()).longValue() +
> (this.maxRegionOpenTime / 2)));
> }
> break;
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
