RegionServer lease expire due to LogRoller take long time and the
RegionServer cannot recovery
-------------------------------------------------------------------------------------------------
Key: HBASE-1060
URL: https://issues.apache.org/jira/browse/HBASE-1060
Project: Hadoop HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.18.1
Environment: I have one master and 4 region servers.
The test use batchUpdate() to insert many record to achieve big data sets.
Reporter: schubert zhang
Usually, after tens GB of data, following exception occurs, it seems LogRoller
take long time and the RegionServer cannot report to master, and the lease
expire. But it cannot recovery.
---------
2008-12-13 11:13:39,347 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
Rolling hlog. Number of entries: 30015
2008-12-13 11:13:39,354 INFO org.apache.hadoop.hbase.regionserver.HLog: New log
writer created at
/hbase/log_10.24.1.14_1229053900800_60020/hlog.dat.1229138019352
2008-12-13 11:13:47,991 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
Rolling hlog. Number of entries: 30015
2008-12-13 11:13:49,138 INFO org.apache.hadoop.hbase.regionserver.HLog: New log
writer created at
/hbase/log_10.24.1.14_1229053900800_60020/hlog.dat.1229138029136
2008-12-13 11:15:49,867 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
120318ms, ten times longer than scheduled: 3000
2008-12-13 11:15:50,175 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to master
for 120343 milliseconds - retrying
2008-12-13 11:15:50,271 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_CALL_SERVER_STARTUP
2008-12-13 11:15:50,419 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6
on 60020, call get([...@4b2a82b3, [...@53eadc52, [...@6026b688, -1, -1) from
10.24.1.18:58581: error: org.apache.hadoop.hbase.NotServingRegionException:
TableDes,,1228799531753
org.apache.hadoop.hbase.NotServingRegionException: TableDes,,1228799531753
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1859)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1321)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-13 11:15:50,434 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
on 60020, call get([...@26ee4c87, [...@66e24708, [...@11a592fd, -1, -1) from
10.24.1.12:46262: error: org.apache.hadoop.hbase.NotServingRegionException:
TableDes,,1228799531753
org.apache.hadoop.hbase.NotServingRegionException: TableDes,,1228799531753
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1859)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1321)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-13 11:15:50,431 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2
on 60020, call get([...@519182c2, [...@281fb864, [...@3a82c1d0, -1, -1) from
10.24.1.16:34950: error: org.apache.hadoop.hbase.NotServingRegionException:
TableDes,,1228799531753
org.apache.hadoop.hbase.NotServingRegionException: TableDes,,122879953175
.................many exceptions.......................
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
