[
https://issues.apache.org/jira/browse/HADOOP-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560195#action_12560195
]
stack commented on HADOOP-2364:
-------------------------------
Here's more info...
Does above exceptions for 4 hours and then...
{code}
...
2008-01-16 07:18:58,757 DEBUG org.apache.hadoop.hbase.HRegionServer: Done
telling master we are up
2008-01-16 07:18:58,757 DEBUG org.apache.hadoop.hbase.HRegionServer: Config
from master: fs.default.name=XX.XX.XX.XX:10000
2008-01-16 07:18:58,757 DEBUG org.apache.hadoop.hbase.HRegionServer: Config
from master: hbase.rootdir=/hbase/XX.XX.XX.XX
...
{code}
... then it does this... (which don't look too pretty):
{code}
...
2008-01-16 07:18:59,517 INFO org.apache.hadoop.hbase.HRegionServer:
HRegionServer started at: XX.XX.XX.232:60020
2008-01-16 07:18:59,517 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9
on 60020: starting
2008-01-16 07:36:01,780 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:36:01,804 DEBUG org.apache.hadoop.hbase.HStore: starting
.META.,,1/info (1028785192/info) (no reconstruction log)
2008-01-16 07:36:01,832 DEBUG org.apache.hadoop.hbase.HStore: maximum sequence
id for hstore .META.,,1/info (1028785192/info) is 26067834
2008-01-16 07:36:01,852 DEBUG org.apache.hadoop.hbase.HRegion: Next sequence id
for region .META.,,1 is 26067835
2008-01-16 07:36:01,853 INFO org.apache.hadoop.hbase.HRegion: region .META.,,1
available
2008-01-16 07:36:01,854 DEBUG org.apache.hadoop.hbase.HLog: changing sequence
number from 0 to 26067835
2008-01-16 07:37:04,137 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:37:10,144 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:37:21,476 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:39:01,468 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:39:22,489 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:40:27,559 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:41:12,603 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:41:27,619 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:41:36,628 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:41:42,633 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:41:48,640 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:41:57,650 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:42:03,657 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:42:21,675 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:43:03,719 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:46:47,721 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:47:05,738 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:47:11,744 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:47:43,403 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:47:49,408 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:48:04,423 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:48:37,457 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:52:18,805 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:52:30,817 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:55:39,002 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:59:40,590 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 07:59:55,605 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:03:34,072 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:03:40,079 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:04:01,099 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:04:10,108 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:04:16,113 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:04:22,120 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:04:46,724 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:04:52,730 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:05:16,754 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:05:22,760 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:09:11,033 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:12:01,093 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:12:07,100 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:13:55,994 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:14:05,002 INFO org.apache.hadoop.hbase.HRegionServer:
MSG_REGION_OPEN : .META.,,1
2008-01-16 08:17:20,146 WARN org.apache.hadoop.hbase.HRegionServer: Processing
message (Retry: 0)
java.net.SocketTimeoutException: timed out waiting for rpc response
at org.apache.hadoop.ipc.Client.call(Client.java:484)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
at $Proxy0.regionServerReport(Unknown Source)
at org.apache.hadoop.hbase.HRegionServer.run(HRegionServer.java:714)
at java.lang.Thread.run(Unknown Source)
2008-01-16 08:17:20,147 FATAL org.apache.hadoop.hbase.HRegionServer: unable to
report to master for 63015 milliseconds - aborting server
..
{code}
.. and shuts itself down.
Back later w/ master side of things...
> when hbase regionserver restarts, it says "impossible state for createLease()"
> ------------------------------------------------------------------------------
>
> Key: HADOOP-2364
> URL: https://issues.apache.org/jira/browse/HADOOP-2364
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Reporter: Michael Bieniosek
> Assignee: Jim Kellerman
> Priority: Minor
>
> I restarted a regionserver, and got this error in its logs:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> java.lang.AssertionError: Impossible state for createLease(): Lease
> -435227488/-435227488 is still held.
> at org.apache.hadoop.hbase.Leases.createLease(Leases.java:145)
> at
> org.apache.hadoop.hbase.HMaster.regionServerStartup(HMaster.java:1278
> )
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> at org.apache.hadoop.ipc.Client.call(Client.java:482)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
> at $Proxy0.regionServerStartup(Unknown Source)
> at
> org.apache.hadoop.hbase.HRegionServer.reportForDuty(HRegionServer.jav
> a:1025)
> at org.apache.hadoop.hbase.HRegionServer.run(HRegionServer.java:659)
> at java.lang.Thread.run(Unknown Source)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
