[
https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662840#action_12662840
]
stack commented on HBASE-1121:
------------------------------
It looks like its the regionhistorian trying to update about a compaction that
is failing:
{code}
2009-01-11 06:11:56,405 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG
org.apache.hadoop.hbase.regionserver.HStore: moving
/hbasetrunk2/TestTable/compaction.dir/1701213566/info/mapfiles/4916277877406001197
to /hbasetrunk2/TestTable/1701213566/info/mapfiles/3125166216229916041
2009-01-11 06:11:56,472 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG
org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of
1701213566/info store size is 188.2m
2009-01-11 06:11:56,480 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] INFO
org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
TestTable,0008454525,1231561159116 in 1mins, 8sec
2009-01-11 06:11:56,484 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10
failed with <org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
>. Retrying after sleep of 2000
2009-01-11 06:11:56,487 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT
REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED =>
70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE',
VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false',
BLOCKCACHE => 'true'}], INDEXES => []}}
2009-01-11 06:11:58,491 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10
failed with <org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
>. Retrying after sleep of 2000
...
{code}
Above goes on till attempt 8, then it starts over again at 0 and just keeps on
going.
Why ain't we recalibrating to find new -ROOT- location.
Cluster is hosed unless I restart a regionserver one at a time or just restart
whole cluster.
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of
> 10 failed with <org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742
> >[regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG
> >org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT
> >REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED =>
> >70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
> >FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE',
> >VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false',
> >BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759
> >[regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG
> >org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of
> >10 failed with <org.apache.hadoop.hbase.NotServingRegionException:
> >org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as
> though the regionservers' internal cache of -ROOT- location cannot be
> changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
