[
https://issues.apache.org/jira/browse/HBASE-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658348#action_12658348
]
stack commented on HBASE-1073:
------------------------------
It looks like it though its hard to tell with these binary keys. At the time
that problematic region is first registered, there is shutdown processing going
on for another server altogether, server 214. This seems to mess things up.
Regions that were just written into the meta as being successfully opened on
server 218 are then assigned out to server 215 and 210.
Here is illustration of above (grep on 1229470182375):
{code}
2008-12-16 15:31:20,375 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
streamitems,^...@^@^...@^@^...@�^d;,1229470182375 open on 72.34.249.218:60020
2008-12-16 15:31:20,375 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
streamitems,^...@^@^...@^@^...@�^d;,1229470182375 in region .META.,,1 with
startcode 1229369663539 and server 72.34.249.218:60020
2008-12-16 15:31:20,376 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: PendingOpenOperation from 72.34.249.218:60020
2008-12-16 15:31:20,376 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
streamitems,^...@^@^...@^@^...@�^v�,1229470182375 open on 72.34.249.218:60020
2008-12-16 15:31:20,376 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
streamitems,^...@^@^...@^@^...@�^v�,1229470182375 in region .META.,,1 with
startcode 1229369663539 and server 72.34.249.218:60020
2008-12-16 15:31:21,745 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^B�|s,1229459878511 to server
72.34.249.212:60020
2008-12-16 15:31:21,746 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^B�^Us,1229459878511 to server
72.34.249.212:60020
2008-12-16 15:31:21,747 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region commentchunks,,1229364208367 to server 72.34.249.212:60020
2008-12-16 15:31:21,747 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^B�B�,1229459764918 to server
72.34.249.212:60020
2008-12-16 15:31:21,747 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^...@xƙ,1229460120667 to server
72.34.249.212:60020
2008-12-16 15:31:21,748 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^...@r��,1229459815399 to server
72.34.249.212:60020
2008-12-16 15:31:21,748 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^B7Oi,1229374367471 to server
72.34.249.212:60020
2008-12-16 15:31:21,749 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^C|��,1229375922174 to server
72.34.249.212:60020
2008-12-16 15:31:21,749 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^C�n�,1229459949444 to server
72.34.249.212:60020
2008-12-16 15:31:21,749 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^...@^z�q,1229375049350 to server
72.34.249.212:60020
2008-12-16 15:31:22,101 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^...@�p-,1229460052735 to server
72.34.249.215:60020
2008-12-16 15:31:22,101 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^A�n/,1229373923594 to server
72.34.249.215:60020
2008-12-16 15:31:22,102 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^...@��[,1229460052735 to server
72.34.249.215:60020
2008-12-16 15:31:22,102 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^...@uu1,1229460120667 to server
72.34.249.215:60020
2008-12-16 15:31:22,103 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^A��,1229373923594 to server
72.34.249.215:60020
2008-12-16 15:31:22,103 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^B:7�,1229374367471 to server
72.34.249.215:60020
2008-12-16 15:31:22,103 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^C���,1229375603887 to server
72.34.249.215:60020
2008-12-16 15:31:22,104 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^...@.@3,1229374602767 to server
72.34.249.215:60020
2008-12-16 15:31:22,104 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^...@�^v�,1229470182375 to server
72.34.249.215:60020
2008-12-16 15:31:22,105 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^C^E(!,1229459984826 to server
72.34.249.215:60020
2008-12-16 15:31:22,119 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^C^Bi�,1229459984826 to server
72.34.249.219:60020
2008-12-16 15:31:22,119 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^...@eϭ,1229374034359 to server
72.34.249.219:60020
2008-12-16 15:31:22,120 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^B^Y/a,1229374146691 to server
72.34.249.219:60020
2008-12-16 15:31:22,120 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^B��,1229459764918 to server
72.34.249.219:60020
2008-12-16 15:31:22,120 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^...@pz�,1229459815399 to server
72.34.249.219:60020
2008-12-16 15:31:22,121 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^C�hg,1229375603887 to server
72.34.249.219:60020
2008-12-16 15:31:22,121 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^...@k^g/,1229369754111 to server
72.34.249.219:60020
2008-12-16 15:31:22,122 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^AfB^Q,1229459739597 to server
72.34.249.219:60020
2008-12-16 15:31:22,122 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^C^?�+,1229375922174 to server
72.34.249.219:60020
2008-12-16 15:31:22,123 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region sourceurls,,1229364124122 to server 72.34.249.219:60020
2008-12-16 15:31:22,239 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>, server: 72.34.249.216:60020}
2008-12-16 15:31:22,300 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^...@^@^...@^@^...@�^d;,1229470182375 to server
72.34.249.210:60020
{code}
Here is lifetime of the regions of rid 1229470182375
{code}
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:29:45,601 INFO
org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT:
streamitems,^...@^@^...@^@^...@�^v�,1229374149929:
streamitems,^...@^@^...@^@^...@�^v�,1229374149929 split; daughters:
streamitems,^...@^@^...@^@^...@�^v�,1229470182375,
streamitems,^...@^@^...@^@^...@�^d;,1229470182375 from XX.XX.XX.217:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:29:51,115 INFO
org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^...@^@^...@^@^...@�^d;,1229470182375 to server XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:29:51,119 INFO
org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^...@^@^...@^@^...@�^v�,1229470182375 to server XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:29:54,126 INFO
org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
streamitems,^...@^@^...@^@^...@�^v�,1229470182375 from XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:29:54,126 INFO
org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
streamitems,^...@^@^...@^@^...@�^d;,1229470182375 from XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:29:54,126 INFO
org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
streamitems,^...@^@^...@^@^...@�^v�,1229470182375 from XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:30:22,421 DEBUG
org.apache.hadoop.hbase.master.BaseScanner:
streamitems,^...@^@^...@^@^...@�^v�,1229470182375 no longer has references to
streamitems,^...@^@^...@^@^...@�^v�,1229374149929
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:30:22,423 DEBUG
org.apache.hadoop.hbase.master.BaseScanner:
streamitems,^...@^@^...@^@^...@�^d;,1229470182375 no longer has references to
streamitems,^...@^@^...@^@^...@�^v�,1229374149929
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:20,288 DEBUG
org.apache.hadoop.hbase.master.RegionServerOperation: shutdown scanner looking
at streamitems,^...@^@^...@^@^...@�^v�,1229470182375
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:20,289 DEBUG
org.apache.hadoop.hbase.master.RegionServerOperation: shutdown scanner looking
at streamitems,^...@^@^...@^@^...@�^d;,1229470182375
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:20,375 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
streamitems,^...@^@^...@^@^...@�^d;,1229470182375 open on XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:20,375 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
streamitems,^...@^@^...@^@^...@�^d;,1229470182375 in region .META.,,1 with
startcode 1229369663539 and server XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:20,376 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
streamitems,^...@^@^...@^@^...@�^v�,1229470182375 open on XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:20,376 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
streamitems,^...@^@^...@^@^...@�^v�,1229470182375 in region .META.,,1 with
startcode 1229369663539 and server XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:22,104 INFO
org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^...@^@^...@^@^...@�^v�,1229470182375 to server XX.XX.XX.215:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:22,300 INFO
org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^...@^@^...@^@^...@�^d;,1229470182375 to server XX.XX.XX.210:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:25,113 INFO
org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
streamitems,^...@^@^...@^@^...@�^v�,1229470182375 from XX.XX.XX.215:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:25,334 INFO
org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
streamitems,^...@^@^...@^@^...@�^d;,1229470182375 from XX.XX.XX.210:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:25,335 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
streamitems,^...@^@^...@^@^...@�^d;,1229470182375 open on XX.XX.XX.210:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:25,335 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
streamitems,^...@^@^...@^@^...@�^d;,1229470182375 in region .META.,,1 with
startcode 1229369663563 and server XX.XX.XX.210:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:28,129 INFO
org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
streamitems,^...@^@^...@^@^...@�^v�,1229470182375 from XX.XX.XX.215:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:31,142 INFO
org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
streamitems,^...@^@^...@^@^...@�^v�,1229470182375 from XX.XX.XX.215:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:31,143 INFO
org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
streamitems,^...@^@^...@^@^...@�^v�,1229470182375 from XX.XX.XX.215:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:31,145 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
streamitems,^...@^@^...@^@^...@�^v�,1229470182375 open on XX.XX.XX.215:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:31,145 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
streamitems,^...@^@^...@^@^...@�^v�,1229470182375 in region .META.,,1 with
startcode 1229369663604 and server XX.XX.XX.215:60020
{code}
> HStore out of sync with what is on filesystem
> ---------------------------------------------
>
> Key: HBASE-1073
> URL: https://issues.apache.org/jira/browse/HBASE-1073
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.19.0
>
>
> On streamy cluster, TRUNK, I saw this when we went to major compact:
> {code}
> 2008-12-17 16:21:48,725 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
> Major compaction triggered on store: 657002096/content. Time since last major
> compaction: 89517 seconds
> 2008-12-17 16:21:48,842 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
> Started compaction of 1 file(s) into
> /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172
> 2008-12-17 16:21:56,354 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172/data
> File does not exist. [Lease. Holder: DFSClient_-425857031, pendingcreates:
> 1]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1331)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1322)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1250)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)
> at org.apache.hadoop.ipc.Client.call(Client.java:696)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
> at $Proxy1.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy1.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2815)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2697)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2008-12-17 16:21:56,354 WARN org.apache.hadoop.hdfs.DFSClient:
> NotReplicatedYetException sleeping
> /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172/data
> retries left 4
> 2008-12-17 16:21:56,757 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172/data
> File does not exist. [Lease. Holder: DFSClient_-425857031, pendingcreates:
> 1]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1331)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1322)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1250)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)
> at org.apache.hadoop.ipc.Client.call(Client.java:696)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
> at $Proxy1.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy1.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2815)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2697)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2008-12-17 16:21:56,757 WARN org.apache.hadoop.hdfs.DFSClient:
> NotReplicatedYetException sleeping
> /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172/data
> retries left 3
> 2008-12-17 16:21:57,561 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172/data
> File does not exist. [Lease. Holder: DFSClient_-425857031, pendingcreates:
> 1]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1331)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1322)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1250)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)
> at org.apache.hadoop.ipc.Client.call(Client.java:696)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
> at $Proxy1.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy1.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2815)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2697)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2008-12-17 16:21:57,561 WARN org.apache.hadoop.hdfs.DFSClient:
> NotReplicatedYetException sleeping
> /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172/data
> retries left 2
> 2008-12-17 16:21:59,162 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172/data
> File does not exist. [Lease. Holder: DFSClient_-425857031, pendingcreates:
> 1]
> ...
> {code}
> Thereafter, an attempt to open a scanner on region fails...
> {code}
> 2008-12-18 00:41:03,964 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> compaction completed on region streamitems,^...@^@^...@^@^Bv�^K,1229373844647
> in 0sec
> 2008-12-18 01:08:50,274 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> java.io.IOException: HStoreScanner failed construction
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:70)
> at
> org.apache.hadoop.hbase.regionserver.HStoreScanner.<init>(HStoreScanner.java:84)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2120)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1978)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1160)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1581)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:892)
> Caused by: java.io.FileNotFoundException: File does not exist:
> hdfs://mb0:9000/hbase/streamitems/657002096/content/mapfiles/2509136481693763805/data
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:679)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
> at
> org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:302)
> at
> org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:284)
> at
> org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:81)
> at
> org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:66)
> at
> org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:67)
> ... 10 more
> ...
> {code}
> The first error seems to mess up our view of whats on the filesystem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
