[
https://issues.apache.org/jira/browse/HBASE-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617989#action_12617989
]
Jonathan Gray commented on HBASE-766:
-------------------------------------
stack, please confirm if i need to apply both patches or just hs.patch.
Have actually tested with both patches, neither seems to change behavior at all.
Again, happens during the second split of the problem table:
2008-07-29 15:11:51,677 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 426199566/content store size is 378.0m
2008-07-29 15:11:51,684 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
items,0050dab2-a057-43d9-8e99-0a41a3e6bdbb,1217369332384 in 7sec
2008-07-29 15:11:51,686 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region
items,0050dab2-a057-43d9-8e99-0a41a3e6bdbb,1217369332384
2008-07-29 15:11:51,689 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Compactions and cache flushes disabled for region
items,0050dab2-a057-43d9-8e99-0a41a3e6bdbb,1217369332384
2008-07-29 15:11:51,689 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates and scanners disabled for region
items,0050dab2-a057-43d9-8e99-0a41a3e6bdbb,1217369332384
2008-07-29 15:11:51,689 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No
more active scanners for region
items,0050dab2-a057-43d9-8e99-0a41a3e6bdbb,1217369332384
2008-07-29 15:11:51,689 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No
more row locks outstanding on region
items,0050dab2-a057-43d9-8e99-0a41a3e6bdbb,1217369332384
2008-07-29 15:11:51,689 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memcache flush for region
items,0050dab2-a057-43d9-8e99-0a41a3e6bdbb,1217369332384. Current region
memcache size 1.9m
2008-07-29 15:11:51,781 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Added /hbase/items/426199566/content/mapfiles/7100309375208596278 with 10296
entries, sequence id 2537536, data size 1.9m, file size 2.1m
2008-07-29 15:11:51,781 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region
items,0050dab2-a057-43d9-8e99-0a41a3e6bdbb,1217369332384 in 92ms, sequence
id=2537536, compaction requested=true
2008-07-29 15:11:51,781 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 426199566/uclusters
2008-07-29 15:11:51,781 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 426199566/covisit
2008-07-29 15:11:51,781 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 426199566/content
2008-07-29 15:11:51,781 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 426199566/clusters
2008-07-29 15:11:51,781 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 426199566/readby
2008-07-29 15:11:51,781 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 426199566/cfrecs
2008-07-29 15:11:51,781 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 426199566/blooms
2008-07-29 15:11:51,781 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 426199566/ranked
2008-07-29 15:11:51,782 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 426199566/savedby
2008-07-29 15:11:51,782 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 426199566/sentby
2008-07-29 15:11:51,782 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 426199566/receivedby
2008-07-29 15:11:51,782 INFO org.apache.hadoop.hbase.regionserver.HRegion:
closed items,0050dab2-a057-43d9-8e99-0a41a3e6bdbb,1217369332384
2008-07-29 15:11:52,033 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region
items,0050dab2-a057-43d9-8e99-0a41a3e6bdbb,1217369511688/509041795
2008-07-29 15:11:52,049 INFO org.apache.hadoop.hbase.regionserver.HStore:
HSTORE_LOGINFOFILE
509041795/savedby/4080673342751735986-426199566/8532274243649733439/bottom does
not contain a sequence number - ignoring
2008-07-29 15:11:52,052 ERROR
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for
region items,0050dab2-a057-43d9-8e99-0a41a3e6bdbb,1217369332384
java.io.FileNotFoundException: File does not exist:
hdfs://mb0:9000/hbase/items/426199566/savedby/mapfiles/8532274243649733439/data
at
org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:369)
at
org.apache.hadoop.hbase.regionserver.HStoreFile.length(HStoreFile.java:444)
at
org.apache.hadoop.hbase.regionserver.HStore.loadHStoreFiles(HStore.java:437)
at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:219)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1618)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:466)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:405)
at
org.apache.hadoop.hbase.regionserver.HRegion.splitRegion(HRegion.java:800)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplitThread.java:133)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:86)
> FileNotFoundException trying to load HStoreFile 'data'
> ------------------------------------------------------
>
> Key: HBASE-766
> URL: https://issues.apache.org/jira/browse/HBASE-766
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: 766.patch, hs.patch
>
>
> From Renaud Delbru up on the list:
> {code}
> Hi,
> after our issues ("Replay of HLog required", in a precious thread) with
> HBase, it seems that HBase has corrupted regions.
> We have, on the three region servers, errors stating that HBase cannot open
> certain regions because some map files on hdfs are missing (see the log
> attached).
> Do you have any ideas how to fix this ?
> Thanks.
> java.io.FileNotFoundException: File does not exist:
> hdfs://hadoop1.sindice.net:54310/hbase/page-repository/1105668475/field/mapfiles/5122893264992435570/data
> at
> org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:369)
> at
> org.apache.hadoop.hbase.regionserver.HStoreFile.length(HStoreFile.java:464)
> at
> org.apache.hadoop.hbase.regionserver.HStore.loadHStoreFiles(HStore.java:409)
> at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:236)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1575)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:451)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:901)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:876)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:816)
> at java.lang.Thread.run(Thread.java:619)
> {code}
> I'd thought we'd added handling of this kind of event post HDFS crashings but
> looking in code, it seems that TRUNK may not have been fixed up properly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
