[
https://issues.apache.org/jira/browse/HBASE-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618569#action_12618569
]
Jonathan Gray commented on HBASE-766:
-------------------------------------
Stack,
I used the jar you built, confirmed across the cluster it was the one being
used, and ran the test... I don't see any additional logging. Check out the
region logs on hb5 (where items table was located).
Here is the error again:
2008-07-30 17:41:58,970 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memcache flush for region
items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950. Current region
memcache size 64.0m
2008-07-30 17:42:00,424 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Added /hbase/items/1211238171/content/mapfiles/793072914512989834 with 323136
entries, sequence id 2527714, data size 64.0m, file size 68.8m
2008-07-30 17:42:00,424 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region
items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950 in 1454ms, sequence
id=2527714, compaction requested=true
2008-07-30 17:42:00,424 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested
for region: items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:00,424 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region
items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:00,428 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Compaction size 396611811, skipped 1, 252131417
2008-07-30 17:42:00,454 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
started compaction of 2 files into
/hbase/items/compaction.dir/1211238171/content/mapfiles/4240641699209031496
2008-07-30 17:42:08,256 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/items/compaction.dir/1211238171/content/mapfiles/4240641699209031496 to
/hbase/items/1211238171/content/mapfiles/8894612533024691980
2008-07-30 17:42:08,293 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 1211238171/content store size is 377.8m
2008-07-30 17:42:08,299 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950 in 7sec
2008-07-30 17:42:08,301 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region
items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,304 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Compactions and cache flushes disabled for region
items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,304 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates and scanners disabled for region
items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,304 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No
more active scanners for region
items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,305 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No
more row locks outstanding on region
items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,305 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memcache flush for region
items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950. Current region
memcache size 1.9m
2008-07-30 17:42:08,406 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Added /hbase/items/1211238171/content/mapfiles/2539623802632323222 with 10032
entries, sequence id 2537747, data size 1.9m, file size 2.0m
2008-07-30 17:42:08,406 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region
items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950 in 101ms, sequence
id=2537747, compaction requested=true
2008-07-30 17:42:08,406 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1211238171/uclusters
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1211238171/covisit
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1211238171/content
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1211238171/clusters
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1211238171/readby
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1211238171/cfrecs
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1211238171/blooms
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1211238171/ranked
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1211238171/savedby
2008-07-30 17:42:08,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1211238171/sentby
2008-07-30 17:42:08,408 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1211238171/receivedby
2008-07-30 17:42:08,408 INFO org.apache.hadoop.hbase.regionserver.HRegion:
closed items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
2008-07-30 17:42:08,638 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region
items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464928303/2132462761
2008-07-30 17:42:08,650 INFO org.apache.hadoop.hbase.regionserver.HStore:
HSTORE_LOGINFOFILE
2132462761/readby/8482419619225016670-1211238171/6573863319629365608/bottom
does not contain a sequence number - ignoring
2008-07-30 17:42:08,653 ERROR
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for
region items,0050a173-36b8-45a4-a991-2bab7f5c3cd6,1217464581950
java.io.FileNotFoundException: File does not exist:
hdfs://mb0:9000/hbase/items/1211238171/readby/mapfiles/6573863319629365608/data
at
org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:369)
at
org.apache.hadoop.hbase.regionserver.HStoreFile.length(HStoreFile.java:444)
at
org.apache.hadoop.hbase.regionserver.HStore.loadHStoreFiles(HStore.java:437)
at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:219)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1618)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:466)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:405)
at
org.apache.hadoop.hbase.regionserver.HRegion.splitRegion(HRegion.java:800)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplitThread.java:133)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:86)
> FileNotFoundException trying to load HStoreFile 'data'
> ------------------------------------------------------
>
> Key: HBASE-766
> URL: https://issues.apache.org/jira/browse/HBASE-766
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: 766.patch, hs.patch
>
>
> From Renaud Delbru up on the list:
> {code}
> Hi,
> after our issues ("Replay of HLog required", in a precious thread) with
> HBase, it seems that HBase has corrupted regions.
> We have, on the three region servers, errors stating that HBase cannot open
> certain regions because some map files on hdfs are missing (see the log
> attached).
> Do you have any ideas how to fix this ?
> Thanks.
> java.io.FileNotFoundException: File does not exist:
> hdfs://hadoop1.sindice.net:54310/hbase/page-repository/1105668475/field/mapfiles/5122893264992435570/data
> at
> org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:369)
> at
> org.apache.hadoop.hbase.regionserver.HStoreFile.length(HStoreFile.java:464)
> at
> org.apache.hadoop.hbase.regionserver.HStore.loadHStoreFiles(HStore.java:409)
> at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:236)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1575)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:451)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:901)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:876)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:816)
> at java.lang.Thread.run(Thread.java:619)
> {code}
> I'd thought we'd added handling of this kind of event post HDFS crashings but
> looking in code, it seems that TRUNK may not have been fixed up properly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
