Hello, We are trying to recover a corrupt META table over here. The region server holding the META table went down, and since then our table scans cannot complete because of this error which happens consistently on the same region:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server d3.colo.networkedinsights.com:60020 for region feedData,20100807 8130b47666b2362736f7d4a72320fc4eb9571fe4, 1281690917171.b9f985a5382f8b4db594fc1725bd4322., row '20100807 8130b47666b2362736f7d4a72320fc4eb9571fe4', but failed after 10 attempts. Exceptions: java.io.IOException: java.io.IOException: Could not seek StoreFileScanner[HFileScanner for reader reader= hdfs://n1.colo/hbase/feedData/ b9f985a5382f8b4db594fc1725bd4322/core/8130141976997173515, compression=lzo, inMemory=false, firstKey=20100807 81380c5178aaee066514daee610964f880979ce5/core:content.date/1281706679432/Put, lastKey=20100807 9a8467d77b0a9371dd588f53191977d4da799c94/core:report.topics/1281706664933/Put, avgKeyLen=80, avgValueLen=51, entries=6184, length=228335, cur=null] When looking on HDFS for that region, it does not exist! That is obviously the problem, but our question is, why does the META table reference a non-existent region? We tried to rebuild the META table using add_table.rb, and we recieved WARN messages stating .regioninfo was missing on almost half of the regions. Here is the output of add_table.rb: http://pastebin.com/T2kWKtuH We are using CDH3b2 with 3 region servers. writeToWAL is false for all our puts as we are importing this data, could that be related to our problem ? Thanks, Luke Forehand http://www.networkedinsights.com
