Hi Jon: Please ignore my last email. We found it was a bug, fix it by a patch and rebuild, and it works now. Data are back! Thanks.
Best, Arber On Sun, Apr 15, 2012 at 12:47 PM, Yabo Xu <[email protected]> wrote: > Dear Jon: > > We just ran OfflineMetaRepair, while getting the following exceptions. > Checked online...it seems that is bug. Any suggestions on how to check out > the most-updated version of OfflineMetaRepair to work with our version of > HBase? Thanks in advance. > > 12/04/15 12:28:35 INFO util.HBaseFsck: Loading HBase regioninfo from > HDFS... > 12/04/15 12:28:39 ERROR util.HBaseFsck: Bailed out due to: > java.lang.IllegalArgumentException: Wrong FS: hdfs:// > n4.example.com:12345/hbase/summba.yeezhao.content/03cde9116662fade27545d86ea71a372/.regioninfo, > expected: file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:310) > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:357) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125) > at > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356) > at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256) > at > org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284) > at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402) > at > org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90) > > We checked on hdfs, and the files shown in exception are available. Any > point > > Best, > Arber > > > On Sun, Apr 15, 2012 at 11:48 AM, Yabo Xu <[email protected]>wrote: > >> Thanks, St. Ack & Jon. To answer St. Ack's question, we are using HBase >> 0.90.6, and the data corruption happens when some data nodes are lost due >> to the power issue. We've tried hbck and it reports that ROOT is not found, >> and hfsk reports two blocks of ROOT and META are CORUPT status. >> >> Jon: We just checked OfflineMetaRepair, it seems to be the right tool, >> and is trying it now. Just want to confirm: is it compatible with 0.90.6? >> >> Best, >> Arber >> >> >> On Sun, Apr 15, 2012 at 8:55 AM, Jonathan Hsieh <[email protected]> wrote: >> >>> There is a two tools that can try to help you (unfortunately, I haven't >>> written the user documentation for either yet) >>> >>> One is called OfflineMetaRepair. This assumes that hbase is offline >>> reads >>> the data in HDFS to create a new ROOT and new META. If you data is in >>> good shape, this should work for you. Depending on which version of >>> hadoop >>> you are using, you may need to apply HBASE-5488. >>> >>> On the latest branches of hbase (0.90/0.92/0.94/trunk) the hbck tool has >>> been greatly enhanced and may be able to help out as well once an initial >>> META table is built, and your hbase is able to get online. This will >>> currently will require a patch HBASE-5781 to be applied to be useful. >>> >>> Jon. >>> >>> >>> On Sat, Apr 14, 2012 at 1:35 PM, Yabo Xu <[email protected]> >>> wrote: >>> >>> > Hi all: >>> > >>> > Just had a desperate night....We had a small production hbase >>> cluster( 8 >>> > nodes), and due to the accident crash of a few nodes, ROOT and META are >>> > corrupted, while the rest of tables are mostly there. Are there any >>> way to >>> > restore ROOT and META? >>> > >>> > Any of the hints would be appreciated very much! Waiting on line... >>> > >>> > Best, >>> > Arber >>> > >>> >>> >>> >>> -- >>> // Jonathan Hsieh (shay) >>> // Software Engineer, Cloudera >>> // [email protected] >>> >> >> >
