This sounds like a bug. Which version of HBase is this. Could you file a bug?
Thanks. -- Lars ________________________________ From: Kyle McGovern <[email protected]> To: [email protected] Sent: Sunday, December 9, 2012 7:09 PM Subject: Re: hbase corruption - missing region files in HDFS We recently had a very similar issue on a couple of our clusters. What ended up happening was a split failed and there was a leftover file in the region telling it where the new split region was located. The destination region folder/file did not exist so our region server would try endlessly to read a file that didn't exist. The end result was exhaustion of open file descriptors for the region server due to the number of connections it was making. Our fix was to remove the bad "split file" and assign the region again. 15:38:21 # hdfs dfs -ls -R /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a drwxr-xr-x - root hadoop 0 2012-12-07 13:21 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/.oldlogs -rw-r--r-- 3 root hadoop 124 2012-12-07 13:21 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/.oldlogs/hlog.1354760917669 -rw-r--r-- 3 root hadoop 352 2012-12-07 13:27 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/.regioninfo drwxr-xr-x - root hadoop 0 2012-12-07 13:27 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW -rw-r--r-- 3 root hadoop 554522 2012-12-07 13:27 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/195cc6d2cc384b39bd5ad30e95385bd8 -rw-r--r-- 3 root hadoop 4558378 2012-12-07 13:27 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/1c42fa9bc26a4550a439f4bd31bb08b0 -rw-r--r-- 3 root hadoop 3498028 2012-12-07 13:27 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/28a356081046422b8c057bc20c0ae658 -rw-r--r-- 3 root hadoop 1948108 2012-12-07 13:27 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/3353dc2d99184fe4b9d73f39503dfbc7 -rw-r--r-- 3 root hadoop 4390731 2012-12-07 12:01 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/4ce59f31c1b74db5804953fa7967f791 -rw-r--r-- 3 root hadoop 3116921421 2012-12-07 12:22 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/5313858989b24752ae31322333de02e0 -rw-r--r-- 3 root hadoop 5395692 2012-12-07 12:22 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/54c11a7e4f9d4ebfafaf2b93d3c9e954 -rw-r--r-- 3 root hadoop 5981971640 2012-12-07 13:20 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/5d965eba35df44d2851a8186fe6e8cc8 -rw-r--r-- 3 root hadoop 23 2012-12-07 13:20 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/5d965eba35df44d2851a8186fe6e8cc8.7d4f7401d2fe7a813778248970b03515 -rw-r--r-- 3 root hadoop 2251800 2012-12-07 13:20 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/673b36462014480cb7d91088412b85a7 -rw-r--r-- 3 root hadoop 408794 2012-12-07 13:20 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/73261dd86f634f2086ec745642425d7c -rw-r--r-- 3 root hadoop 2676245 2012-12-07 13:20 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/769728d25b5b4e78be6b36f9716a82c4 -rw-r--r-- 3 root hadoop 1262744 2012-12-07 13:20 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/81f414cb3fe449f6a80310dd38ea467f -rw-r--r-- 3 root hadoop 940502 2012-12-07 13:20 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/8f818b3c45344ad68c0b4afc7fe20bbb -rw-r--r-- 3 root hadoop 3492843 2012-12-07 13:20 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/ae7cb412e5da4a908b0f2ea4d5cd5c76 -rw-r--r-- 3 root hadoop 2894474 2012-12-07 12:01 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/b6ee14a0a75341d0aa58187fb6159a41 -rw-r--r-- 3 root hadoop 14257782 2012-12-07 12:01 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/bd4fff3291d647eb9cc533d66f9685a3 -rw-r--r-- 3 root hadoop 4880699 2012-12-07 12:01 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/c4d3f1c8511743579588162616beeea1 -rw-r--r-- 3 root hadoop 35238595 2012-12-07 12:01 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/c69a406d54b1492ba52cd296de8320a1 -rw-r--r-- 3 root hadoop 23 2012-12-07 12:01 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/c69a406d54b1492ba52cd296de8320a1.7d4f7401d2fe7a813778248970b03515 -rw-r--r-- 3 root hadoop 3181138002 2012-12-07 12:20 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/cad9f4cc0ef54a7896a3a47253250e71 -rw-r--r-- 3 root hadoop 1747856 2012-12-07 12:20 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/cca2ad1698984a73abd9c58c78945be0 -rw-r--r-- 3 root hadoop 6264897732 2012-12-07 13:21 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/d876f1f4734e4778b2efa527ef1ef3ee -rw-r--r-- 3 root hadoop 463704 2012-12-07 13:21 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/f2efc4a6ec054a62a44f664cc0b01c0a -rw-r--r-- 3 root hadoop 686868 2012-12-07 13:21 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/f34384ae8c1d4e16afb79cb41bf6cf74 -rw-r--r-- 3 root hadoop 838234 2012-12-07 13:21 /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/fc1dc425cf324beaa283ef82fdc073e3 For example, if I remove /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/c69a406d54b1492ba52cd296de8320a1.7d4f7401d2fe7a813778248970b03515 and /hbase/mytable/3ff87b4b16037f2000f4f4fb1bae820a/RAW/5d965eba35df44d2851a8186fe6e8cc8.7d4f7401d2fe7a813778248970b03515 the region will successfully assign and hbck does not show errors for this region anymore. The contents of the file appear to just be a split key.
