Hi Guys -
This is a fairly complete little Tool (Configured) whose purpose is to move out
a whole slew of regions into a backup directory and restore .META. when done.
We found that we needed to do this when a huge volume of keys had been
generated into a production table, and it turned out the whole set of keys had
an incorrect prefix. Thus, what we really wanted to do was move the data out of
all the regions into some backup directory in one fell swoop. This tool accepts
some parameters with -D (hadoop arguments). It will remove a slew of contiguous
regions, relink the .META., and place the removed data in a backup directory in
HDFS. It has been tested on big tables and includes some more subtle "gotchas"
catches like being careful when parsing region names, to guard against rowkeys
actually containing commas. It worked for me, but use at your own risk.
Basically you give it -Dregion.remove.regionname.start=STARTREGION and
region.remove.regionname.end=ENDREGION and all the data between STARTREGION and
ENDREGION will be moved out of your table, where STARTREGION and ENDREGION are
region names.
import java.io.IOException;
import java.io.InputStream;
import java.util.Iterator;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.NotServingRegionException;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.FSUtils;
import org.apache.hadoop.hbase.util.Writables;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
/**
* @author ghendrey
*/
public class RemoveRegions extends Configured implements Tool {
public static void main(String[] args) throws Exception {
int exitCode = ToolRunner.run(new RemoveRegions(), args);
System.exit(exitCode);
}
private static void deleteMetaRow(HRegionInfo closedRegion, HTable
hMetaTable) throws IOException {
Delete del = new Delete(closedRegion.getRegionName()); //Delete the
original row from .META.
hMetaTable.delete(del);
System.out.println("Deleted the region's row from .META. " +
closedRegion.getRegionNameAsString());
}
private static HRegionInfo closeRegion(Result result, HBaseAdmin admin)
throws RuntimeException, IOException {
byte[] bytes = result.getValue(HConstants.CATALOG_FAMILY,
HConstants.REGIONINFO_QUALIFIER);
HRegionInfo closedRegion = Writables.getHRegionInfo(bytes);
try {
admin.closeRegion(closedRegion.getRegionName(), null); //. Close
the existing region if open.
System.out.println("Closed the Region " +
closedRegion.getRegionNameAsString());
} catch (Exception nse) {
System.out.println("Skipped closing the region because: " +
nse.getMessage());
}
return closedRegion;
}
private static HRegionInfo getRegionInfo(String exclusiveStartRegionName,
Configuration hConfig) throws IOException {
HTable readTable = new HTable(hConfig, Bytes.toBytes(".META."));
Get readGet = new Get(Bytes.toBytes(exclusiveStartRegionName));
Result readResult = readTable.get(readGet);
byte[] readBytes = readResult.getValue(HConstants.CATALOG_FAMILY,
HConstants.REGIONINFO_QUALIFIER);
HRegionInfo regionInfo = Writables.getHRegionInfo(readBytes); //Read
the existing hregioninfo.
System.out.println("got region info: " + regionInfo);
return regionInfo;
}
private static void createBackupDir(Configuration conf) throws IOException {
String path = conf.get("region.remove.backupdir", "regionBackup-" +
System.currentTimeMillis());
Path backupDirPath = new Path(path);
FileSystem fs = backupDirPath.getFileSystem(conf);
FSUtils.DirFilter dirFilt = new FSUtils.DirFilter(fs);
System.out.println("creating backup dir: " + backupDirPath.toString());
fs.mkdirs(backupDirPath);
}
public int run(String[] strings) throws Exception {
try {
System.setProperty("javax.xml.parsers.DocumentBuilderFactory",
"com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl");
Configuration conf = getConf();
Configuration hConfig = HBaseConfiguration.create(conf);
hConfig.set("hbase.zookeeper.quorum",
System.getProperty("hbase.zookeeper.quorum",
"doop2.dt.sv4.decarta.com,doop3.dt.sv4.decarta.com,doop4.dt.sv4.decarta.com,doop5.dt.sv4.decarta.com,doop7.dt.sv4.decarta.com,doop8.dt.sv4.decarta.com,doop9.dt.sv4.decarta.com,doop10.dt.sv4.decarta.com"));
HBaseAdmin admin = new HBaseAdmin(hConfig);
HBaseAdmin.checkHBaseAvailable(hConfig);
System.out.println("regions will be moved out from between
region.remove.regionname.start and region.remove.regionname.end (exclusive)");
String exclusiveStartRegionName =
conf.get("region.remove.regionname.start");
if (null == exclusiveStartRegionName) {
throw new RuntimeException("Current implementation requires an
exclusive region.remove.regionname.start");
}
System.out.println("region.remove.regionname.start=" +
exclusiveStartRegionName);
String exclusiveEndRegionName =
conf.get("region.remove.regionname.end");
if (null == exclusiveEndRegionName) {
throw new RuntimeException("Current implementation requires an
exclusive region.remove.endrow");
}
System.out.println("region.remove.regionname.end=" +
exclusiveEndRegionName);
//CREATE A BACKUP DIR FOR THE REGION DATA TO BE MOVED INTO
createBackupDir(hConfig);
Path hbaseRootPath = FSUtils.getRootDir(hConfig);
if (null == hbaseRootPath) {
throw new RuntimeException("couldn't determine hbase root dir");
} else {
System.out.println("hbase rooted at " +
hbaseRootPath.toString());
}
HTable hMetaTable = new HTable(hConfig, Bytes.toBytes(".META."));
System.out.println("connected to .META.");
//get region info for start and end regions
HRegionInfo exclusiveStartRegionInfo =
getRegionInfo(exclusiveStartRegionName, hConfig);
HRegionInfo exclusiveEndRegionInfo =
getRegionInfo(exclusiveEndRegionName, hConfig);
//CLOSE all the regions starting with the exclusiveStartRegionName
(including it), and up to but excluding closing the exclusiveEndRegionName
//and DELETE rows from .META.
Scan scan = new Scan(Bytes.toBytes(exclusiveStartRegionName),
Bytes.toBytes(exclusiveEndRegionName));
ResultScanner metaScanner = hMetaTable.getScanner(scan);
int i = 0;
for (Iterator<Result> iter = metaScanner.iterator();
iter.hasNext();) {
Result res = iter.next();
//CLOSE REGION
HRegionInfo closedRegion = closeRegion(res, admin);
//MOVE ACTUAL DATA OUT OF HBASE HDFS INTO BACKUP AREA
moveDataToBackup(closedRegion, hConfig);
//DELETE ROW FROM META TABLE
deleteMetaRow(closedRegion, hMetaTable);
}
//now reinsert the startrow into .META. with it's endrow pointing
to the startrow of the exclusiveEndRegionInfo
//This effectively "relinks" the link list of .META., now that all
the interstitial region-rows have been removed from .META.
relinkStartRow(exclusiveStartRegionInfo, exclusiveEndRegionInfo,
hConfig, admin);
return 0;
} catch (Exception ex) {
throw new RuntimeException(ex.getMessage(), ex);
}
}
private void relinkStartRow(HRegionInfo exclusiveStartRegionInfo,
HRegionInfo exclusiveEndRegionInfo, Configuration hConfig, HBaseAdmin admin)
throws IllegalArgumentException, IOException {
//Now we are going to recreate the region info for
exclusiveStartRegion, such that it's endKey points to the startKey
//of the exclusiveEndRegion.
HTableDescriptor descriptor = new
HTableDescriptor(exclusiveStartRegionInfo.getTableDesc()); //Use existing
hregioninfo htabledescriptor and this construction
// Just changing the End key , nothing else. This performs the "unlink"
step
byte[] startKey = exclusiveStartRegionInfo.getStartKey();
byte[] endKey = exclusiveEndRegionInfo.getStartKey();
HRegionInfo newStartRegion = new HRegionInfo(descriptor, startKey,
endKey);
byte[] value = Writables.getBytes(newStartRegion);
Put put = new Put(newStartRegion.getRegionName()); // Same time stamp
from the record.
put.add(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER,
value); //Insert the new entry in .META. using new hregioninfo name as row key
and add an info:regioninfo whose contents is the serialized new hregioninfo.
HTable metaTable = new HTable(hConfig, ".META.");
metaTable.put(put);
System.out.println("New row in .META.: " +
newStartRegion.getRegionNameAsString() + " End key is " +
Bytes.toString(exclusiveEndRegionInfo.getStartKey()));
admin.assign(newStartRegion.getRegionName(), true); //Assign the new
region.
System.out.println("Assigned the new region " +
newStartRegion.getRegionNameAsString());
}
private static void moveDataToBackup(HRegionInfo closedRegion,
Configuration conf) throws IOException {
Path rootPath = FSUtils.getRootDir(conf);
String tablename = closedRegion.getRegionNameAsString().split(",")[0];
//split regionname on comma. tablename comes before first comma
Path tablePath = new Path(rootPath, tablename);
String[] dotSplit = closedRegion.getRegionNameAsString().split("\\.",
0);
String regionId = dotSplit[dotSplit.length - 1]; //split regionname on
dot. regionId between last two dots
Path regionPath = new Path(tablePath, regionId);
System.out.println(regionPath);
FileSystem fs = FileSystem.get(conf);
Path regionBackupPath = new Path(conf.get("region.remove.backupdir",
"regionBackup-" + System.currentTimeMillis()) + "/" + regionId);
//Path regionBackupPath = new Path(backupPath, regionId);
System.out.println("moving to: " + regionBackupPath);
fs.rename(regionPath, regionBackupPath);
}
}
-----Original Message-----
From: Stuart Smith [mailto:[email protected]]
Sent: Saturday, October 29, 2011 1:39 PM
To: [email protected]
Subject: Re: PENDING_CLOSE for too long
Hello Geoff,
I usually don't show up here, since I use CDH, and good form means I should
stay on CDH-users,
But!
I've been seeing the same issues for months:
- PENDING_CLOSE too long, master tries to reassign - I see an continuous
stream of these.
- WrongRegionExceptions due to overlapping regions & holes in the regions.
I just spent all day yesterday cribbing off of St.Ack's check_meta.rb script to
write a java program to fix up overlaps & holes in an offline fashion (hbase
down, directly on hdfs), and will start testing next week (cross my fingers!).
It seems like the pending close messages can be ignored?
And once I test my tool, and confirm I know a little bit about what I'm doing,
maybe we could share notes?
Take care,
-stu
________________________________
From: Geoff Hendrey <[email protected]>
To: [email protected]
Cc: [email protected]
Sent: Saturday, September 3, 2011 12:11 AM
Subject: RE: PENDING_CLOSE for too long
"Are you having trouble getting to any of your data out in tables?"
depends what you mean. We see corruptions from time to time that prevent
us from getting data, one way or another. Today's corruption was regions
with duplicate start and end rows. We fixed that by deleting the
offending regions from HDFS, and running add_table.rb to restore the
meta. The other common corruption is the holes in ".META." that we
repair with a little tool we wrote. We'd love to learn why we see these
corruptions with such regularity (seemingly much higher than others on
the list).
We will implement timeout you suggest, and see how it goes.
Thanks,
Geoff
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of
Stack
Sent: Friday, September 02, 2011 10:51 PM
To: [email protected]
Cc: [email protected]
Subject: Re: PENDING_CLOSE for too long
Are you having trouble getting to any of your data out in tables?
To get rid of them, try restarting your master.
Before you restart your master, do "HBASE-4126 Make timeoutmonitor
timeout after 30 minutes instead of 3"; i.e. set
"hbase.master.assignment.timeoutmonitor.timeout" to 1800000 in
hbase-site.xml.
St.Ack
On Fri, Sep 2, 2011 at 1:40 PM, Geoff Hendrey <[email protected]>
wrote:
> In the master logs, I am seeing "regions in transition timed out" and
> "region has been PENDING_CLOSE for too long, running forced unasign".
> Both of these log messages occur at INFO level, so I assume they are
> innocuous. Should I be concerned?
>
>
>
> -geoff
>
>
