FYI, Ted, I see this exact similar issue being discussed in the past here as well:
http://mail-archives.apache.org/mod_mbox/hbase-user/201406.mbox/%3CCAKrkF=thi8g4Ks=viqgC+Y=ivuqysogoq41rmkutfriunal...@mail.gmail.com%3E Regards, Shahab On Fri, Nov 14, 2014 at 11:35 AM, Shahab Yunus <[email protected]> wrote: > Hi Ted. > > The log bit is below at the end of the email. This is the command to merge > that I gave just now through hbase shell. forcible was false but it behaves > similarly if forcible is true too. This is from master log. Indeed the > region merging was skipped! What does this mean? Data seems to be intact > for this table. > > Just to give you a background. This table was first merge by the auto > mated java application. What we are doing is that we are merging tables > programmatically. As the HBaseAdmin.mergeRegions calls i async, we poll for > the number of regions getting lowered after this merge call. The > application hangs and continues polling for ever as the previous merge > didn't happen. > > In this poll loop, we do get the number of regions by a fresh call to > HBaseAdmin.getTableRegions(tableName).getSize(). > > What are these merge qualifiers and what are we doing wrong or should do? > > In the polling loop we can somehow retry merge again? But how can we know, > that we need to call merge again as it works for some regions. Is the table > meta corrupted for some reason by the above logic? > > Thanks a lot. > > > > ------------------------------------------------------------------------ > > 2014-11-14 11:25:02,643 INFO org.apache.zookeeper.ZooKeeper: Session: > 0x348c7017707236b closed > 2014-11-14 11:25:02,643 INFO org.apache.zookeeper.ClientCnxn: EventThread > shut down > 2014-11-14 11:25:02,645 INFO org.apache.zookeeper.ZooKeeper: Initiating > client connection, > connectString=ip-1010019.ec2.internal:2181,ip-1010017.ec2.internal:2181,ip-1010018.ec2.internal:2181 > sessionTimeout=60000 watcher=catalogtracker-on-hconnection-0x47d865f2, > quorum=ip-1010019.ec2.internal:2181,ip-1010017.ec2.internal:2181,ip-1010018.ec2.internal:2181, > baseZNode=/hbase > 2014-11-14 11:25:02,645 INFO > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process > identifier=catalogtracker-on-hconnection-0x47d865f2 connecting to ZooKeeper > ensemble=ip-1010019.ec2.internal:2181,ip-1010017.ec2.internal:2181,ip-1010018.ec2.internal:2181 > 2014-11-14 11:25:02,645 INFO org.apache.zookeeper.ClientCnxn: Opening > socket connection to server ip-1010018.ec2.internal/1010019:2181. Will not > attempt to authenticate using SASL (unknown error) > 2014-11-14 11:25:02,646 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to ip-1010018.ec2.internal/1010019:2181, initiating > session > 2014-11-14 11:25:02,648 INFO org.apache.zookeeper.ClientCnxn: Session > establishment complete on server ip-1010018.ec2.internal/1010019:2181, > sessionid = 0x348c7017707236c, negotiated timeout = 60000 > 2014-11-14 11:25:02,703 INFO org.apache.zookeeper.ZooKeeper: Session: > 0x348c7017707236c closed > 2014-11-14 11:25:02,703 INFO org.apache.zookeeper.ClientCnxn: EventThread > shut down > 2014-11-14 11:25:30,713 INFO > org.apache.hadoop.hbase.master.handler.DispatchMergingRegionHandler: Skip > merging regions > TABLE_NAME,,1415915112497.7373f75181c71eb5061a6673cee15931., > TABLE_NAME,\x02\xFA\xF0\x80\x00\x00\x01I\xAA\xD5\x87\xA8\x19\x99\x99\x99\x99\x99\x99\x90,1415910559217.43f4d3685d113d3ae18eea9f189de096., > because region 7373f75181c71eb5061a6673cee15931 has merge qualifier > 2014-11-14 11:25:41,383 INFO org.apache.zookeeper.ZooKeeper: Initiating > client connection, > connectString=ip-1010019.ec2.internal:2181,ip-1010017.ec2.internal:2181,ip-1010018.ec2.internal:2181 > sessionTimeout=60000 watcher=catalogtracker-on-hconnection-0x47d865f2, > quorum=ip-1010019.ec2.internal:2181,ip-1010017.ec2.internal:2181,ip-1010018.ec2.internal:2181, > baseZNode=/hbase > 2014-11-14 11:25:41,384 INFO > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process > identifier=catalogtracker-on-hconnection-0x47d865f2 connecting to ZooKeeper > ensemble=ip-1010019.ec2.internal:2181,ip-1010017.ec2.internal:2181,ip-1010018.ec2.internal:2181 > 2014-11-14 11:25:41,384 INFO org.apache.zookeeper.ClientCnxn: Opening > socket connection to server ip-1010018.ec2.internal/1010019:2181. Will not > attempt to authenticate using SASL (unknown error) > 2014-11-14 11:25:41,386 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to ip-1010018.ec2.internal/1010019:2181, initiating > session > 2014-11-14 11:25:41,389 INFO org.apache.zookeeper.ClientCnxn: Session > establishment complete on server ip-1010018.ec2.internal/1010019:2181, > sessionid = 0x348c7017707236e, negotiated timeout = 60000 > 2014-11-14 11:25:41,397 INFO org.apache.zookeeper.ZooKeeper: Session: > 0x348c7017707236e closed > 2014-11-14 11:25:41,398 INFO org.apache.zookeeper.ClientCnxn: EventThread > shut down > > ------------------------------------------------------------------------------------------------------------------------------------ > > Regards, > Shahab > > On Fri, Nov 14, 2014 at 10:56 AM, Ted Yu <[email protected]> wrote: > >> Looking at DispatchMergingRegionHandler, it does some check before >> initiating the merge. >> e.g.: >> >> LOG.info("Skip merging regions " + region_a.getRegionNameAsString() >> >> + ", " + region_b.getRegionNameAsString() + ", because region " >> >> + (regionAHasMergeQualifier ? region_a.getEncodedName() : >> region_b >> >> .getEncodedName()) + " has merge qualifier"); >> >> Can you take a look at master log around the time merge request was issued >> to see if you can get some clue ? >> >> Cheers >> >> On Fri, Nov 14, 2014 at 6:41 AM, Shahab Yunus <[email protected]> >> wrote: >> >> > The documentation of online merge tool (merge_region) states that if we >> > forcibly merge regions (by setting the 3rd attribute as true) then it >> can >> > create overlapping regions. if this happens then will this render the >> > region or table unusable or it is just a performance hit? I mean how >> bigger >> > of a deal it is? >> > >> > Actually, we are merging regions using the programmatic API for this and >> > setting this flag ('forcible') as false. But for some tables (we haven't >> > figured out a pattern yet, data is still accessible), merge of regions >> do >> > not happen at all. Afterwards we tried with this flag = true, and it >> still >> > doesn't merge them. >> > >> > CDH 5.1.0 >> > (Hbase is 0.98.1-cdh5.1.0) >> > >> > Regards, >> > Shahab >> > >> > >
