Ivan: You can utilize the following method in HBaseAdmin to retrieve compaction status:
public CompactionState getCompactionState(final TableName tableName) This API is available in 0.98+ FYI On Thu, Jul 16, 2015 at 6:58 AM, Ivan Brondino <[email protected]> wrote: > Hi Shahab, > > We issue the splits on the table and right away the major compact of the > table. > Then we connect to the HRegionServer info server at http://host:60030 and > we check html output for the task to be completed using an HttpClient. > So we actually wait for major compact and not for split. > We've been using it since 0.89 but we believe there must be a bettter > approach than that. > > Thanks, > > IVan > > > > 2015-07-16 15:50 GMT+02:00 Shahab Yunus <[email protected]>: > > > Ivan, > > > > Is it possible that you poll for number of regions (e.g. in a loop) after > > invoking split or merge to confirm that the action has been performed? I > > know it is a crude way but maybe something can be done in these lines. Or > > are you already doing this when you said 'look for Region Info'? > > > > Regards, > > Shahab > > > > On Thu, Jul 16, 2015 at 9:46 AM, Ivan Brondino <[email protected]> > > wrote: > > > > > Hi, > > > > > > I can confirm that it was what Jean Marc was saying. Thanks JM! > > > > > > Regarding question #1, any help? I am rephrasing it in case was not > > clear. > > > > > > 1) Since both split and merge are async invocations on the client side, > > we > > > have a naîve > > > workaround to make it synchronous that is to look at the RS Info Server > > via > > > an HttpClient checking > > > for pending tasks to be finished. Once all tasks are finished in all RS > > we > > > issue the next split/merge for the same table. Is there a better way to > > > check if major compact has finished? > > > > > > Thanks again, > > > > > > Iván > > > > > > > > > 2015-07-15 17:49 GMT+02:00 Ivan Brondino <[email protected]>: > > > > > > > Hi Ted, Jean Marc, > > > > > > > > Thank you guys for your answer. > > > > > > > > I think Jean Marc spotted the issue, although we do slightly > different. > > > > > > > > Let's say we have regions A and B. We split both A into A1 and A2 > and > > B > > > > into B1 and B2. > > > > Then we merge A2 and B1. Note that A and B are adjacent and so A1 > with > > > A2, > > > > A2 with B1 and B1 with B2. > > > > > > > > We were not waiting for the major compact after the splits. Thus it > > could > > > > be the reason. > > > > > > > > We will give it a try and come back to you. > > > > > > > > Thanks, > > > > > > > > Ivan > > > > > > > > > > > > > > > > > > > > > > > > 2015-07-15 16:54 GMT+02:00 Jean-Marc Spaggiari < > > [email protected] > > > >: > > > > > > > >> Hi Ivan, > > > >> > > > >> > > > >> For #1, let's consider regions A, B, C and D all in server S. > > > >> > > > >> If you merge A and B togher and C and D togher, still in S, you will > > > have > > > >> A' and C' regions only on S. > > > >> > > > >> Now, if you try to merge again A' and C' into a new bigger region, > > this > > > >> will fail silently until A' and C' are major compacted. > > > >> > > > >> Indeed, before major compaction, A' and C' will both still contain > > > >> references to A+B and C+D files/regions. So to be able to merge A' > and > > > C' > > > >> you need this to be cleaned and so you need to major compact them. > > > >> > > > >> HTH, > > > >> > > > >> JM > > > >> > > > >> 2015-07-15 10:43 GMT-04:00 Ted Yu <[email protected]>: > > > >> > > > >> > bq. that some times merge just does not work. > > > >> > > > > >> > Can you identify under what scenario the merge doesn't work > (through > > > >> closer > > > >> > inspection of the region server log - assuming you have DEBUG > > logging > > > >> > turned on) ? > > > >> > > > > >> > bq. Are there minimum requirements for two regions to be merged? > > > >> > > > > >> > If the two adjacent regions to be merged are on different servers, > > > HBase > > > >> > would move one of them to the other region's server. > > > >> > > > > >> > bq. If those requirements are not met, does the merge process exit > > > >> > silently? > > > >> > > > > >> > Nothing I am aware of. > > > >> > > > > >> > Cheers > > > >> > > > > >> > On Wed, Jul 15, 2015 at 3:08 AM, Ivan Brondino < > [email protected] > > > > > > >> > wrote: > > > >> > > > > >> > > Hi, > > > >> > > > > > >> > > We are using Hbase 0.98.6 and Hadoop 2.5.0 - cdh 5.3.5. > > > >> > > > > > >> > > We have a couple of questions regarding splits and merge: > > > >> > > > > > >> > > 1) Since both split and merge are async invocations on the > client > > > >> side, > > > >> > we > > > >> > > have a naîve > > > >> > > workaround to make it synchronous that is to look at the RS Info > > > >> Server > > > >> > via > > > >> > > an HttpClient checking > > > >> > > for pending tasks to be finished. Once all tasks are finished in > > all > > > >> RS > > > >> > we > > > >> > > issue the next split/merge for the same table. Is there a better > > way > > > >> to > > > >> > do > > > >> > > it? > > > >> > > > > > >> > > 2) Regarding merges, we've noticed that some times merge just > does > > > not > > > >> > > work. We see on the master log that merge request is issued and > it > > > >> > forwards > > > >> > > it to the corresponding RS but then on the RS nothing happens. > We > > > >> always > > > >> > > merge two regions on the same region server. > > > >> > > Are there minimum requirements for two regions to be merged? If > > > those > > > >> > > requirements are not met, does the merge process exit silently? > > > >> > > > > > >> > > Thanks in advance, > > > >> > > > > > >> > > Iván > > > >> > > > > > >> > > > > >> > > > > > > > > > > > > > >
