We used to do our updates through coprocessors, but it was not scalable. We extracted the update code into a separate system, added row transaction IDs, and haven't looked back.
For each incoming message, we compute the set of updates that message will generate. With a batch of messages, we merge updates to the same row together (for example, 3 increments of 1 each becomes 1 increment of 3). This means fewer overall writes to HBase, and no risk of cross-regionserver communication deadlocks. --Tom On Thu, Oct 10, 2013 at 1:23 PM, Vladimir Rodionov <[email protected]>wrote: > Nope. It is not so obvious, but definitely the anti-pattern is still there. > > Each RPC call is served by a thread from RPC-handlers pool (which is 10? > by default). > > Making RPC call from within handler's therad is: > > A. expensive > B. may result in some deadlock -type of situations when no more incoming > RPC calls can be accepted because > everyone is connected to everyone in a spaghetti way and waiting on RPC > calls to complete > > Let us say we have 2 region servers for simplicity: > RS1 and RS2. Both have pool of handler threads = 10 > > What happen when all 10 handlers in RS1 are trying to RPC RS2 and all 10 > handlers are trying to RPC RS2? > > The same deadlock. Its all probabilistic . > > Best regards, > Vladimir Rodionov > Principal Platform Engineer > Carrier IQ, www.carrieriq.com > e-mail: [email protected] > > ________________________________________ > From: Vladimir Rodionov > Sent: Thursday, October 10, 2013 12:09 PM > To: [email protected] > Subject: RE: Coprocessor Increments > > Classical deadlock : > > CP-Region1 updates counter in CP-Region2 (waits on RPC) > CP-Region2 updates counter in CP-Region1 (waits on RPC) > > I think its an anti-pattern. Do not do cross region calls in region CP > code. > > Best regards, > Vladimir Rodionov > Principal Platform Engineer > Carrier IQ, www.carrieriq.com > e-mail: [email protected] > > ________________________________________ > From: Michael Segel [[email protected]] > Sent: Thursday, October 10, 2013 9:55 AM > To: [email protected] > Cc: Ted Yu > Subject: Re: Coprocessor Increments > > I think Andrew has a handle on it… my take is that you end up running out > of resources to handle an RPC connection while within your coprocessor and > you're waiting for a resource to be free and it can't because another > coprocessor has an RPC resource and is also waiting for a free resource. > > Maybe its an over simplification, but if that's the case… you could always > try thing to limit the RPC call, which would delay updating the counter. > (Which may not be a problem) or redesign the coprocessors so that the > coprocessors don't share the same RPC resources. > > But the key is to really understanding and confirming what's causing the > Deadlock in detail. > > On Oct 10, 2013, at 11:15 AM, John Weatherford < > [email protected]> wrote: > > > Michael, > > I would also really like to know how this issue is caused also. I can't > even give a solid way to reproduce our deadlock. It *seems* to happen more > under load, but nothing can be proved yet. While google-ing and looking for > an answer I came across that old message post ( > http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5cn3ekzdjz_z-2qt8xo...@mail.gmail.com%3E). > This seemed to line up with what we are doing, so we _hope_ this will be a > fix for us, but we aren't entirely sure. > > > > > > > > On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote: > >> Can we just take a quick pause… > >> > >> John you wrote the following: > >> "We have been running into an RPC deadlock issue on HBase and from > >> investigation, we believe the root of the issue is in us doing cross > >> region increments from a coprocessor. After some further searching and > >> reading over this > >> < > http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5cn3ekzdjz_z-2qt8xo...@mail.gmail.com%3E > > > >> we think that we can solve this by doing the increments locally on the > >> region. " > >> > >> Which goes back to some thing Andrew wrote concerning an RPC deadlock. > >> > >> Can either you or Andrew explain in detail what is meant by the RPC > >> deadlock? > >> > >> This goes back to rethink how to implement coprocessors. > >> > >> > >> On Oct 9, 2013, at 11:03 PM, John Weatherford > >> <[email protected] <mailto:[email protected]>> > >> wrote: > >> > >>> Thank you ted. I might have to rethink my key design to accomodate > >>> this. With this design and using the totals in keys as you suggested, > >>> I would have to scan the entire "California" group to find the > >>> totals. Thank you very much for your help. > >>> > >>> -John > >>> > >>> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote: > >>>> John: > >>>> Suppose 'California-**12346' is the start key of region 1 and > >>>> 'California-** > >>>> 95424' is the start key of region 2. You can choose > >>>> 'California-**12346#total' > >>>> to be the row key where increment is done for region 1 and > >>>> 'California-**95424#total' > >>>> to be the row key where increment is done for region 2. > >>>> > >>>> w.r.t. verification of whether given row key is indeed inside the > >>>> hosting > >>>> region, see the following: > >>>> > >>>> void checkRow(final byte [] row, String op) throws IOException { > >>>> > >>>> if (!rowIsInRange(getRegionInfo(), row)) { > >>>> > >>>> throw new WrongRegionException("Requested row out of range for " + > >>>> > >>>> op + " on HRegion " + this + ", startKey='" + > >>>> > >>>> Cheers > >>>> > >>>> > >>>> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford < > >>>> [email protected] > >>>> <mailto:[email protected]>> wrote: > >>>> > >>>>> First, Thank you everyone for the quick response. > >>>>> > >>>>> Ted: > >>>>> The custom split policy could be an interesting solution. > >>>>> Regarding the > >>>>> other option you sent, if I pull the end row key, I could construct > >>>>> an end > >>>>> key, but if I suffix the row key with the end key of the region, > >>>>> would that > >>>>> actually solve my problem? In the contrived case, wouldn't the > >>>>> suffixed key > >>>>> now be "California-total-California-**12346" and the other region's > >>>>> counter would be "California-total-California-**95424" and both of > >>>>> these > >>>>> keys would actually end up on the second region since the sorting > >>>>> of the > >>>>> keys would impose that "California-t*" comes after > "California-[1-9]". > >>>>> > >>>>> Part of the question is that I don't understand if calling > >>>>> incrementColumnValue() on the region will always execute on the > region > >>>>> called, regardles of rowkey. If so, what happens when those regions > are > >>>>> merged? > >>>>> > >>>>> Thanks again for the help! > >>>>> > >>>>> > >>>>> On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote: > >>>>> > >>>>>> bq. 'California-total' row in each region > >>>>>> > >>>>>> Of course such row key needs to be suffixed with either start or > >>>>>> end key > >>>>>> of > >>>>>> the corresponding region. > >>>>>> > >>>>>> > >>>>>> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <[email protected] > >>>>>> <mailto:[email protected]>> wrote: > >>>>>> > >>>>>> John: > >>>>>>> Looks like you need a 'California-total' row in each region. > >>>>>>> > >>>>>>> Custom Split Policy can help you achieve that, see 9.7.4.1 in: > >>>>>>> http://hbase.apache.org/book.**html#d0e6541< > http://hbase.apache.org/book.html#d0e6541> > >>>>>>> > >>>>>>> Cheers > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov < > >>>>>>> [email protected] <mailto:[email protected]> > >>>>>>> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>> > >>>>>>> Contrived Example > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> Insert rowkey "California-12345" triggers a coprocessor to call > >>>>>>>>>> incrementColumnValue() with a rowkey of "California-total" all > on > >>>>>>>>>> > >>>>>>>>> Region 1. > >>>>>>>> > >>>>>>>> This would likely be on an insert on the same region. But as > >>>>>>>> the table > >>>>>>>>>> grows, this secondary insert could end up on another region. > >>>>>>>>>> If it is > >>>>>>>>>> confined, then suppose we later insert "California-95424" > >>>>>>>>>> which still > >>>>>>>>>> triggers a call to incrementColumnValue() with a rowkey of > >>>>>>>>>> "California-total" all on Region 2. > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> Are we now left with two rowkeys of "California-total"? One on > each > >>>>>>>>>> region server? If so, what happens if these two regions are > >>>>>>>>>> compacted > >>>>>>>>>> into one? > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> Best regards, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> Nope, Your "California-total" will migrate to Region 2 after > region > >>>>>>>> split > >>>>>>>> is complete. > >>>>>>>> > >>>>>>>> Vladimir Rodionov > >>>>>>>> Principal Platform Engineer > >>>>>>>> Carrier IQ, www.carrieriq.com > >>>>>>>> e-mail: [email protected] > >>>>>>>> > >>>>>>>> ______________________________**__________ > >>>>>>>> > >>>>>>>> Confidentiality Notice: The information contained in this > message, > >>>>>>>> including any attachments hereto, may be confidential and is > >>>>>>>> intended > >>>>>>>> to be > >>>>>>>> read only by the individual or entity to whom this message is > >>>>>>>> addressed. If > >>>>>>>> the reader of this message is not the intended recipient or an > >>>>>>>> agent or > >>>>>>>> designee of the intended recipient, please note that any review, > >>>>>>>> use, > >>>>>>>> disclosure or distribution of this message or its attachments, > >>>>>>>> in any > >>>>>>>> form, > >>>>>>>> is strictly prohibited. If you have received this message in > error, > >>>>>>>> please > >>>>>>>> immediately notify the sender and/or [email protected] > >>>>>>>> delete or destroy any copy of this message and its attachments. > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>> > >>> > >> > > > > > Confidentiality Notice: The information contained in this message, > including any attachments hereto, may be confidential and is intended to be > read only by the individual or entity to whom this message is addressed. If > the reader of this message is not the intended recipient or an agent or > designee of the intended recipient, please note that any review, use, > disclosure or distribution of this message or its attachments, in any form, > is strictly prohibited. If you have received this message in error, please > immediately notify the sender and/or [email protected] and > delete or destroy any copy of this message and its attachments. >
