Michael,
I would also really like to know how this issue is caused also. I
can't even give a solid way to reproduce our deadlock. It *seems* to
happen more under load, but nothing can be proved yet. While google-ing
and looking for an answer I came across that old message post
(http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5cn3ekzdjz_z-2qt8xo...@mail.gmail.com%3E).
This seemed to line up with what we are doing, so we _hope_ this will
be a fix for us, but we aren't entirely sure.
On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
Can we just take a quick pause…
John you wrote the following:
"We have been running into an RPC deadlock issue on HBase and from
investigation, we believe the root of the issue is in us doing cross
region increments from a coprocessor. After some further searching and
reading over this
<http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5cn3ekzdjz_z-2qt8xo...@mail.gmail.com%3E>
we think that we can solve this by doing the increments locally on the
region. "
Which goes back to some thing Andrew wrote concerning an RPC deadlock.
Can either you or Andrew explain in detail what is meant by the RPC
deadlock?
This goes back to rethink how to implement coprocessors.
On Oct 9, 2013, at 11:03 PM, John Weatherford
<[email protected] <mailto:[email protected]>>
wrote:
Thank you ted. I might have to rethink my key design to accomodate
this. With this design and using the totals in keys as you suggested,
I would have to scan the entire "California" group to find the
totals. Thank you very much for your help.
-John
On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
John:
Suppose 'California-**12346' is the start key of region 1 and
'California-**
95424' is the start key of region 2. You can choose
'California-**12346#total'
to be the row key where increment is done for region 1 and
'California-**95424#total'
to be the row key where increment is done for region 2.
w.r.t. verification of whether given row key is indeed inside the
hosting
region, see the following:
void checkRow(final byte [] row, String op) throws IOException {
if (!rowIsInRange(getRegionInfo(), row)) {
throw new WrongRegionException("Requested row out of range for " +
op + " on HRegion " + this + ", startKey='" +
Cheers
On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
[email protected]
<mailto:[email protected]>> wrote:
First, Thank you everyone for the quick response.
Ted:
The custom split policy could be an interesting solution.
Regarding the
other option you sent, if I pull the end row key, I could construct
an end
key, but if I suffix the row key with the end key of the region,
would that
actually solve my problem? In the contrived case, wouldn't the
suffixed key
now be "California-total-California-**12346" and the other region's
counter would be "California-total-California-**95424" and both of
these
keys would actually end up on the second region since the sorting
of the
keys would impose that "California-t*" comes after "California-[1-9]".
Part of the question is that I don't understand if calling
incrementColumnValue() on the region will always execute on the region
called, regardles of rowkey. If so, what happens when those regions are
merged?
Thanks again for the help!
On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
bq. 'California-total' row in each region
Of course such row key needs to be suffixed with either start or
end key
of
the corresponding region.
On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <[email protected]
<mailto:[email protected]>> wrote:
John:
Looks like you need a 'California-total' row in each region.
Custom Split Policy can help you achieve that, see 9.7.4.1 in:
http://hbase.apache.org/book.**html#d0e6541<http://hbase.apache.org/book.html#d0e6541>
Cheers
On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <
[email protected] <mailto:[email protected]>
wrote:
Contrived Example
Insert rowkey "California-12345" triggers a coprocessor to call
incrementColumnValue() with a rowkey of "California-total" all on
Region 1.
This would likely be on an insert on the same region. But as
the table
grows, this secondary insert could end up on another region.
If it is
confined, then suppose we later insert "California-95424"
which still
triggers a call to incrementColumnValue() with a rowkey of
"California-total" all on Region 2.
Are we now left with two rowkeys of "California-total"? One on each
region server? If so, what happens if these two regions are
compacted
into one?
Best regards,
Nope, Your "California-total" will migrate to Region 2 after region
split
is complete.
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [email protected]
______________________________**__________
Confidentiality Notice: The information contained in this message,
including any attachments hereto, may be confidential and is
intended
to be
read only by the individual or entity to whom this message is
addressed. If
the reader of this message is not the intended recipient or an
agent or
designee of the intended recipient, please note that any review,
use,
disclosure or distribution of this message or its attachments,
in any
form,
is strictly prohibited. If you have received this message in error,
please
immediately notify the sender and/or [email protected] and
delete or destroy any copy of this message and its attachments.
