[
https://issues.apache.org/jira/browse/HBASE-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186396#comment-15186396
]
Anoop Sam John edited comment on HBASE-15417 at 3/9/16 3:06 AM:
----------------------------------------------------------------
This is not a test issue.. This is a bug only IMO.
We call batchMutate with some puts. 1st we will call prePut() on all of them
one after the other. When for any put, the CP calls bypass, we will mark that
Mutation status as success. Then we will call doMiniBatchMutate().. The actual
mutation happens in 1 or more mini batches as we obtain rowlocks.
Then we try to obtain rowlocks in doMiniBatchMutate() iff the status of that
mutation is not done. But the pre hook made it to success. So no need to obtain
row lock and there is 0 puts to write. Then we have this code in
doMiniBatchMutate
{code}
if (numReadyToWrite <= 0) {
return 0L;
}
{code}
The post hook is after the actual write on memstore and WAL in
doMiniBatchMutate.
So when all puts handled by pre hook and bypassed, there is no call to post
hook.
But when at least one of the put is not done by pre hook and actual write on
memstore happens in doMiniBatchMutate, we will call post hook on ALL puts which
appeared in THAT mini batch. So it is not even all the puts.. If all puts can
happen in one mini batch, yes on all we will call post hook.
The problem in when we call post hook, we know the status of the op and success
or not and dont know whether it is set by pre hook call and bypass or by the
actual put on memstore. If we can distinguish that and decide to call post
hook based on that, we can solve this
Even the pre and post hook for batch mutate (pre/postBatchMutate) also to be
consistent with pre/postPut. Ie. when prePut bypass one mutation, that
mutation should not be part of mutations that we pass to pre/postBatchMutate
hooks.
If u can work on a patch based on the above we will review.. If u are busy ,
let me know, I can give a patch :-)
Thanks for the find..
was (Author: anoop.hbase):
This is not a test issue.. This is a bug only IMO.
We call batchMutate with some puts. 1st we will call prePut() on all of them
one after the other. When for any put, the CP calls bypass, we will mark that
Mutation status as success. Then we will call doMiniBatchMutate().. The actual
mutation happens in 1 or more mini batches as we obtain rowlocks.
Then we try to obtain rowlocks in doMiniBatchMutate() iff the status of that
mutation is not done. But the pre hook made it to success. So no need to obtain
row lock and there is 0 puts to write. Then we have this code in
doMiniBatchMutate
{code}
if (numReadyToWrite <= 0) {
return 0L;
}
{code}
The post hook is after the actual write on memstore and WAL in
doMiniBatchMutate.
So when all puts handled by pre hook and bypassed, there is no call to post
hook.
But when at least one of the put is not done by pre hook and actual write on
memstore happens in doMiniBatchMutate, we will call post hook on ALL puts which
appeared in THAT mini batch. So it is not even all the puts.. If all puts can
happen in one mini batch, yes on all we will call post hook.
The problem in when we call post hook, we know the status of the op and success
or not and dont know whether it is set by pre hook call and bypass or by the
actual put on memstore. If we can distinguish that and decide to call post
hook based on that, we can solve this
Even the pre and post hook for batch mutate (pre/postBatchMutate) also to be
consistent with pre/postPut. Ie. when prePut bypass one mutation, that
mutation should not be part of mutations that we pass to pre/postBatchMutate
hooks.
If u can work on a patch based on the above we will reviews.. If u are busy ,
let me know, I can give a patch :-)
Thanks for the find..
> Calls to ObserverContext#bypass in a region observer's prePut method are
> inconsistent
> -------------------------------------------------------------------------------------
>
> Key: HBASE-15417
> URL: https://issues.apache.org/jira/browse/HBASE-15417
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
> Reporter: Harry Harpham
> Priority: Minor
>
> Calling ctx.bypass(), where ctx is the ObserverContext object passed in to
> the region observer's prePut method, results in some inconsistent behavior.
> If every other put in the batch is also bypassed, the region observer sees
> none of these in its postPut method. If there is at least one other put
> which is not bypassed, the region observer sees all of the puts in the batch
> _including those which were bypassed_.
> The end result is that, after bypassing a put, that put may or may not end up
> in the region observer's postPut method. This behavior is dependent solely
> on which other puts the bypassed put is batched together with.
> I tried to find existing tickets for this issue, but was unable to.
> Apologies if I missed something. The closest issues I could find were
> HBASE-4331 and HBASE-11503, but those didn't seem to quite hit it.
> Additionally, I threw together a quick demonstration of this issue:
> https://github.com/hwh33/bypass-inconsistency-demo. You can run that demo in
> memory using the testing utility or against a running cluster. I actually
> haven't had time to test it against a cluster though, so you may encounter
> bugs if running in that mode (but hopefully not!).
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
