[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255713#comment-16255713 ] Umesh Agashe commented on HBASE-18703: -- [~stack], thats true. Write paths are unified so there are no inconsistencies. This issue is essentially done. There are a couple of JIRAs that are left but can be worked upon after marking this closed. > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: hbase-18703.master.001.patch, > hbase-18703.master.002.patch, hbase-18703.master.003.patch, > hbase-18703.master.004.patch, hbase-18703.master.005.patch, > hbase-18703.master.005.patch, hbase-18703.master.005.patch, > hbase-18703.master.006.patch, hbase-18703.master.007.patch > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254895#comment-16254895 ] stack commented on HBASE-18703: --- [~uagashe] Isn't it true to say that we now have consistent behavior... Is this issue 'done'? > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: hbase-18703.master.001.patch, > hbase-18703.master.002.patch, hbase-18703.master.003.patch, > hbase-18703.master.004.patch, hbase-18703.master.005.patch, > hbase-18703.master.005.patch, hbase-18703.master.005.patch, > hbase-18703.master.006.patch, hbase-18703.master.007.patch > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208331#comment-16208331 ] Josh Elser commented on HBASE-18703: Thanks [~uagashe]! I wasn't quite sure the relationship from the patch attached here and the sub-tasks (admittedly, need to read the entire discussion). Thanks for the summary. > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: hbase-18703.master.001.patch, > hbase-18703.master.002.patch, hbase-18703.master.003.patch, > hbase-18703.master.004.patch, hbase-18703.master.005.patch, > hbase-18703.master.005.patch, hbase-18703.master.005.patch, > hbase-18703.master.006.patch, hbase-18703.master.007.patch > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208327#comment-16208327 ] Umesh Agashe commented on HBASE-18703: -- [~elserj], parent JIRA needs to be moved out of alpha-4. If and when patches are ready for sub-tasks, they can be included in alpha-4. > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: hbase-18703.master.001.patch, > hbase-18703.master.002.patch, hbase-18703.master.003.patch, > hbase-18703.master.004.patch, hbase-18703.master.005.patch, > hbase-18703.master.005.patch, hbase-18703.master.005.patch, > hbase-18703.master.006.patch, hbase-18703.master.007.patch > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208299#comment-16208299 ] Josh Elser commented on HBASE-18703: Any reason to not move this out of alpha-4, folks? Trying to catch up here and I'm just worried this will slow us down from getting the CP-related changes out the door for downstream projects. > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: hbase-18703.master.001.patch, > hbase-18703.master.002.patch, hbase-18703.master.003.patch, > hbase-18703.master.004.patch, hbase-18703.master.005.patch, > hbase-18703.master.005.patch, hbase-18703.master.005.patch, > hbase-18703.master.006.patch, hbase-18703.master.007.patch > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199167#comment-16199167 ] Umesh Agashe commented on HBASE-18703: -- [~appy], I can not agree with you more about RowProcessor/ processRowsWithLocks() combo being very generic and powerful. I think calling it a RowProcessor/ processRowsWithLocks() understates what can be done with it as processing is not only restricted to gets, puts, deletes or for that matter rows. All this can be done with or without acquiring row locks. IMO if we decide to retain this feature, we can consider renaming it to *ANY(Custom/ Generic etc)*Processor/ processWithOrWithoutRowsLocked() . Concern here is: * its exposed to user through Region APIs as a catch all solution for overcoming any current and future limitations in HBase. See example code of CP Endpoint BaseRowProcessorEndpoint.java. * Its not used internally except for MultiRowMutationEndpoint which can be easily removed (add and support isAtomic() attribute to batchOp in batchMutate()). * I also think CPs are used more compared to RPs (if any). * Its inscrutable which makes calling CP hooks, checking Quota enforcement almost impossible without re-design * Its vaguely defined feature * There already exists a code path with more up to date fixes, code (including above mentioned). Making batchMutate() use RowProcessor is interesting proposal but that means re-designing and re-writing out-dated feature and moving more up-to-date code path to use it. This sounds like difficult goal to achieve for 2.0. > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: hbase-18703.master.001.patch, > hbase-18703.master.002.patch, hbase-18703.master.003.patch, > hbase-18703.master.004.patch, hbase-18703.master.005.patch, > hbase-18703.master.005.patch, hbase-18703.master.005.patch, > hbase-18703.master.006.patch, hbase-18703.master.007.patch > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198180#comment-16198180 ] stack commented on HBASE-18703: --- Below is a bit of a reply. bq. Other path is very high level, generic, row processing mechanism. And is only supposed to invoke right RowProcessors steps at right time. Yes bq. Other path is very high level, generic, row processing mechanism. And is only supposed to invoke right RowProcessors steps at right time. ... all well and good but there should also be one mutation path only, not two. bq. So if RowProcessor is more generic, shouldn't we move to it? Why move to an option which is the limited one. We shoudn't move to it because it is (mostly) unused, unknown (years after its original intro), incomplete, lagging in updates, and it doesn't support CPs. Nor do we need a generic engine at this point (we have limited API -- we'd not be able to exploit the general facility -- and if you want more in absence of RP, write yourself a Coprocessor Endpoint). We want a purposed code path by the time we land inside Region. The variants are currently such that core ops in Region are inscrutable. We can't have that. A project to refactor sounds grand. Its all internal Private classes so could happen anytime. Would like to start over though if we were going to do this rather than try and hack around a bit of stale, RP code written w/ a motive now years old. > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: hbase-18703.master.001.patch, > hbase-18703.master.002.patch, hbase-18703.master.003.patch, > hbase-18703.master.004.patch, hbase-18703.master.005.patch, > hbase-18703.master.005.patch, hbase-18703.master.005.patch, > hbase-18703.master.006.patch, hbase-18703.master.007.patch > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198045#comment-16198045 ] Appy commented on HBASE-18703: -- Catching up from the top. There has already been a lot of discussion, some makes sense, while a lot doesn't :). Let me start with the main points. bq. HRegion.processRowsWithLocks() implementation, doesn't call coprocessor hooks but instead calls RowProcessor hooks at appropriate point in execution. Many of these hooks/ methods have same names and are called at similar points during the course of execution but they are not related! That's because RowProcessor is not necessarily doing mutations. It's can be doing anything- reading, writing, aggregating, etc. For eg. if i am storing analytics data, I may want to write a RowProcessor which can aggregates large amount of data on server itself and just returns the result. There's no need of coprocessor hooks for this use case. (yeah yeahbetter use something else for analytics, but you get the point). It's not necessarily for mutations, it's much more generic and powerful. bq. HRegion.batchMutate() methods call coprocessor hooks but not row RowProcessor hooks. I think hook is bad word to use here. Yeah both CP and RP have overridable function, but that's where similarity ends. CP hooks are events which are invoked when specific code path is travelled. This is a promise we make to third party. That's why on every put, get, delete, we invoke pre\*/post\* functions. RowProcessor functions are not events. RowProcessor class is like a task - a collection of functions - which is given to HRegion. And at different stages, HRegion invokes appropriate function of the processor. The act of running a RowProcessor has no *direct* relation to Coprocessors (in specific implementations, work done by RP might be related to CP). So batchMutate(), which has nothing to do with RowProcessor and should not be invoking RowProcessor hooks anyways. bq. The major inconsistency here is, one code path uses coprocessors while other uses RowProcessor. There are other minor inconsistencies along those two code paths. It's confusing as hell, sure, but my understanding is that there is no inconsistency here. One path is mutations, and is supposed to invoke coprocessors. Other path is very high level, generic, row processing mechanism. And is only supposed to invoke right RowProcessors steps at right time. Since we use the second more generic path for mutateRows(), with the help of MultiRowMutationProcessor, it's only correct to invoke appropriate CP hooks from inside it, otherwise we'll be violating our CP promises. bq. Comparing these list of steps for 2 methods, we can see the correlation for most hooks except for RowProcessor.process() I think it's only expected. batchMutate() already knows what the mutations are. It's coming into it as a param. Easy peesy. But poor processRowsWithLocks() doesn't know what it was called for, so it still has to figure it out, and it does so by asking RP for the mutations. Let's be sympathetic. On a serious note, that's by design. bq. I am proposing that Coprocessors can be used for customized processing instead of RowProcessor. Currently this can be done either with RowProcessors by calling Region.processRowsWithLocks() or with coprocessors by calling Region.batchMutate(). The intended difference between these 2 APIs is that Region.batchMutate() will only perform PUT and DELETE operations and Region.processRowsWithLocks() can perform any of GET, PUT, DELETE, CheckAndMutate etc operations. So if RowProcessor is more generic, shouldn't we move to it? Why move to an option which is the limited one. Note that in batchMutate(), mutations are not related. The batch is not a transaction. Some can fail, some can pass. Whereas in processRowsWithLocks(), it's transactional. So even if we merge two code paths, we need to keep two kinds of locking . The way i see RowProcessor & processRows() combo is - it will allow us to separate out row operations and pack them neatly in classes (outside of HRegion, 8000lines!!), and that too in a way such that it is easy to inject back. I think the better design here would be, improve the RowProcessor to provide native support for mutations - locks, invoking cp hooks etc. That way we can use it for our internal purpose. Advantages are (explicitly listing them again): 1) lot of code sharing : Code around locking, wal edits, pre/postCP hooks can be shared. Move batchMutation() to BatchMutationProcessor, and rename MultiRowMutationProcessor to denote that unlike BatchMutationProcessor, it's transactional. 2) We can split out all mutations related logic from 8000 lines of HRegion class. 3) It'll be a better design to expose for use in CPs. Taking a step back, yes, it's a mess. Unification of two paths would be great! >
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195580#comment-16195580 ] Umesh Agashe commented on HBASE-18703: -- Hi [~stack], [~anoop.hbase], [~ram_krish]: Thanks for your review comments. This is a good discussion. There is a general agreement about the direction. I also agree that this patch has quite a few changes that, as [~anoop.hbase] has suggested, can be split into sub-tasks. After talking to [~stack], I have created 6 sub-tasks to this JIRA. Reviewing individual patches again is more work but I think this will make reviewing the changes a bit less tedious overall. I have attached the patch for first sub-task JIRA along with reviewboard link. Looking forward to your review feedback. Thanks! > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: hbase-18703.master.001.patch, > hbase-18703.master.002.patch, hbase-18703.master.003.patch, > hbase-18703.master.004.patch, hbase-18703.master.005.patch, > hbase-18703.master.005.patch, hbase-18703.master.005.patch, > hbase-18703.master.006.patch, hbase-18703.master.007.patch > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194196#comment-16194196 ] Anoop Sam John commented on HBASE-18703: bq.Regarding giving user an option to specify read or write lock, depending on use case this can be implemented in future. Currently I am trying retain functionality with single code path. Most use case need exclusive/ write locks on multiple rows. Do you have any use case for which shared lock on multiple rows is enough? Not directly.. As per what [~chia7712] said, the RowProcessor was writing one row based on a read and condition check on some other rows. So I believe, in such cases, users might be taking write lock on those rows which has to be evaluated and read lock on the one which has to be written data to. Its fine we can check that later. In this patch, the big part is the (wrt #lines change) is the refactoring of the doMiniBatchMuatate() method by extracting some private methods. May be that itself u can do as a separate jira as that will need careful eye. And then go with unify the this with the processRowsWithLock > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: hbase-18703.master.001.patch, > hbase-18703.master.002.patch, hbase-18703.master.003.patch, > hbase-18703.master.004.patch, hbase-18703.master.005.patch, > hbase-18703.master.005.patch, hbase-18703.master.005.patch, > hbase-18703.master.006.patch, hbase-18703.master.007.patch > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194194#comment-16194194 ] Anoop Sam John commented on HBASE-18703: Seems the RowProcessor way of implementation is possible with the new APIs that we give + using some of the CP hook. So am +1 for removing it. Given it is LimitedPrivate remove or deprecate 1st and then remove? If we can remove RowProcessor , we can remove getRowLock being exposed to CPs. Ya we are giving new APIs which takes rowsToLock so whichever rows need additional lock, user can pass them. It would be really great if we can remove this row lock expose. And so startRegionOp (This is because we are exposing getting row lock). We will have to write one eg: use case with the new APIs and CPs to show how an old RowProcessor way is achievable now. And we will have the unify path. So at least 4 jiras may be there :-) > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: hbase-18703.master.001.patch, > hbase-18703.master.002.patch, hbase-18703.master.003.patch, > hbase-18703.master.004.patch, hbase-18703.master.005.patch, > hbase-18703.master.005.patch, hbase-18703.master.005.patch, > hbase-18703.master.006.patch, hbase-18703.master.007.patch > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[
https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193608#comment-16193608
]
Hadoop QA commented on HBASE-18703:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
10s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
56s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
38m 11s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
28s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}100m
13s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}164m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-18703 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890561/hbase-18703.master.007.patch
|
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 4ad5e7c107c0 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / bafbade |
| Default Java | 1.8.0_144 |
|
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193379#comment-16193379 ] Umesh Agashe commented on HBASE-18703: -- [~anoop.hbase], thanks for your comments and review. I liked your suggestion to split changes into multiple JIRAs. Regarding giving user an option to specify read or write lock, depending on use case this can be implemented in future. Currently I am trying retain functionality with single code path. Most use case need exclusive/ write locks on multiple rows. Do you have any use case for which shared lock on multiple rows is enough? bq. Yes. For the custom compare logic, prePut() CP can be used and based on the result, we can allow/bypass the put op. There are multiple ways user can achieve, reading and modifying multiple rows under exclusive lock/s. User can specify custom rows to lock by calling one of the Region APIs that take rowsToLock as an input. When prePut() hook of CPs is called its seeded with WALEdit but this hook is called prior to locking rows. The rows are locked and preBatchMutate() CP hook is called with mutations as an input. Based on this list user CP can update CP mutations or WALEdit (seeded with prePut() previously) or both. I think you are right, we need to simplify this code and split changes into multiple JIRAs. Let's not commit this patch and discuss things over it. Once we like the direction of the changes, I can work on splitting the changes into multiple JIRAs. Thanks! > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: hbase-18703.master.001.patch, > hbase-18703.master.002.patch, hbase-18703.master.003.patch, > hbase-18703.master.004.patch, hbase-18703.master.005.patch, > hbase-18703.master.005.patch, hbase-18703.master.005.patch, > hbase-18703.master.006.patch, hbase-18703.master.007.patch > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[
https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192174#comment-16192174
]
Hadoop QA commented on HBASE-18703:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
22s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
58s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
38m 44s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
17s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 95m
24s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-18703 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890429/hbase-18703.master.006.patch
|
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 16cafb9c2fb4 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 1ec6ece |
| Default Java | 1.8.0_144 |
|
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190785#comment-16190785 ] Anoop Sam John commented on HBASE-18703: bq.processRowsWithLocks() currently, takes write lock on all user specified rowsToLock In case of RowMutations usage of this processRowsWithLocks , we might not need write locks? Ya I just now noticed how the atomic boolean is been used in the patch. Can we avoid taking write locks always? Using some way define whether R/W locks needed. Just trying to make it general by asking more Qs :-) bq.from inside RowProcessor hooks user can call Region.getRowLock() API. I believe this is true for coprocessors as well. If we can make the processRowsWithLocks API to be generic enough which can work for all possible RowProcessor (And so deprecate that), we can remove the getRowLock() from Region itself. That would be ideal. bq.coprocessors can be used for all custom processing that RowProcessors are used for. I do not have any data about how many users are using RowProcessors. Yes. For the custom compare logic, prePut() CP can be used and based on the result, we can allow/bypass the put op. May be we need split these into diff jiras. > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: hbase-18703.master.001.patch, > hbase-18703.master.002.patch, hbase-18703.master.003.patch, > hbase-18703.master.004.patch, hbase-18703.master.005.patch, > hbase-18703.master.005.patch, hbase-18703.master.005.patch > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[
https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190517#comment-16190517
]
Hadoop QA commented on HBASE-18703:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
27s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
16s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
38m 20s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
18s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 94m
20s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-18703 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890227/hbase-18703.master.005.patch
|
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 691ad7efc934 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2ad7be2 |
| Default Java | 1.8.0_144 |
|
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190243#comment-16190243 ] Umesh Agashe commented on HBASE-18703: -- processRowsWithLocks() currently, takes write lock on all user specified rowsToLock. This will be same with modified batchMutate() that take list of rows to lock. Of course from inside RowProcessor hooks user can call Region.getRowLock() API. I believe this is true for coprocessors as well. bq. So can u pls tell how we will suggest users to do this (When no RowProcessor in place)? AFAIK, coprocessors can be used for all custom processing that RowProcessors are used for. I do not have any data about how many users are using RowProcessors. > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: hbase-18703.master.001.patch, > hbase-18703.master.002.patch, hbase-18703.master.003.patch, > hbase-18703.master.004.patch, hbase-18703.master.005.patch, > hbase-18703.master.005.patch > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190140#comment-16190140 ] Anoop Sam John commented on HBASE-18703: So what abt the combination of read/write locks as needed? The use case is that we have to do checks on N rows (customized compare) and mutate on say one another row. So we would need R/W locks on rows and custom complex checks? So can u pls tell how we will suggest users to do this (When no RowProcessor in place)? > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: hbase-18703.master.001.patch, > hbase-18703.master.002.patch, hbase-18703.master.003.patch, > hbase-18703.master.004.patch, hbase-18703.master.005.patch, > hbase-18703.master.005.patch > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190119#comment-16190119 ] Umesh Agashe commented on HBASE-18703: -- Thanks for your comments, [~anoop.hbase]! The uses case that [~chia7712] has specified is interesting and will be supported. bq. So u propose we dont give RowProcessor kind of way for customized processing? I am proposing that Coprocessors can be used for customized processing instead of RowProcessor. Currently this can be done either with RowProcessors by calling Region.processRowsWithLocks() or with coprocessors by calling Region.batchMutate(). The intended difference between these 2 APIs is that Region.batchMutate() will only perform PUT and DELETE operations and Region.processRowsWithLocks() can perform any of GET, PUT, DELETE, CheckAndMutate etc operations. bq. Use has to do multi row compare and based on that do a mutate op on another row.. I see adding a new API which takes rowsToLock. But here in this issue, user may have to take write locks on certain rows and read lock on another. Also has to do a complex compare op on many row:columns. What is the alternate we can give? Sequence of steps for methods Region.processRowsWithLocks() and Region.batchMutate() can be roughly described as below. for Region.processRowsWithLocks(): * Build empty WALEdit * Call RowProcessor.preProcess(WALEdit) * Lock all user specified rows * Call RowProcessor.process(WALEdit) and get mutations[] to apply to MemStore * Call RowProcessor.preBatchMutate(WALEdit) * Append WALEdit processed by RowProcessor to WAL * Apply mutations to MemStore * Call RowProcessor.postBatchMutate() * Release locks * Call RowProcessor.postProcess() Region.batchMutate(Mutation[]): * Take mutations (PUTs and DELETEs only) as an input * Prepare empty WALEdit * Call cp.prePut(WALEdit) or cp.preDelete(WALEdit) and store WALEdits for each mutation into BatchOperation * Lock as many rows corresponding to mutations as possible * For mutations for which rows can be locked, call cp.preBatchMutate(Mutation[]) * Get Mutations from CP. * For each mutation returned by CP: lock rows and merge them with input list of mutations * Build new WALEdit by applying these merged mutations (input + from cp) * Apply WALEdits from CP (previously stored in BatchOperation after calling prePut/ preDelete) to WALEdit built in previous step * Append the merged WALEdit to WAL * Apply merged mutations to MemStore * call cp.postBatchMutate() * Release locks * Call cp.postPut()/ cp.postDelete() * Call cp.postBatchMutateIndispensably() Comparing these list of steps for 2 methods, we can see the correlation for most hooks except for RowProcessor.process(). Currently processRowsWithLocks() will still be supported but it will not take RowProcessor as an argument. User can still specify customized rows to lock and those rows will be locked by batchMutate(). For customized processing user can write his/ her own coprocessor. What do you think? > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: hbase-18703.master.001.patch, > hbase-18703.master.002.patch, hbase-18703.master.003.patch, > hbase-18703.master.004.patch, hbase-18703.master.005.patch, > hbase-18703.master.005.patch > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[
https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189314#comment-16189314
]
Hadoop QA commented on HBASE-18703:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
59s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
56s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 9s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
18s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 16s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}151m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionOnCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-18703 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890094/hbase-18703.master.005.patch
|
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux eb743c99fa24 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[
https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189197#comment-16189197
]
Hadoop QA commented on HBASE-18703:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
18s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
8s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
37m 1s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
22s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 10s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
51s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay |
| | org.apache.hadoop.hbase.master.procedure.TestModifyTableProcedure |
| | org.apache.hadoop.hbase.regionserver.TestRowTooBig |
| | org.apache.hadoop.hbase.regionserver.TestSplitLogWorker |
| | org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy
|
| | org.apache.hadoop.hbase.master.TestGetLastFlushedSequenceId |
| | org.apache.hadoop.hbase.regionserver.wal.TestFSHLog |
| | org.apache.hadoop.hbase.regionserver.TestCompaction |
| | org.apache.hadoop.hbase.trace.TestHTraceHooks |
| | org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 |
| |
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[
https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187680#comment-16187680
]
Anoop Sam John commented on HBASE-18703:
Sorry for the delay
Making use of RowProcessor for the mutateRows and so many diff with the CP
calls and complexity.. it is fine to unify and make them simple.
bq.Deprecate RowProcessor and API Region.processRowsWithLocks() that takes
RowProcessor as an argument.
But this point am not sure. So u propose we dont give RowProcessor kind of way
for customized processing? In another issue comment discussion Chia-Ping gives
a use case as below
{quote}
The row locks is a dangerous feature for us, but it is also a powerful tool for
cp user. The use cases I had saw use the write locks to implement
compare-and-update over multi-rows.
1) get write-lock for all required rows (Region#getRowLock)
2) get latest data (Region#get)
3) generate the Puts/Deletes according to the latest data
4) do Put/Delete
It seems to me that removing row locks in 2.0 is fine if we provide more
flexible multi-rows operation in follow-up. I guess Phoenix has similar impl of
compare-and-update.
{quote}
Use has to do multi row compare and based on that do a mutate op on another
row.. I see adding a new API which takes rowsToLock. But here in this issue,
user may have to take write locks on certain rows and read lock on another.
Also has to do a complex compare op on many row:columns. What is the alternate
we can give? I believe RowProcessor was added as a means for doing such things.
So still that is useful.
I believe if we can unify the code paths of processRowsWithLocks and
doMiniBatchMutate() and make the mutateRows NOT to follow the RowProcessor way
of impl, the 1st issue of CP call diff and complexity can be avoided. Also we
can make the RowProcessor to be simpler as in the past with just a pre and post
process and a process() call. May be we can add an impl usage in the examples
module with a use case what is mentioned above.
> Inconsistent behavior for preBatchMutate in doMiniBatchMutate and
> processRowsWithLocks
> --
>
> Key: HBASE-18703
> URL: https://issues.apache.org/jira/browse/HBASE-18703
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: Duo Zhang
>Assignee: Umesh Agashe
>Priority: Critical
> Fix For: 2.0.0-alpha-4
>
> Attachments: hbase-18703.master.001.patch,
> hbase-18703.master.002.patch, hbase-18703.master.003.patch,
> hbase-18703.master.004.patch
>
>
> In doMiniBatchMutate, the preBatchMutate is called before building WAL, but
> in processRowsWithLocks, we suggest the RowProcessor implementation to build
> WAL in process method, which is ahead of preBatchMutate.
> If a CP modifies the mutations, especially if it removes some cells from the
> mutations, then the behavior of processRowsWithLocks is broken. The changes
> applied to memstore and WAL will be different. And there is no way to remove
> entries from a WALEdit through CP.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[
https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187340#comment-16187340
]
Hadoop QA commented on HBASE-18703:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
16s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
55s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
36m 9s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
17s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 39s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestFromClientSide3 |
| | hadoop.hbase.client.TestCheckAndMutate |
| | hadoop.hbase.util.TestFromClientSide3WoUnsafe |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-18703 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12889878/hbase-18703.master.004.patch
|
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux b2c8487c1d17 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
|
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[
https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186749#comment-16186749
]
Hadoop QA commented on HBASE-18703:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
25s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
0s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
37m 38s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
22s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 21s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
55s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestDisableTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestModifyTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestCreateTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestEnableTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestDeleteTableProcedure |
| | org.apache.hadoop.hbase.client.TestSnapshotCloneIndependence |
| | org.apache.hadoop.hbase.coprocessor.TestHTableWrapper |
| | org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy
|
| | org.apache.hadoop.hbase.master.TestGetLastFlushedSequenceId |
| |
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[
https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186609#comment-16186609
]
Hadoop QA commented on HBASE-18703:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
24s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
57s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
36m 22s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
18s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 53m 22s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
55s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestDisableTableProcedure |
| | org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay |
| | org.apache.hadoop.hbase.master.procedure.TestModifyTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestCreateTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestEnableTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestDeleteTableProcedure |
| | org.apache.hadoop.hbase.regionserver.TestRowTooBig |
| | org.apache.hadoop.hbase.regionserver.TestSplitLogWorker |
| | org.apache.hadoop.hbase.regionserver.wal.TestAsyncWALReplay |
| |
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[
https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185402#comment-16185402
]
Hadoop QA commented on HBASE-18703:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
57s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
46s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
37m 6s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
32s{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed =
1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
17s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 42m 17s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}107m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Impossible downcast of toArray() result to
org.apache.hadoop.hbase.client.Mutation[] in
org.apache.hadoop.hbase.regionserver.HRegion.doCheckAndRowMutate(byte[],
byte[], byte[], CompareOperator, ByteArrayComparable, RowMutations, Mutation,
boolean) At HRegion.java:to org.apache.hadoop.hbase.client.Mutation[] in
org.apache.hadoop.hbase.regionserver.HRegion.doCheckAndRowMutate(byte[],
byte[], byte[], CompareOperator, ByteArrayComparable, RowMutations, Mutation,
boolean) At HRegion.java:[line 3774] |
| Failed junit tests | hadoop.hbase.filter.TestDependentColumnFilter |
| |
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176853#comment-16176853 ] Anoop Sam John commented on HBASE-18703: Good. Go for it.. Will check once ur patch comes. Ya better try unify these 2 paths. > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-alpha-4 > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175318#comment-16175318 ] Umesh Agashe commented on HBASE-18703: -- [~anoopamz], [~Apache9]: These two code paths look very similar and I found comments/ todos in the code about unifying these two paths. The main difference I see here is: * one is atomic and other is not * Atomic path guarantees the oder of operations and non-atomic path does not To remove inconsistencies (above mentioned + a few other minor), I am working on a patch to unify these paths. I think I will be able to upload the patch soon. Let me know your thoughts. > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-alpha-4 > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18703) Inconsistent behavior for preBatchMutate in doMiniBatchMutate and processRowsWithLocks
[ https://issues.apache.org/jira/browse/HBASE-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143477#comment-16143477 ] Anoop Sam John commented on HBASE-18703: I think the inconsistency came when we done changes in order of ops in batch mutate. I believe there was a jira raised in the past also. Forgot details and whether we did some changes as per that or not. Thanks for the nice find. > Inconsistent behavior for preBatchMutate in doMiniBatchMutate and > processRowsWithLocks > -- > > Key: HBASE-18703 > URL: https://issues.apache.org/jira/browse/HBASE-18703 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang > Fix For: 2.0.0-alpha-3 > > > In doMiniBatchMutate, the preBatchMutate is called before building WAL, but > in processRowsWithLocks, we suggest the RowProcessor implementation to build > WAL in process method, which is ahead of preBatchMutate. > If a CP modifies the mutations, especially if it removes some cells from the > mutations, then the behavior of processRowsWithLocks is broken. The changes > applied to memstore and WAL will be different. And there is no way to remove > entries from a WALEdit through CP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
