[jira] [Commented] (HBASE-8458) Support for batch version of checkAndMutate()
[ https://issues.apache.org/jira/browse/HBASE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176670#comment-17176670 ] Toshihiro Suzuki commented on HBASE-8458: - [~ndimiduk] Thank you for pointing that out. I just changed the Release Note. > Support for batch version of checkAndMutate() > - > > Key: HBASE-8458 > URL: https://issues.apache.org/jira/browse/HBASE-8458 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Hari Mankude >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > The use case is that the user has multiple threads loading hundreds of keys > into a hbase table. Occasionally there are collisions in the keys being > uploaded by different threads. So for correctness, it is required to do > checkAndMutate() instead of a put(). However, doing a checkAndMutate() rpc > for every key update is non optimal. It would be good to have a batch version > of checkAndMutate() similar to batch put(). The client can partition the keys > on region boundaries. > The jira is NOT looking for any type of cross-row locking or multi-row > atomicity with checkAndMutate(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-8458) Support for batch version of checkAndMutate()
[ https://issues.apache.org/jira/browse/HBASE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175885#comment-17175885 ] Nick Dimiduk commented on HBASE-8458: - [~brfrn169] instead of pasting the javadoc, maybe we can link off to the API docs? It's a little tricky because those docs haven't been released yet, but we expect the root path the be consistent across releases. Alternatively, you can paste sample code here, but use markdown format. Jira doesn't process the content of that field, instead it's handled by Yetus and a markdown generator. > Support for batch version of checkAndMutate() > - > > Key: HBASE-8458 > URL: https://issues.apache.org/jira/browse/HBASE-8458 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Hari Mankude >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > The use case is that the user has multiple threads loading hundreds of keys > into a hbase table. Occasionally there are collisions in the keys being > uploaded by different threads. So for correctness, it is required to do > checkAndMutate() instead of a put(). However, doing a checkAndMutate() rpc > for every key update is non optimal. It would be good to have a batch version > of checkAndMutate() similar to batch put(). The client can partition the keys > on region boundaries. > The jira is NOT looking for any type of cross-row locking or multi-row > atomicity with checkAndMutate(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-8458) Support for batch version of checkAndMutate()
[ https://issues.apache.org/jira/browse/HBASE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135901#comment-17135901 ] Hudson commented on HBASE-8458: --- Results for branch master [build #1757 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1757/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/master/1757/General_20Nightly_20Build_20Report/] (x) {color:red}-1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1663//JDK8_Nightly_Build_Report_(Hadoop2)/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1757/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (x) {color:red}-1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://builds.apache.org/job/HBase%20Nightly/job/master/1757/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Support for batch version of checkAndMutate() > - > > Key: HBASE-8458 > URL: https://issues.apache.org/jira/browse/HBASE-8458 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Hari Mankude >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > The use case is that the user has multiple threads loading hundreds of keys > into a hbase table. Occasionally there are collisions in the keys being > uploaded by different threads. So for correctness, it is required to do > checkAndMutate() instead of a put(). However, doing a checkAndMutate() rpc > for every key update is non optimal. It would be good to have a batch version > of checkAndMutate() similar to batch put(). The client can partition the keys > on region boundaries. > The jira is NOT looking for any type of cross-row locking or multi-row > atomicity with checkAndMutate(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-8458) Support for batch version of checkAndMutate()
[ https://issues.apache.org/jira/browse/HBASE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135328#comment-17135328 ] Hudson commented on HBASE-8458: --- Results for branch branch-2 [build #2706 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2706/]: (/) *{color:green}+1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2706/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2706/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2706/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2706/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Support for batch version of checkAndMutate() > - > > Key: HBASE-8458 > URL: https://issues.apache.org/jira/browse/HBASE-8458 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Hari Mankude >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > The use case is that the user has multiple threads loading hundreds of keys > into a hbase table. Occasionally there are collisions in the keys being > uploaded by different threads. So for correctness, it is required to do > checkAndMutate() instead of a put(). However, doing a checkAndMutate() rpc > for every key update is non optimal. It would be good to have a batch version > of checkAndMutate() similar to batch put(). The client can partition the keys > on region boundaries. > The jira is NOT looking for any type of cross-row locking or multi-row > atomicity with checkAndMutate(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-8458) Support for batch version of checkAndMutate()
[ https://issues.apache.org/jira/browse/HBASE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135107#comment-17135107 ] Toshihiro Suzuki commented on HBASE-8458: - Pushed the addendum to mater. Resolving this. Thank you for reviewing this! [~elserj] > Support for batch version of checkAndMutate() > - > > Key: HBASE-8458 > URL: https://issues.apache.org/jira/browse/HBASE-8458 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Hari Mankude >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > The use case is that the user has multiple threads loading hundreds of keys > into a hbase table. Occasionally there are collisions in the keys being > uploaded by different threads. So for correctness, it is required to do > checkAndMutate() instead of a put(). However, doing a checkAndMutate() rpc > for every key update is non optimal. It would be good to have a batch version > of checkAndMutate() similar to batch put(). The client can partition the keys > on region boundaries. > The jira is NOT looking for any type of cross-row locking or multi-row > atomicity with checkAndMutate(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-8458) Support for batch version of checkAndMutate()
[ https://issues.apache.org/jira/browse/HBASE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135044#comment-17135044 ] Anoop Sam John commented on HBASE-8458: --- Excellent work [~brfrn169]. The Release Notes looks great too. > Support for batch version of checkAndMutate() > - > > Key: HBASE-8458 > URL: https://issues.apache.org/jira/browse/HBASE-8458 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Hari Mankude >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > The use case is that the user has multiple threads loading hundreds of keys > into a hbase table. Occasionally there are collisions in the keys being > uploaded by different threads. So for correctness, it is required to do > checkAndMutate() instead of a put(). However, doing a checkAndMutate() rpc > for every key update is non optimal. It would be good to have a batch version > of checkAndMutate() similar to batch put(). The client can partition the keys > on region boundaries. > The jira is NOT looking for any type of cross-row locking or multi-row > atomicity with checkAndMutate(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-8458) Support for batch version of checkAndMutate()
[ https://issues.apache.org/jira/browse/HBASE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135027#comment-17135027 ] Toshihiro Suzuki commented on HBASE-8458: - I've pushed this to branch-2. And I just created a new PR for the master branch for a small addendum. If the QA is okay, I'll merge it. > Support for batch version of checkAndMutate() > - > > Key: HBASE-8458 > URL: https://issues.apache.org/jira/browse/HBASE-8458 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Hari Mankude >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > The use case is that the user has multiple threads loading hundreds of keys > into a hbase table. Occasionally there are collisions in the keys being > uploaded by different threads. So for correctness, it is required to do > checkAndMutate() instead of a put(). However, doing a checkAndMutate() rpc > for every key update is non optimal. It would be good to have a batch version > of checkAndMutate() similar to batch put(). The client can partition the keys > on region boundaries. > The jira is NOT looking for any type of cross-row locking or multi-row > atomicity with checkAndMutate(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-8458) Support for batch version of checkAndMutate()
[ https://issues.apache.org/jira/browse/HBASE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134799#comment-17134799 ] Toshihiro Suzuki commented on HBASE-8458: - I just created a new PR for branch-2 to run QA. If the QA is okay, I'll merge it. https://github.com/apache/hbase/pull/1897 > Support for batch version of checkAndMutate() > - > > Key: HBASE-8458 > URL: https://issues.apache.org/jira/browse/HBASE-8458 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Hari Mankude >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > The use case is that the user has multiple threads loading hundreds of keys > into a hbase table. Occasionally there are collisions in the keys being > uploaded by different threads. So for correctness, it is required to do > checkAndMutate() instead of a put(). However, doing a checkAndMutate() rpc > for every key update is non optimal. It would be good to have a batch version > of checkAndMutate() similar to batch put(). The client can partition the keys > on region boundaries. > The jira is NOT looking for any type of cross-row locking or multi-row > atomicity with checkAndMutate(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-8458) Support for batch version of checkAndMutate()
[ https://issues.apache.org/jira/browse/HBASE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133358#comment-17133358 ] Hudson commented on HBASE-8458: --- Results for branch master [build #1754 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1754/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/master/1754/General_20Nightly_20Build_20Report/] (x) {color:red}-1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1600//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1754/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (x) {color:red}-1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://builds.apache.org/job/HBase%20Nightly/job/master/1754/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Support for batch version of checkAndMutate() > - > > Key: HBASE-8458 > URL: https://issues.apache.org/jira/browse/HBASE-8458 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Hari Mankude >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > The use case is that the user has multiple threads loading hundreds of keys > into a hbase table. Occasionally there are collisions in the keys being > uploaded by different threads. So for correctness, it is required to do > checkAndMutate() instead of a put(). However, doing a checkAndMutate() rpc > for every key update is non optimal. It would be good to have a batch version > of checkAndMutate() similar to batch put(). The client can partition the keys > on region boundaries. > The jira is NOT looking for any type of cross-row locking or multi-row > atomicity with checkAndMutate(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-8458) Support for batch version of checkAndMutate()
[ https://issues.apache.org/jira/browse/HBASE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132894#comment-17132894 ] Toshihiro Suzuki commented on HBASE-8458: - [~elserj] Thank you very much! {quote} I see this is tagged for 2.4.0 – is there appetite to cherry-pick this back to branch-2 as well? {quote} Yes, I will work on doing that. {quote} Could you write some release notes and then resolve this, Toshihiro Suzuki? {quote} Sure. After woking on branch-2, will do that. Thanks. > Support for batch version of checkAndMutate() > - > > Key: HBASE-8458 > URL: https://issues.apache.org/jira/browse/HBASE-8458 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Hari Mankude >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > The use case is that the user has multiple threads loading hundreds of keys > into a hbase table. Occasionally there are collisions in the keys being > uploaded by different threads. So for correctness, it is required to do > checkAndMutate() instead of a put(). However, doing a checkAndMutate() rpc > for every key update is non optimal. It would be good to have a batch version > of checkAndMutate() similar to batch put(). The client can partition the keys > on region boundaries. > The jira is NOT looking for any type of cross-row locking or multi-row > atomicity with checkAndMutate(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-8458) Support for batch version of checkAndMutate()
[ https://issues.apache.org/jira/browse/HBASE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132855#comment-17132855 ] Josh Elser commented on HBASE-8458: --- I've pushed this to master. I see this is tagged for 2.4.0 – is there appetite to cherry-pick this back to branch-2 as well? Could you write some release notes and then resolve this, [~brfrn169]? Thanks for your great work! > Support for batch version of checkAndMutate() > - > > Key: HBASE-8458 > URL: https://issues.apache.org/jira/browse/HBASE-8458 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Hari Mankude >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > The use case is that the user has multiple threads loading hundreds of keys > into a hbase table. Occasionally there are collisions in the keys being > uploaded by different threads. So for correctness, it is required to do > checkAndMutate() instead of a put(). However, doing a checkAndMutate() rpc > for every key update is non optimal. It would be good to have a batch version > of checkAndMutate() similar to batch put(). The client can partition the keys > on region boundaries. > The jira is NOT looking for any type of cross-row locking or multi-row > atomicity with checkAndMutate(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-8458) Support for batch version of checkAndMutate()
[ https://issues.apache.org/jira/browse/HBASE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121379#comment-17121379 ] Josh Elser commented on HBASE-8458: --- One final, gentle nudge for those not following in [https://github.com/apache/hbase/pull/1648] – I think this is ready. I'll commit this tmrw if I don't hear back from anyone. > Support for batch version of checkAndMutate() > - > > Key: HBASE-8458 > URL: https://issues.apache.org/jira/browse/HBASE-8458 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Hari Mankude >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > The use case is that the user has multiple threads loading hundreds of keys > into a hbase table. Occasionally there are collisions in the keys being > uploaded by different threads. So for correctness, it is required to do > checkAndMutate() instead of a put(). However, doing a checkAndMutate() rpc > for every key update is non optimal. It would be good to have a batch version > of checkAndMutate() similar to batch put(). The client can partition the keys > on region boundaries. > The jira is NOT looking for any type of cross-row locking or multi-row > atomicity with checkAndMutate(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-8458) Support for batch version of checkAndMutate()
[ https://issues.apache.org/jira/browse/HBASE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099405#comment-17099405 ] Toshihiro Suzuki commented on HBASE-8458: - Could someone please review the following PR? Actually this feature is important for our usecase. https://github.com/apache/hbase/pull/1648 The highlights of the changes are as follows: - Introduced CheckAndMutate class that's used to perform CheckAndMutate operations. The following is the JavaDoc for this class: {code} * Used to perform CheckAndMutate operations. Currently {@link Put}, {@link Delete} * and {@link RowMutations} are supported. * * This has a fluent style API to instantiate it, the code is like: * * * // A CheckAndMutate operation where do the specified action if the column (specified by the * // family and the qualifier) of the row equals to the specified value * CheckAndMutate checkAndMutate = new CheckAndMutate(row) * .ifEquals(family, qualifier, value) * .action(put); * * // A CheckAndMutate operation where do the specified action if the column (specified by the * // family and the qualifier) of the row doesn't exist * CheckAndMutate checkAndMutate = new CheckAndMutate(row) * .ifNotExists(family, qualifier) * .action(put); * * // A CheckAndMutate operation where do the specified action if the row matches the filter * CheckAndMutate checkAndMutate = new CheckAndMutate(row) * .ifMatches(filter) * .action(delete); * * {code} - Added new checkAndMutate APIs to the Table and AsyncTable interfaces. -- The APIs for the Table interface: {code} /** * checkAndMutate that atomically checks if a row matches the specified condition. If it does, * it performs the specified action. * * @param checkAndMutate The CheckAndMutate object. * @return boolean that represents the result for the CheckAndMutate. * @throws IOException if a remote or network exception occurs. */ default boolean checkAndMutate(CheckAndMutate checkAndMutate) throws IOException { return checkAndMutate(Collections.singletonList(checkAndMutate))[0]; } /** * Batch version of checkAndMutate. * * @param checkAndMutates The list of CheckAndMutate. * @return A array of boolean that represents the result for each CheckAndMutate. * @throws IOException if a remote or network exception occurs. */ default boolean[] checkAndMutate(List checkAndMutates) throws IOException { throw new NotImplementedException("Add an implementation!"); } {code} -- The APIs for the AsyncTable interface: {code} /** * checkAndMutate that atomically checks if a row matches the specified condition. If it does, * it performs the specified action. * * @param checkAndMutate The CheckAndMutate object. * @return A {@link CompletableFuture}s that represent the result for the CheckAndMutate. */ CompletableFuture checkAndMutate(CheckAndMutate checkAndMutate); /** * Batch version of checkAndMutate. * * @param checkAndMutates The list of CheckAndMutate. * @return A list of {@link CompletableFuture}s that represent the result for each * CheckAndMutate. */ List> checkAndMutate(List checkAndMutates); /** * A simple version of batch checkAndMutate. It will fail if there are any failures. * * @param checkAndMutates The list of rows to apply. * @return A {@link CompletableFuture} that wrapper the result boolean list. */ default CompletableFuture> checkAndMutateAll( List checkAndMutates) { return allOf(checkAndMutate(checkAndMutates)); } {code} - Deprecated the old checkAndMutate APIs of the Table and AsyncTable interfaces. - This has Protocol Buffers level changes. I moved MultiRequest#condition to RegionAction and MultiResponse#processed to RegionActionResult. This is already disscussed in the Jira: https://issues.apache.org/jira/browse/HBASE-8458?focusedCommentId=17064155=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17064155 Thanks. > Support for batch version of checkAndMutate() > - > > Key: HBASE-8458 > URL: https://issues.apache.org/jira/browse/HBASE-8458 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Hari Mankude >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > The use case is that the user has multiple threads loading hundreds of keys > into a hbase table. Occasionally there are collisions in the keys being > uploaded by different threads. So for correctness, it is required to do > checkAndMutate() instead of a put(). However, doing a checkAndMutate() rpc > for every key update is non optimal. It would be good to have a batch version > of checkAndMutate() similar to batch put(). The client can partition the keys > on region boundaries. > The jira is NOT looking for any type of