[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521857#comment-16521857 ] huaxiang sun commented on HBASE-20710: -- Thanks [~stack] for committing the change. > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 3.0.0, 2.1.0, 2.0.2 > > Attachments: 2.0621.2.12782.alloc.svg, 2.0623.111354.alloc.svg, > HBASE-20710-master-v001.patch, HBASE-20710-master-v002.patch, > HBASE-20710-master-v002.patch, HBASE-20710.master.001.patch, > alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521601#comment-16521601
]
Hudson commented on HBASE-20710:
Results for branch master
[build #375 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/375/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/375//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/375//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/375//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: 2.0621.2.12782.alloc.svg, 2.0623.111354.alloc.svg,
> HBASE-20710-master-v001.patch, HBASE-20710-master-v002.patch,
> HBASE-20710-master-v002.patch, HBASE-20710.master.001.patch,
> alloc-put-fix.svg, alloc-put-orig.svg
>
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB
> backed ByteBuffKeyValueCell (which is default with Netty Rpc),
> cell.getFamilyArray() will call cloneFamily() and there is again a
> cloneFamily() in the following line of the code. since this is the critical
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521475#comment-16521475
]
Hudson commented on HBASE-20710:
Results for branch branch-2.0
[build #466 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/466/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/466//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/466//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/466//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: 2.0621.2.12782.alloc.svg, 2.0623.111354.alloc.svg,
> HBASE-20710-master-v001.patch, HBASE-20710-master-v002.patch,
> HBASE-20710-master-v002.patch, HBASE-20710.master.001.patch,
> alloc-put-fix.svg, alloc-put-orig.svg
>
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB
> backed ByteBuffKeyValueCell (which is default with Netty Rpc),
> cell.getFamilyArray() will call cloneFamily() and there is again a
> cloneFamily() in the following line of the code. since this is the critical
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521473#comment-16521473
]
Hudson commented on HBASE-20710:
Results for branch branch-2
[build #900 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/900/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/900//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/900//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/900//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: 2.0621.2.12782.alloc.svg, 2.0623.111354.alloc.svg,
> HBASE-20710-master-v001.patch, HBASE-20710-master-v002.patch,
> HBASE-20710-master-v002.patch, HBASE-20710.master.001.patch,
> alloc-put-fix.svg, alloc-put-orig.svg
>
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB
> backed ByteBuffKeyValueCell (which is default with Netty Rpc),
> cell.getFamilyArray() will call cloneFamily() and there is again a
> cloneFamily() in the following line of the code. since this is the critical
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521367#comment-16521367
]
stack commented on HBASE-20710:
---
Pushed a checkstyle addendum to branch-2.0+
commit 959d9c16f305fbed0298e6da4fab43f4c1a5e478 (HEAD -> m, origin/master,
origin/HEAD)
Author: Michael Stack
Date: Sat Jun 23 20:41:47 2018 -0700
HBASE-20710 extra cloneFamily() in Mutation.add(Cell); ADDENDUM checkstyle
diff --git
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
index 3d935011dd..dc8199df72 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
@@ -115,10 +115,10 @@ public abstract class Mutation extends
OperationWithAttributes implements Row, C
super(clone);
this.row = clone.getRow();
this.ts = clone.getTimestamp();
-this.familyMap = clone.getFamilyCellMap().entrySet().stream()
- .collect(Collectors.toMap(e -> e.getKey(), e -> new
ArrayList<>(e.getValue()), (k, v) -> {
-throw new RuntimeException("collisions!!!");
- }, () -> new TreeMap<>(Bytes.BYTES_COMPARATOR)));
+this.familyMap = clone.getFamilyCellMap().entrySet().stream().
+ collect(Collectors.toMap(e -> e.getKey(), e -> new
ArrayList<>(e.getValue()), (k, v) -> {
+throw new RuntimeException("collisions!!!");
+ }, () -> new TreeMap<>(Bytes.BYTES_COMPARATOR)));
}
> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: 2.0621.2.12782.alloc.svg, 2.0623.111354.alloc.svg,
> HBASE-20710-master-v001.patch, HBASE-20710-master-v002.patch,
> HBASE-20710-master-v002.patch, HBASE-20710.master.001.patch,
> alloc-put-fix.svg, alloc-put-orig.svg
>
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB
> backed ByteBuffKeyValueCell (which is default with Netty Rpc),
> cell.getFamilyArray() will call cloneFamily() and there is again a
> cloneFamily() in the following line of the code. since this is the critical
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521353#comment-16521353 ] stack commented on HBASE-20710: --- I tried the patch. Here is before... [^2.0621.2.12782.alloc.svg] and here is with the patch applied [^2.0623.111354.alloc.svg] You can see the two clone families in the first graph above the CallRunner#run hump. Hard to find the single clone family in second graph... its really small. > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.2 > > Attachments: 2.0621.2.12782.alloc.svg, 2.0623.111354.alloc.svg, > HBASE-20710-master-v001.patch, HBASE-20710-master-v002.patch, > HBASE-20710-master-v002.patch, HBASE-20710.master.001.patch, > alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521351#comment-16521351
]
Hadoop QA commented on HBASE-20710:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
29s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
30s{color} | {color:red} hbase-client: The patch generated 2 new + 23 unchanged
- 0 fixed = 25 total (was 23) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
24s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 59s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
55s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20710 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928887/HBASE-20710.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux cc85110d7180 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 55147c7eae |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/13360/artifact/patchprocess/diff-checkstyle-hbase-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13360/testReport/ |
| Max.
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521340#comment-16521340 ] stack commented on HBASE-20710: --- Bit confusing but .001 is rebase (its [~huaxiang]'s .002 fixed so works on master branch) > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.2 > > Attachments: HBASE-20710-master-v001.patch, > HBASE-20710-master-v002.patch, HBASE-20710-master-v002.patch, > HBASE-20710.master.001.patch, alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521253#comment-16521253 ] stack commented on HBASE-20710: --- +1 And +1 for branch-2.0. Why tests fail? Thanks [~huaxiang] > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.2 > > Attachments: HBASE-20710-master-v001.patch, > HBASE-20710-master-v002.patch, HBASE-20710-master-v002.patch, > alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519595#comment-16519595
]
Hadoop QA commented on HBASE-20710:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 3s{color}
| {color:red} HBASE-20710 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.7.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20710 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928647/HBASE-20710-master-v002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13336/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.2
>
> Attachments: HBASE-20710-master-v001.patch,
> HBASE-20710-master-v002.patch, HBASE-20710-master-v002.patch,
> alloc-put-fix.svg, alloc-put-orig.svg
>
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB
> backed ByteBuffKeyValueCell (which is default with Netty Rpc),
> cell.getFamilyArray() will call cloneFamily() and there is again a
> cloneFamily() in the following line of the code. since this is the critical
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519592#comment-16519592 ] huaxiang sun commented on HBASE-20710: -- reattach the patch as there is no test run against it. > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.2 > > Attachments: HBASE-20710-master-v001.patch, > HBASE-20710-master-v002.patch, HBASE-20710-master-v002.patch, > alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513182#comment-16513182 ] huaxiang sun commented on HBASE-20710: -- Yeah, for the table.put path, we can save copyFamily as it has been already copied out. For table.batch case (cellblock), this copy is unavoidable. I have attached the allocation profiles for the put case (netty rpc server), with the patch, the cloneFamily() disappears in the allocation profile. It was run with PE randomWrite. > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.1 > > Attachments: HBASE-20710-master-v001.patch, > HBASE-20710-master-v002.patch, alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512712#comment-16512712 ] Anoop Sam John commented on HBASE-20710: I see.. In IndividualBytesFieldCell we dont copy family at all.. > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.1 > > Attachments: HBASE-20710-master-v001.patch, > HBASE-20710-master-v002.patch, alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512575#comment-16512575 ] huaxiang sun commented on HBASE-20710: -- Thanks [~anoop.hbase]. Checking at the following line of the code https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java#L502 ByteString's toByteArray() allocate space and do a deep copy. https://github.com/google/protobuf/blob/master/java/core/src/main/java/com/google/protobuf/ByteString.java#L637 That is the reason I think IndividualBytesFieldCell is backed by each individual field. Something missing here? > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.1 > > Attachments: HBASE-20710-master-v001.patch, > HBASE-20710-master-v002.patch, alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512241#comment-16512241
]
Anoop Sam John commented on HBASE-20710:
bq. if (cell instanceof IndividualBytesFieldCell) {
Why the check is based this way? Other parts we have checks on instanceof
ByteBufferExtendedCell.. Why not that here?
If the Cell is not DBB backed, the getXXXArray() wont do any extra copy any
way.. And the clone below is any way needed..
> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.1
>
> Attachments: HBASE-20710-master-v001.patch,
> HBASE-20710-master-v002.patch, alloc-put-fix.svg, alloc-put-orig.svg
>
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB
> backed ByteBuffKeyValueCell (which is default with Netty Rpc),
> cell.getFamilyArray() will call cloneFamily() and there is again a
> cloneFamily() in the following line of the code. since this is the critical
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511458#comment-16511458 ] huaxiang sun commented on HBASE-20710: -- I put a v2 which just adds an optimization for IndividualBytesFieldCell based on Stack's patch. Cpu profile does not show much, the alloc profile during a PE randomWrite run shows the cloneFamily() allocation gone under toPut(). Let me attach alloc profile with/without the patch. > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.1 > > Attachments: HBASE-20710-master-v001.patch, > HBASE-20710-master-v002.patch > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510064#comment-16510064 ] huaxiang sun commented on HBASE-20710: -- If this is the case, I will upload a new patch which is a minor update based on the patch you have [~stack]. It is going to help the case when the cell is IndividualBytesFieldCell. I will put a new patch shortly. > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.1 > > Attachments: HBASE-20710-master-v001.patch > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509990#comment-16509990 ] stack commented on HBASE-20710: --- bq. Call this out in the doc that the caller needs to make sure the input family and family cell are same, is that enough? I don't think so. We need to protect ourselves against the bad stuff (Engineers' don't read doc., not unless they get really stuck smile). > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.1 > > Attachments: HBASE-20710-master-v001.patch > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509959#comment-16509959 ] huaxiang sun commented on HBASE-20710: -- Thanks Stack and now I got [~chia7712]'s point. Do the sanity check will introduce the extra cloneFamily() call and defeats the purpose. Call this out in the doc that the caller needs to make sure the input family and family cell are same, is that enough? I am going to run and put the results at my cluster. > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.1 > > Attachments: HBASE-20710-master-v001.patch > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509937#comment-16509937 ] huaxiang sun commented on HBASE-20710: -- Thanks [~mdrob] for review, will address the comments. The main idea is as follows: Cellblock: [family1:qualifer1, v1], [family1:qualifer2, v2], [family1:qualifer3, v3] cell1 cell2 cell3 The first family byte array "family1" is added to the map(familyAdded). For cell2, its family is read into familyFromCell(allocate once). After it finds out that it is the same family, it will use familyAdded to put the cell2 into the TreeMap(very fast). For cell3, the family is read into familyFromCell(already allocated, no new allocation is needed), it will again compare with familyAdded and reuse familyAdded for put into the TreeMap. For cell4 and on, there will be no new allocation for family, and familyFromCell is reused. With this, there is no need to clone family for each cell and save heap allocation. Compared with the pre-patch case, the save is huge as it calls cloneFamily() twice for each cell (cellblock case). Similar applies to normal put case. > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.1 > > Attachments: HBASE-20710-master-v001.patch > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509918#comment-16509918
]
stack commented on HBASE-20710:
---
Could the Cell family be different from the family passed in as an arg? We
don't check. Should we?
Mutation add(final byte[] family, Cell cell) throws IOException {
Keep this internal? public Put add(final byte[] family, Cell cell) throws
IOException {
You see benefit from dong the machinations in ProtobufUtil.java?
Thanks.
> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.1
>
> Attachments: HBASE-20710-master-v001.patch
>
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB
> backed ByteBuffKeyValueCell (which is default with Netty Rpc),
> cell.getFamilyArray() will call cloneFamily() and there is again a
> cloneFamily() in the following line of the code. since this is the critical
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509902#comment-16509902
]
Mike Drob commented on HBASE-20710:
---
Can you add javadoc for {{copyFamilyTo}}?
I'm having trouble following the main change, like when we think familyFromCell
and familyAdded are null or not. Can you walk me through the intent briefly?
> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.1
>
> Attachments: HBASE-20710-master-v001.patch
>
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB
> backed ByteBuffKeyValueCell (which is default with Netty Rpc),
> cell.getFamilyArray() will call cloneFamily() and there is again a
> cloneFamily() in the following line of the code. since this is the critical
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509307#comment-16509307
]
Hadoop QA commented on HBASE-20710:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
59s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 38s{color}
| {color:red} hbase-client generated 1 new + 102 unchanged - 1 fixed = 103
total (was 103) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
35s{color} | {color:red} hbase-client: The patch generated 1 new + 216
unchanged - 0 fixed = 217 total (was 216) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
57s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 37s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
52s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
12s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20710 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927432/HBASE-20710-master-v001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 050bdc42716a 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509286#comment-16509286
]
Chia-Ping Tsai commented on HBASE-20710:
{quote}Family can be different from Cell's family if you are referring to the
previous check.
{quote}
Pardon me. why the families can be different? Given the Put#add(byte[], Cell)
is Public API, is it ok to pass different families?
> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.1
>
> Attachments: HBASE-20710-master-v001.patch
>
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB
> backed ByteBuffKeyValueCell (which is default with Netty Rpc),
> cell.getFamilyArray() will call cloneFamily() and there is again a
> cloneFamily() in the following line of the code. since this is the critical
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509281#comment-16509281 ] huaxiang sun commented on HBASE-20710: -- Thanks Chia-Ping. Family can be different from Cell's family if you are referring to the previous check. Just found that the idea can be applied to branch-1 as well. > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.1 > > Attachments: HBASE-20710-master-v001.patch > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509280#comment-16509280 ] Chia-Ping Tsai commented on HBASE-20710: Should we check that the family is same with the cell's family? > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.1 > > Attachments: HBASE-20710-master-v001.patch > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509279#comment-16509279
]
Chia-Ping Tsai commented on HBASE-20710:
{code:java}
+// TODO: It is fraught with risk if user pass the wrong row.
+// Throwing the IllegalArgumentException is more suitable I'd say.
+if (!CellUtil.matchingRows(cell, this.row)) {
+ throw new WrongRowIOException("The row in " + cell.toString() +
+ " doesn't match the original one " +
Bytes.toStringBinary(this.row));
+}{code}
agreed. A unchecked exception (IllegalArgumentException) is more suitable.
> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.1
>
> Attachments: HBASE-20710-master-v001.patch
>
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB
> backed ByteBuffKeyValueCell (which is default with Netty Rpc),
> cell.getFamilyArray() will call cloneFamily() and there is again a
> cloneFamily() in the following line of the code. since this is the critical
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507053#comment-16507053 ] stack commented on HBASE-20710: --- @hsun do as you see best. Above was just confirmation that another sees similar to what you were seeing. Sounds good > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.1 > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507046#comment-16507046 ] huaxiang sun commented on HBASE-20710: -- Thanks Stack. For the case of MutationProto (not cell block), it is looping cells for each family, the family should be checked once for all cells under it (no need to clone for each cell), will see if similar approach can be taken for cell bock. > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.1 > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507030#comment-16507030
]
stack commented on HBASE-20710:
---
+1
I had this as a 'fix' in branch:
diff --git
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
index a6ddc14cca..1b750b950e 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
@@ -788,11 +788,10 @@ public abstract class Mutation extends
OperationWithAttributes implements Row, C
" doesn't match the original one " + Bytes.toStringBinary(this.row));
}
-if (cell.getFamilyArray() == null || cell.getFamilyLength() == 0) {
+byte [] family = CellUtil.cloneFamily(cell);
+if (family == null || family.length == 0) {
throw new IllegalArgumentException("Family cannot be null");
}
-
-byte[] family = CellUtil.cloneFamily(cell);
if (cell instanceof ExtendedCell) {
getCellList(family).add(cell);
} else {
The offence is the extra copy and the deserializations to find lengths to use
locating the family bytes to copy.
Good one [~huaxiang].
> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB
> backed ByteBuffKeyValueCell (which is default with Netty Rpc),
> cell.getFamilyArray() will call cloneFamily() and there is again a
> cloneFamily() in the following line of the code. since this is the critical
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
