[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140915#comment-15140915 ] ramkrishna.s.vasudevan commented on HBASE-15204: Ping!! > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, HBASE-15204_4.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141801#comment-15141801 ] stack commented on HBASE-15204: --- +1 > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, HBASE-15204_4.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142379#comment-15142379 ] Hudson commented on HBASE-15204: FAILURE: Integrated in HBase-Trunk_matrix #700 (See [https://builds.apache.org/job/HBase-Trunk_matrix/700/]) HBASE-15204 Try to estimate the cell count for adding into WALEdit (Ram) (ramkrishna: rev fec97338931f2617ddb99bf7faad67d0a0ee2ddf) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, HBASE-15204_4.patch, HBASE-15204_branch-1.patch, > WAlEdit_add_allocation.jpg, WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142327#comment-15142327 ] Hudson commented on HBASE-15204: SUCCESS: Integrated in HBase-1.3-IT #494 (See [https://builds.apache.org/job/HBase-1.3-IT/494/]) HBASE-15204 Try to estimate the cell count for adding into WALEdit (Ram) (ramkrishna: rev cd2b4dfa1242a5febfc1be517c5d84cc75fb1723) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, HBASE-15204_4.patch, HBASE-15204_branch-1.patch, > WAlEdit_add_allocation.jpg, WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142341#comment-15142341 ] Hudson commented on HBASE-15204: SUCCESS: Integrated in HBase-1.3 #549 (See [https://builds.apache.org/job/HBase-1.3/549/]) HBASE-15204 Try to estimate the cell count for adding into WALEdit (Ram) (ramkrishna: rev cd2b4dfa1242a5febfc1be517c5d84cc75fb1723) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, HBASE-15204_4.patch, HBASE-15204_branch-1.patch, > WAlEdit_add_allocation.jpg, WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[
https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138837#comment-15138837
]
Hadoop QA commented on HBASE-15204:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 12s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-server
(total was 216, now 216). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
21m 33s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 79m 12s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 80m 33s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 202m 22s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-09 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12787020/HBASE-15204_4.patch |
| JIRA Issue | HBASE-15204 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140304#comment-15140304 ] ramkrishna.s.vasudevan commented on HBASE-15204: I think the report was clean this time. bq.Patch generated 1 new checkstyle issues in hbase-server (total was 216, now 216). No checkstyle added newly. bq../hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:2940:3: error: Method length is 310 lines (max allowed is 150). This was not added by this patch. So will not change it here because it needs proper refactoring. Ping for reviews [~saint@gmail.com]. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, HBASE-15204_4.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[
https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137184#comment-15137184
]
Hadoop QA commented on HBASE-15204:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 13s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-server
(total was 216, now 216). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 25s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 6s
{color} | {color:red} hbase-server introduced 1 new FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 99m 24s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 103m 38s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 247m 12s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Dead store to walEdit in
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion$BatchOperation)
At
HRegion.java:org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion$BatchOperation)
At HRegion.java:[line 2954] |
\\
\\
|| Subsystem ||
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[
https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136852#comment-15136852
]
Hadoop QA commented on HBASE-15204:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 3s {color}
| {color:red} HBASE-15204 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/latest/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12786792/HBASE-15204_3.patch |
| JIRA Issue | HBASE-15204 |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/476/console |
This message was automatically generated.
> Try to estimate the cell count for adding into WALEdit
> --
>
> Key: HBASE-15204
> URL: https://issues.apache.org/jira/browse/HBASE-15204
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15204.patch, HBASE-15204_1.patch,
> HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch,
> WAlEdit_add_allocation.jpg, WAlEdit_add_allocation_after_patch.jpg
>
>
> The write path profiling shows that when we try to add Cells to WALEdits we
> try to do a lot of Array copy inorder to grow the Arraylist backing the
> WALEdits. In a simple one min profiling of the write path with 50 YCSB
> threads shows around 261MB of allocation done for the Array copy to happen.
> We can try to avoid that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[
https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136667#comment-15136667
]
Anoop Sam John commented on HBASE-15204:
{code}
2779public void addtoCellCountFromCP(int count) {
2780 this.cellCountFromCP += count;
2781}
2782
2783public int getCellCountFromCP() {
2784 return this.cellCountFromCP;
2785}
{code}
We don'e even need this.
{code}
int cellCount = batchOp.getCellCountFromCP();
3128 for (int i = firstIndex; !isInReplay && i < lastIndexExclusive;
i++) {
for (List cells : familyMaps[i].values()) {
3143 cellCount += cells.size();
3144}
{code}
Within the outer for loop, we can get the cell count in wal edit from cp
associated with the corresponding Mutation.
batchOp.walEditsFromCoprocessors will give the WALEdit from which u can know.
That will be the exact count need also. Because getCellCountFromCP() will
give all Mutation's CP returned WALEdit cell count. We will be splitting these
mutations into mini batches as per row lock avail.
> Try to estimate the cell count for adding into WALEdit
> --
>
> Key: HBASE-15204
> URL: https://issues.apache.org/jira/browse/HBASE-15204
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15204.patch, HBASE-15204_1.patch,
> HBASE-15204_1.patch, HBASE-15204_2.patch, WAlEdit_add_allocation.jpg,
> WAlEdit_add_allocation_after_patch.jpg
>
>
> The write path profiling shows that when we try to add Cells to WALEdits we
> try to do a lot of Array copy inorder to grow the Arraylist backing the
> WALEdits. In a simple one min profiling of the write path with 50 YCSB
> threads shows around 261MB of allocation done for the Array copy to happen.
> We can try to avoid that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136671#comment-15136671 ] Anoop Sam John commented on HBASE-15204: bq.walEdit = new WALEdit(cellCount); We don't use isInReplay while creating WALEdit. Pls see comment from Enis. We have to pass this. Also while in replay u dont want to use the cell count estimation improvement? > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[
https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136005#comment-15136005
]
Hadoop QA commented on HBASE-15204:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 2s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-server
(total was 284, now 284). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 27s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 112m 46s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 111m 29s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 267m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests |
hadoop.hbase.regionserver.TestMobStoreCompaction |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-06 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12786676/HBASE-15204_2.patch |
| JIRA Issue | HBASE-15204 |
| Optional Tests |
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135860#comment-15135860 ] ramkrishna.s.vasudevan commented on HBASE-15204: bq.So, the call to the cp contract should be updated to say something like, "if you add new cells, you MUST return the count of cells added in the result ... else they will be ignored"... something like that? Stack, I think I can explain here. There is nothing am doing with the CP call here. The code in doPreMutationHook calls all the prehooks and we collect all those WALEdits already. Am just using that WALEdits and counting the size. There is no need for any contract on the CP side for doing this. While we collect those WALEdits into the 'batchOp.walEditsFromCoprocessors' already. If there was no edit added from CP this count is going to be 0. No contract explicitly added here. Simple way to not pass this cellcout is just to add those cellCount to the BatchOp directly and then use a getter to add the actual cellCount. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135741#comment-15135741 ] Hudson commented on HBASE-15204: FAILURE: Integrated in HBase-Trunk_matrix #687 (See [https://builds.apache.org/job/HBase-Trunk_matrix/687/]) HBASE-15204 Try to estimate the cell count for adding into WALEdit (ramkrishna: rev 4e44f4f5050bf2720762f754d3756763026c0dbd) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135799#comment-15135799 ] Hudson commented on HBASE-15204: SUCCESS: Integrated in HBase-1.3-IT #486 (See [https://builds.apache.org/job/HBase-1.3-IT/486/]) HBASE-15204 Try to estimate the cell count for adding into WALEdit (ramkrishna: rev bd4a4ecf5e1e23321801d5f18dab1dc068ba43da) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135779#comment-15135779 ] stack commented on HBASE-15204: --- bq. Stack Not very clear on this part. If CP is returning some mutations - we explicitly add that to a WALEdit. I am just counting if something is there. That is only to get an exact count because that will also be added to the WALEdit that we create in doMiniBatchMutation. So, the call to the cp contract should be updated to say something like, "if you add new cells, you MUST return the count of cells added in the result ... else they will be ignored"... something like that? bq. But if we don't want the cellCountFromCP to be passed then am fine with it too. I thought since we can use it and get the exact count it is preferable to use it. It just is odd having this parameter on this macro method. I appreciate the improvement but having to pass the param down from on high is not clean. Thanks [~ram_krish] > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135719#comment-15135719 ] Hudson commented on HBASE-15204: SUCCESS: Integrated in HBase-1.3 #542 (See [https://builds.apache.org/job/HBase-1.3/542/]) HBASE-15204 Try to estimate the cell count for adding into WALEdit (ramkrishna: rev bd4a4ecf5e1e23321801d5f18dab1dc068ba43da) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135646#comment-15135646 ] ramkrishna.s.vasudevan commented on HBASE-15204: bq.the return is orthogonal to why we call the CP. We are now dependent on CP implementation to do work for us. [~saint@gmail.com] Not very clear on this part. If CP is returning some mutations - we explicitly add that to a WALEdit. I am just counting if something is there. That is only to get an exact count because that will also be added to the WALEdit that we create in doMiniBatchMutation. bq.It is kind of odd that doMiniBatch takes a cell count when the other param is a fat batchOp with description of what the edit its. If cell count should be anywhere, it should be in batchop? I can remove that if we don't want that. Ya in batchOp we could add the Cellcount but many of the mutations may not be applied in that batch. But if we don't want the cellCountFromCP to be passed then am fine with it too. I thought since we can use it and get the exact count it is preferable to use it. bq.Do we have to add a Cell at a time? Can we not pass the whole list in one go? LEt me check this out. Since WALEdit was backed by an Arraylist I thought if we can size that backing list properly we are fine with it. bq.I appreciate the savings in garbage but perhaps we can do it in a cleaner way. Okie. I can revert this patch and see what else can be done. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15134292#comment-15134292 ] Hudson commented on HBASE-15204: FAILURE: Integrated in HBase-1.3 #538 (See [https://builds.apache.org/job/HBase-1.3/538/]) HBASE-15204 Try to estimate the cell count for adding into WALEdit (Ram) (ramkrishna: rev f9ce069e15f5b66f02072b15d23df6409fd40f68) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133811#comment-15133811 ] ramkrishna.s.vasudevan commented on HBASE-15204: How can we check the logs of the failed test case - The surefire folder where the logs are printed? > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133802#comment-15133802 ] ramkrishna.s.vasudevan commented on HBASE-15204: The checkstyle seems to be not created by this patch. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15134295#comment-15134295 ] Hudson commented on HBASE-15204: SUCCESS: Integrated in HBase-1.3-IT #481 (See [https://builds.apache.org/job/HBase-1.3-IT/481/]) HBASE-15204 Try to estimate the cell count for adding into WALEdit (Ram) (ramkrishna: rev f9ce069e15f5b66f02072b15d23df6409fd40f68) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15134031#comment-15134031 ] Hudson commented on HBASE-15204: FAILURE: Integrated in HBase-Trunk_matrix #683 (See [https://builds.apache.org/job/HBase-Trunk_matrix/683/]) HBASE-15204 Try to estimate the cell count for adding into WALEdit (Ram) (ramkrishna: rev 6f6a8ed71fe98b83e8a8db974fc15b0d8597b174) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135460#comment-15135460 ] Enis Soztutar commented on HBASE-15204: --- bq. You don't want to preserve passing replay flag to WALEdit? We need to purge the replay stuff but in another patch? Not important but it is odd that we retain the WALEdit constructor that takes a replay flag. Replay is still being used by region replicas. We replay the edits read from the primary in the secondaries. Even if we purge dist log replay, the replay inside region might have to stay. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135448#comment-15135448 ] stack commented on HBASE-15204: --- A doPreMutationHook returning a count of cells is unexpected behavior; while convenient, the return is orthogonal to why we call the CP. We are now dependent on CP implementation to do work for us. It is kind of odd that doMiniBatch takes a cell count when the other param is a fat batchOp with description of what the edit its. If cell count should be anywhere, it should be in batchop? You don't want to preserve passing replay flag to WALEdit? We need to purge the replay stuff but in another patch? Not important but it is odd that we retain the WALEdit constructor that takes a replay flag. Do we have to add a Cell at a time? Can we not pass the whole list in one go? I appreciate the savings in garbage but perhaps we can do it in a cleaner way. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[
https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133792#comment-15133792
]
Hadoop QA commented on HBASE-15204:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 1s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-server
(total was 284, now 284). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 52s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 104m 9s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 45s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 161m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests | hadoop.hbase.mob.mapreduce.TestMobSweepJob
|
| | hadoop.hbase.client.TestBlockEvictionFromClient |
| JDK v1.8.0_72 Timed out junit tests | org.apache.hadoop.hbase.TestZooKeeper |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| |
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133673#comment-15133673 ] Anoop Sam John commented on HBASE-15204: +1 > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15130516#comment-15130516 ] Anoop Sam John commented on HBASE-15204: Yes as Stack said. More over, here we are considering the Cells passed via PayLoad cellblock but don't know abt the Cells in PB message. If the client is not having Codec, we might not get any number of Cells associated with Mutation. So may be this is an overkill? Passing number of Cells to Region API looks a bit odd IMO. bq. private ArrayList cells = new ArrayList(1); We can change this only in WALEdit ? Initialize with size 1 seems too less. At least the usage from doMiniBatchMutate(). Pass some bigger value (Like 10)? As it is initialized with size =1 , there will be many growing happening (Like to 2, 3, 4, 6, 9). When it is init with size 10 will be lesser #grows any way. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15204.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15131831#comment-15131831 ] ramkrishna.s.vasudevan commented on HBASE-15204: Thanks for the reviews. As I said this was only a first cut to show the change. Passing cellCount to batchMutate is not neat I agree. Let me see what to do. But one thing - this change is only for Clients that work with CellBlock and not for PB based clients. Still I would say this is better if we can get the estimate. Let me check for the retry failures case. But some how if we can get an estimate - even if it is slighly bigger we can reduce the count of the garbage. Changing the default size of the Arraylist in WAL is another way which I too thought but what to set as the initial value was not clear to me. Let me see if we can do something better. If not will leave this for now. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15204.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127971#comment-15127971 ] Ted Yu commented on HBASE-15204: Please add java doc for the new batchMutate() method. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15204.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128603#comment-15128603 ] stack commented on HBASE-15204: --- Good one [~ram_krish] Is that right in batchMutate? You pass in total amount but we may break up the batch as we retry yet you are passing same cellcount each time > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15204.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[
https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128408#comment-15128408
]
Ted Yu commented on HBASE-15204:
{code}
-WALEdit walEdit = new WALEdit(isInReplay);
+WALEdit walEdit = new WALEdit(cellCount);
{code}
We don't need to distinguish replay mode any more ?
> Try to estimate the cell count for adding into WALEdit
> --
>
> Key: HBASE-15204
> URL: https://issues.apache.org/jira/browse/HBASE-15204
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-15204.patch, WAlEdit_add_allocation.jpg,
> WAlEdit_add_allocation_after_patch.jpg
>
>
> The write path profiling shows that when we try to add Cells to WALEdits we
> try to do a lot of Array copy inorder to grow the Arraylist backing the
> WALEdits. In a simple one min profiling of the write path with 50 YCSB
> threads shows around 261MB of allocation done for the Array copy to happen.
> We can try to avoid that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[
https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128226#comment-15128226
]
Hadoop QA commented on HBASE-15204:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 27s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-server
(total was 422, now 423). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 53s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 112m 8s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 97m 23s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 255m 47s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests |
hadoop.hbase.replication.TestReplicationKillMasterRSCompressed |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-02 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12785744/HBASE-15204.patch |
| JIRA Issue | HBASE-15204 |
| Optional
