[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to branch-1 and master. Thanks for all the reviews. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, HBASE-15204_4.patch, HBASE-15204_branch-1.patch, > WAlEdit_add_allocation.jpg, WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Attachment: HBASE-15204_branch-1.patch Rebased patch for branch-1. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, HBASE-15204_4.patch, HBASE-15204_branch-1.patch, > WAlEdit_add_allocation.jpg, WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Attachment: HBASE-15204_4.patch A findbug was added due to the rebase on the latest code. This patch resolves that. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, HBASE-15204_4.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Status: Patch Available (was: Open) > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, HBASE-15204_4.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Status: Open (was: Patch Available) > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Attachment: HBASE-15204_3.patch Rebased patch. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Status: Patch Available (was: Open) > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Status: Open (was: Patch Available) > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Status: Patch Available (was: Open) > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > WAlEdit_add_allocation.jpg, WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Attachment: HBASE-15204_3.patch Attaching patch addressing comments. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > WAlEdit_add_allocation.jpg, WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Status: Open (was: Patch Available) > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Attachment: HBASE-15204_2.patch Just an updated version of the patch so that we don't pass on the cellCount. Let me know what you think [~saint@gmail.com]? > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Status: Patch Available (was: Reopened) > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.3.0 Status: Resolved (was: Patch Available) Pushed to master and branch-1. Thanks for the reviews. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Status: Open (was: Patch Available) > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15204.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Attachment: HBASE-15204_1.patch Updated patch. Does not pass the cellcount from RsRpcServices. Instead tries to find the cellCount from the familyMaps associated with each of the mutation and then sets the WALEdit. So anyway we are now having a value > 1 for the ArrayList backing the WALEdit. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > WAlEdit_add_allocation.jpg, WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Status: Patch Available (was: Open) > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[
https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-15204:
---
Attachment: HBASE-15204_1.patch
Reattaching the patch for hadoop QA. In my previous patch I was doing
{code}
cellCountFromCP += doPreMutationHook(batchOp);
{code}
But it is going to be executed single time only. So '+=' may not be needed. A
minor nit suggested by Anoop.
Tested the patch with YCSB and that TLAB allocation due to the ArrayList
expansion seems to be avoided as attached in the
WAlEdit_add_allocation_after_patch.jpg attachment.
> Try to estimate the cell count for adding into WALEdit
> --
>
> Key: HBASE-15204
> URL: https://issues.apache.org/jira/browse/HBASE-15204
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-15204.patch, HBASE-15204_1.patch,
> HBASE-15204_1.patch, WAlEdit_add_allocation.jpg,
> WAlEdit_add_allocation_after_patch.jpg
>
>
> The write path profiling shows that when we try to add Cells to WALEdits we
> try to do a lot of Array copy inorder to grow the Arraylist backing the
> WALEdits. In a simple one min profiling of the write path with 50 YCSB
> threads shows around 261MB of allocation done for the Array copy to happen.
> We can try to avoid that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Status: Patch Available (was: Open) > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15204.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Attachment: HBASE-15204.patch WAlEdit_add_allocation_after_patch.jpg WAlEdit_add_allocation.jpg Just attaching a first cut version of the patch to get feedback. We can also estimate the WALEdit size based on the number of Cells added to WALEdits by the CP. That is also covered here. But in order to pass the cellcount either we need to change the signature of batchMutate in Region or need to directly call the new batchMutate API with CellCount in HRegion directly. Attached jpgs shows the allocation size and count just for this Arraylist to grow. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15204.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
