[jira] [Comment Edited] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15605275#comment-15605275 ] Edward Bortnikov edited comment on HBASE-16608 at 10/25/16 1:12 PM: Thanks, everyone, for the awesome teamwork on this jira. The work continues in HBASE-16417 (performance), HBASE-16851 (documentation) and HBASE-16421 (CellChunkMap impl). was (Author: ebortnik): Thanks, everyone, for the awesome teamwork on this jira. The work continues in HBASE-16417 (performance), HBASE-16851 (documentation) and the TBD jira for CellChunkMap impl. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-Final.patch, HBASE-16608-Final.patch, > HBASE-16608-V01.patch, HBASE-16608-V03.patch, HBASE-16608-V04.patch, > HBASE-16608-V08.patch, HBASE-16608-V09.patch, HBASE-16608-V09.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15605275#comment-15605275 ] Edward Bortnikov edited comment on HBASE-16608 at 10/25/16 1:10 PM: Thanks, everyone, for the awesome teamwork on this jira. The work continues in HBASE-16417 (performance), HBASE-16851 (documentation) and the TBD jira for CellChunkMap impl. was (Author: ebortnik): Thanks, everyone, ofr the awesome teamwork on this jira. The work continues in HBASE-16417 (performance), HBASE-16851 (documentation) and the TBD jira for CellChunkMap impl. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-Final.patch, HBASE-16608-Final.patch, > HBASE-16608-V01.patch, HBASE-16608-V03.patch, HBASE-16608-V04.patch, > HBASE-16608-V08.patch, HBASE-16608-V09.patch, HBASE-16608-V09.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15605275#comment-15605275 ] Edward Bortnikov edited comment on HBASE-16608 at 10/25/16 1:10 PM: Thanks, everyone, ofr the awesome teamwork on this jira. The work continues in HBASE-16417 (performance), HBASE-16851 (documentation) and the TBD jira for CellChunkMap impl. was (Author: ebortnik): Thanks, everyone, on the awesome teamwork on submitting this jira. The work continues in HBASE-16417 (performance), HBASE-16851 (documentation) and the TBD jira for CellChunkMap impl. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-Final.patch, HBASE-16608-Final.patch, > HBASE-16608-V01.patch, HBASE-16608-V03.patch, HBASE-16608-V04.patch, > HBASE-16608-V08.patch, HBASE-16608-V09.patch, HBASE-16608-V09.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[
https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591021#comment-15591021
]
ramkrishna.s.vasudevan edited comment on HBASE-16608 at 10/20/16 7:00 AM:
--
Latest test results after the CollectionMSLAB fix to avoid Full GC.
{code}
Default memstore on trunk 50 threads
Min: 808073ms Max: 844064ms Avg: 833224ms
compacting memstore on trunk 50 threads
Min: 822968ms Max: 858813ms Avg: 848612ms
{code}
||Memstore type||Number of GC pauses||Total GC in secs||No of mixed GC||Avg
mixed GC in secs||No of young GC||Avg young GC in secs||
|default memstore|600|95.45|49|0.410|398|0.155|
|compacting memstore|724|128.34|92|0.552|452|0.126|
I think the memory leak is solved as per this run and we can see if we are ok
to commit this patch after you guys can review it.
was (Author: ram_krish):
Latest test results after the CollectionMSLAB fix to avoid Full GC.
{code}
Default memstore on trunk 50 threads
Min: 808073ms Max: 844064ms Avg: 833224ms
compacting memstore on trunk 50 threads
Min: 822968ms Max: 858813ms Avg: 848612ms
{code}
||Number of GC pauses||Total GC in secs||No of mixed GC||Avg mixed GC in
secs||No of young GC||Avg young GC in secs||
|default memstore|600|95.45|49|0.410|398|0.155|
|compacting memstore|724|128.34|92|0.552|452|0.126|
I think the memory leak is solved as per this run and we can see if we are ok
to commit this patch after you guys can review it.
> Introducing the ability to merge ImmutableSegments without copy-compaction or
> SQM usage
> ---
>
> Key: HBASE-16608
> URL: https://issues.apache.org/jira/browse/HBASE-16608
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch,
> HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch,
> HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch,
> HBASE-16608-V04.patch, HBASE-16608-V08.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588008#comment-15588008 ] Anoop Sam John edited comment on HBASE-16608 at 10/19/16 10:56 AM: --- Another imp thing which came out now We elsewhere discussed abt the duplication of either cell data or index (In compaction or merge) case and Ram just mentioned it once again. Just case of merge which just copies the cell refs from 2+ arrays into one. It is not temp existance of duplciated refs and it is not just one copy. Let me explain with an eg: 2 segments in pipeline getting merged where each having 100 cells. So each having a cell array with 100 refs. (Each ref is 8 bytes) Now we make a merge and that create a new segment and that contain a cell array with 200 entries. Ya there are no copies of the cells. But new 200 cell refs coming up. So before 1st merge there were 200 cell refs totally and now it is 400. Now say one more segment came in and we merge together this new and old *merged* one. Consider new segment also have 100 cells. So before this merge op there are totally 400 + 100 = 500 refs 2nd merge also wont touch the segments and it creates a new segment and new array will contain 300 refs. So totally 800 refs. Actual cell objects are 300 and 2 merges made the refs# to be 800. So this not even doubling.. Like this if there are so many merge happens before a flush we will waste lots and lots of heap space with filling up refs. was (Author: anoop.hbase): Another imp thing which came out now We elsewhere discussed abt the duplication of either cell data or index (In compaction or merge) case and Ram just mentioned it once again. Just case of merge which just copies the cell refs from 2+ arrays into one. It is not temp existance of duplciated refs and it is not just one copy. Let me explain with an eg: 2 segments in pipeline getting merged where each having 100 cells. So each having a cell array with 100 refs. (Each ref is 8 bytes) Now we make a merge and that create a new segment and that contain a cell array with 200 entries. Ya there are no copies of the cells. But new 200 cell refs coming up. So before 1st merge there were 200 cell refs totally and now it is 400. Now say one more segment came in and we merge together this new and old *merged* one. Consider new segment also have 100 cells. So before this merge op there are totally 400 + 100 = 500 refs 2nd merge also wont touch the segments and it creates a new segment and new array will contain 300 refs. So totally 800 refs. Actual cell objects are 300 and 2 merges made the refs# to be 800. So this not even doubling.. Like this if there are so many merge happens before a flush we will waste lots and lots of heap space with filling up refs. Just test with this attached patch which is not solving the problem correctly. It is a hack and that can not be considered any way. Just to make sure this fix is improving the full GC pattern am sending u. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch, HBASE-16608-V08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15587876#comment-15587876 ] Anoop Sam John edited comment on HBASE-16608 at 10/19/16 8:14 AM: -- NP.. We are helping you in all possible ways :-) I just now did a test with the way I mentioned (ie. Collection backed new MSLAB impl to be used in merged segment). It behave like DefaultMS ie. there is no miss in MSLABPool#getChunk. We always get from pool. I did not see ur patch in detail what is the issue there which make the return of chunks not working normally. As said, we will discuss abt the merge thing and then go ahead with this. Any way believe me, the Collection based MALAB and usage of that, it looks much much simpler now :-) JFYI sharing the test results was (Author: anoop.hbase): NP.. We are helping you will all possible ways :-) I just now did a test with the way I mentioned (ie. Collection backed new MSLAB impl to be used in merged segment). It behave like DefaultMS ie. there is no miss in MSLABPool#getChunk. We always get from pool. I did not see ur patch in detail what is the issue there which make the return of chunks not working normally. As said, we will discuss abt the merge thing and then go ahead with this. Any way believe me, the Collection based MALAB and usage of that, it looks much much simpler now :-) JFYI sharing the test results > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch, HBASE-16608-V08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
