subject:"\[jira\] \[Commented\] \(HBASE\-14921\) Memory optimizations"

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-28 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15443088#comment-15443088
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

[~anoop.hbase], [~ram_krish], [~stack], [~tedyu], please find my very belated, 
but very sincere THANK YOU! 

Looking forward to continue working with you all on the next issues!! :)

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Fix For: 2.0.0
>
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-24 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434264#comment-15434264
 ] 

Hudson commented on HBASE-14921:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1471 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/1471/])
HBASE-14921 In-Memory MemStore Flush and Compaction : Memory (anoopsamjohn: rev 
32c21f45942f9f45e34b28df47180fe6498fc337)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWalAndCompactingMemStoreFlush.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactingMemStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SegmentScanner.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingMemStore.java
* (add) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCellFlatSet.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreScanner.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SegmentFactory.java
* (edit) hbase-shell/src/main/ruby/hbase/admin.rb
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/VersionedSegmentsList.java
* (add) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CellArrayMap.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionWithInMemoryFlush.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreCompactor.java
* (add) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreCompactorIterator.java
* (add) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingToCellArrayMapMemStore.java
* (add) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CellFlatMap.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MutableSegment.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/ClassSize.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractMemStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CellSet.java


> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-23 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434222#comment-15434222
 ] 

stack commented on HBASE-14921:
---

Thanks [~anoop.hbase] I've not reviewed yet. Will raise new issues if issues. 
Thanks for commit (and thanks for the work [~anastas]

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-23 Thread Edward Bortnikov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434220#comment-15434220
 ] 

Edward Bortnikov commented on HBASE-14921:
--

Fantastic. Thanks, all, for helping out. Let's move on with the next jiras and 
make the whole thing happen. 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-23 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432618#comment-15432618
 ] 

Anoop Sam John commented on HBASE-14921:


Will commit tonight my time unless obejctions.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-23 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432202#comment-15432202
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


+1 to commit.
[~devaraj]
I think it is ok to commit it. We should add an ITBLL test case that runs with 
the new memstore impl. I think that is part of a follow up JIRA. Thanks you.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-22 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432102#comment-15432102
 ] 

Anoop Sam John commented on HBASE-14921:


[~ram_krish], [~saint@gmail.com] You good with latest patch?  Will see 
once. Any minor comments can fix and major we can add TODO for doing in 
followup issues.  I can do that and get it committed.  Thanks for the great 
work 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-22 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432100#comment-15432100
 ] 

Anoop Sam John commented on HBASE-14921:


These patches as such do not touch the existing memstore and write/flush path 
much.   On the new type of memstore and flush from there, ya we can do the 
ITBLL tests.  Can be done later also NP.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-22 Thread Devaraj Das (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431686#comment-15431686
 ] 

Devaraj Das commented on HBASE-14921:
-

I am trying to get hold of a rig for running itbll. But that might take some 
time, and if others feel comfortable getting this patch in, it's fine with me. 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-22 Thread Edward Bortnikov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431348#comment-15431348
 ] 

Edward Bortnikov commented on HBASE-14921:
--

Thanks for suggesting, [~devaraj]. We are actually not aware of ITBLL. We're 
generating high loads with YCSB already, but it's always good to have one more 
test. Is there any "golden standard" for ITBLL benchmark?  
[~saint@gmail.com], [~ram_krish], [~anoop.hbase] - do you have experience 
with this tool? Can help? Thanks!

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-22 Thread Devaraj Das (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431204#comment-15431204
 ] 

Devaraj Das commented on HBASE-14921:
-

Great work, [~anastas]. I'd like to know whether you ran ITBLL and such other 
"correctness" benchmarks using these family of patches. Given the magnitude of 
the changes, I was thinking we should get some runs of ITBLL on these. Pardon 
me if you have already covered that aspect earlier.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-22 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430659#comment-15430659
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

To get this in, this week, would be just awesome!!!

I have targeted (fixed or answered) just all the comments in Review Board.
Everything is already in the latest patch which is V12.
When I am looking on RB, I do not see anything new there, which wasn't targeted 
in V12.
If I miss something, please reopen the issue on RB or write it here.

Tonight I am flying to a conference in Europe and I am not going to be highly 
responsive till next Monday.
However, I am going to take a look from time to time.
I hope you have everything to take it in. If you see something small that 
disturbs you, feel free to fix it yourself, in order not to delay the commit.

Thank you all very very much!
Anastasia



> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-22 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430237#comment-15430237
 ] 

Anoop Sam John commented on HBASE-14921:


Only very few minor comments left for this over in RB I believe.  Can u make a 
latest patch ASAP pls..  We can target to get this in this week. :-)

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-18 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426934#comment-15426934
 ] 

Hadoop QA commented on HBASE-14921:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 7 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s 
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s 
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 3s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s 
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s 
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s 
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 0s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s 
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 10s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
47s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m 11s 
{color} | {color:red} The patch generated 5 new + 703 unchanged - 3 fixed = 708 
total (was 706) {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m 4s 
{color} | {color:red} The patch generated 5 new + 507 unchanged - 5 fixed = 512 
total (was 512) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
30m 28s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s 
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s 
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 49s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 95m 20s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 46s 
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
45s

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-18 Thread Edward Bortnikov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426647#comment-15426647
 ] 

Edward Bortnikov commented on HBASE-14921:
--

Agree with Anoop - the CellChunkMap implementation is targeted mainly for the 
off-heap path. It is tracked in HBase-16421, and is unrelated to the this jira 
and the candidate patch (smile). 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-18 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426640#comment-15426640
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

Published yet another patch with recent code review comments.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> HBASE-14921-V12-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-17 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424938#comment-15424938
 ] 

Anoop Sam John commented on HBASE-14921:


What we observed in our tests with G1GC is that the MSLAB pool is not behaving 
worse.  And in off heap write path, our idea is to have an MSLAB with off heap 
backing and here we would need pool any way.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-17 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424881#comment-15424881
 ] 

Lars Hofhansl commented on HBASE-14921:
---

MSLAB forces another copy of the data backing each Cell coming in, consuming 
memory bandwidth. With G1GC that does not appear to be necessary, since G1 
manages small'ish memory regions anyway.

I will repeat what I always say: Let's not try to be smarter than the garbage 
collector. Chances are we're not :)


> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-17 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424799#comment-15424799
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


As I said I won't block the commit here. You can just let us know about the 
YCSB param that you had tested. If @anoop.hbase and [~saint@gmail.com] are 
fine then no problem with me too. 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-17 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424793#comment-15424793
 ] 

Hadoop QA commented on HBASE-14921:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 7 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s 
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s 
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s 
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s 
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s 
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s 
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
42s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m 9s 
{color} | {color:red} The patch generated 1 new + 705 unchanged - 1 fixed = 706 
total (was 706) {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m 2s 
{color} | {color:red} The patch generated 5 new + 507 unchanged - 5 fixed = 512 
total (was 512) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 25s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 7s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s 
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s 
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 42s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 29s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 55s 
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
49s {color} |

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-17 Thread Edward Bortnikov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424540#comment-15424540
 ] 

Edward Bortnikov commented on HBASE-14921:
--

Hate to be pushy - but next week we might be short of resources to work on this 
patch. [~stack], [~anoop.hbase], [~ram_krish], mind making a (final) pass on RB 
and go for commit if all is clean? 

That would be a huge relief.  We'll strive to make the future jiras more 
manageable. 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-17 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424523#comment-15424523
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

Hey,

Thank for taking the look! 
The recent patch is V11. I don't know why QA didn't work for the last patch. 
Now I have added the same patch once again, hoping this time the QA will be 
triggered.

Can we please move all performance discussions to HBASE-16417?

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, HBASE-14921-V11-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-17 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424310#comment-15424310
 ] 

Anoop Sam John commented on HBASE-14921:


Ya for YCSB test, what is the load? per row size and total data size? How many 
client threads? What is the RS heap size?   G1GC? conf?

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-17 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424241#comment-15424241
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


I did a test with the latest patch - V10 version.

With 40G heap space and the below PE configs,
{code}
 ./hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=50 
--size=150 --columns=50 --valueSize=200 --writeToWAL=false --bloomFilter=NONE 
--inmemoryCompaction=false/true  randomWrite 50
{code}
 Am still not able to run with InMemoryCompaction enabled and we get a FullGC. 
With default memstore am able to run.
Good news is that in that 2 min that the sytem ran there was no negative flush 
size happening. So it may be we are good here.

The reason should be because, 
>From the code I think though flattening is enabled, we see if the number of 
>segments is > 3 and so we do compaction. I think it goes under that logic. 
>Speculative scan is default false.
Since compaction is happening I think we have duplicated segments till the new 
one is updated in the pipeline. This is what I was saying in the last test 
also. 
You said you were able to run with YCSB. With YCSB what were the number of cols 
and the data size?
As said I won't block the check in - we need to identify the reason for the 
Full GC and add some policy or do the all segment flush thing that we had done 
based on the policy. Lets discuss that in other JIRA.



> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-16 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422577#comment-15422577
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

I created HBASE-16421 to continue with the CellChunkMap there.

Pay attention, both JIRAs numbers end with 21! :) Is it a good sign?

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-16 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422571#comment-15422571
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

This would be amazing, if we could get this in ASAP and continue to next level 
:)
I have published the last V11 patch (I hope) it is ready to be committed. The 
patch includes small review fixes that I got so far.
It is also on the RB. 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> HBASE-14921-V11-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-16 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422361#comment-15422361
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


May be I can apply the patch and test with current trunk and see if the negtive 
size still persists. 
For defaultMemstore also with the PE tool config that we specified you may not 
be able to run with 32G and you may need 50G atleast I think. Anyway that is 
fine. We need not spend time there now I believe. LEts focus on getting this 
in. 
I can do the testing and report back by EOD tomorrow my time. Today quite busy 
with some other things.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-15 Thread Edward Bortnikov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420875#comment-15420875
 ] 

Edward Bortnikov commented on HBASE-14921:
--

Indeed let's defer this discussion to the new JIRA. Not sure we have a good 
definition of admissible overhead for speculative scan, but it's good to be 
talking around numbers. 

With this JIRA, it is our strong preference to commit ASAP (maybe this week?) 
unless there is some insurmountable problem. We'll be posting a new patch soon 
- please weigh in on the RB!

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-15 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420825#comment-15420825
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

I created HBASE-16417 to follow this JIRA and to deal with all the policy 
issues, meaning whether to flat/compact/merge the compaction pipeline scans and 
how to decide about that. Please move all the discussion about that to 
HBASE-16417. Thanks!

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-15 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420787#comment-15420787
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

Hi Guys!

I have published the patch that measures how long it takes to do the 
speculative scan. It appears we can not run PE and we try to evaluate using 
YCSB. On YCSB we see no performance impact due to the compaction-evaluation 
scan. The scan per store takes in average 280 milliseconds and happens in 
average every 3.5 minutes.  Can you please run this patch with your PE and tell 
us what you see?

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-14 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420309#comment-15420309
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

bq. Makes sense. So were you able to observe the problems that we highlighted 
in the previous discussions - We got two problems one is the invalid negative 
size for memstore and the 'Too many open files'?
Were you able to get this patch running with the PE tool and the configs that 
we had updated in the earlier discussions?

We are yet unable to use the PE tool. Even on DefaultMemStore it doesn't work 
properly. We are currently running with YCSB, stressing the writes. We didn't 
get the exceptions that you mentioned, neither "negative size" nor "too many 
open files".
However, we were able to see the problem of "too many open files" without 
getting the exception. From here, the quick fix of limiting the number of 
segments in the pipeline. This to be managed by a proper policy later. 

Later today I am going to publish the patch where we have instrumentation 
measuring the time it takes to do the speculative scan. Will you be able to 
give it a run on your PE tool? You can by the way take a look if you are still 
seeing the "negative size" exception. This one (if still exists) can later be 
investigated while dealing with the sizes issues.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-11 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418386#comment-15418386
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


bq.1. Size arithmetic. I'll be publishing a design proposal for this issue here 
and on HBASE-16229 soon, let's manage the rest of the discussion there. There 
seems to be an elegant solution, which I even have a partial code for. Can post 
a suggested patch immediately after HBASE-14921 is committed.
Yes. We are infact waiting for gettig in HBASE-16229 after this JIRA is done. 
Size arithmetic is important for us for the offheap accounting also.

bq.The current patch circumvents the "too many open files" with a simple 
heuristic that prevents too many segments from piling up. However, we realize 
that this issue might require more discussion, so let's open a new jira for it. 
Makes sense. So were you able to observe the problems that we highlighted in 
the previous discussions - We got two problems one is the invalid negative size 
for memstore and the 'Too many open files'?
Were you able to get this patch running with the PE tool and the configs that 
we had updated in the earlier discussions?

bq.And finally, the crown jewel - the CellChunkMap project. Its main value is 
providing platform for the Off-Heap Memory project. Here too, let us create a 
new jira, focusing on the following:
Yes. Once you can get the CellChunkMap code into it - we can help you with the 
new type of Cell to avoid chunk copying and the MSLAB chunk creation related 
code which we have maintained internally.

bq.The Big Cells issue.
+1 to address this. Need to see what best can be done here. May be discuss what 
we call as a BigCell. 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, 
> MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-11 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417149#comment-15417149
 ] 

Hadoop QA commented on HBASE-14921:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 7 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m 20s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s 
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s 
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s 
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s 
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s 
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 5s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
42s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m 8s 
{color} | {color:red} The patch generated 1 new + 705 unchanged - 1 fixed = 706 
total (was 706) {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m 3s 
{color} | {color:red} The patch generated 5 new + 507 unchanged - 5 fixed = 512 
total (was 512) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 28s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 3s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s 
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 40s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 94m 58s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 41s 
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
49s

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-11 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417138#comment-15417138
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

The current patch includes fixes to all accepted code review comments. Thank 
you [~ram_krish], [~anoop.hbase], and [~stack] for your comments!!

The goal of this patch is finalize the implementation reflecting the design 
published on this Jira. Our intent is to isolate non-critical issues to the 
following jira's, before the current code becomes unmanageable :) The current 
patch will go thru the normal review cycle, of course. In the future, we'll be 
shooting for much smaller jira's to converge faster.

Suggested roadmap of taking care about the related issues.
1. Size arithmetic. I'll be publishing a design proposal for this issue here 
and on HBASE-16229 soon, let's manage the rest of the discussion there.  There 
seems to be an elegant solution, which I even have a partial code for. Can post 
a suggested patch immediately after HBASE-14921 is committed.
2. In-memory flattening/compaction policy. The current patch circumvents the 
"too many open files" with a simple heuristic that prevents too many segments 
from piling up. However, we realize that this issue might require more 
discussion, so let's open a new jira for it.  We can deal with all performance 
issues and  configuration possibilities there.
3. And finally, the crown jewel - the CellChunkMap project. Its main value is 
providing platform for the Off-Heap Memory project. Here too, let us create a 
new jira, focusing on the following:
- The CellChunkMap implementation itself (already prototyped but not 
integrated yet)
- Related design issues (some refactoring of MemStoreChunkPool, MSLAB and 
HeapMSLAB).
- Flattening to CellChunkMap (integrating with new [~anoop.hbase] and 
[~ram_krish] code)
- The Big Cells issue.
We might need to manage this project as an umbrella jira, to de-clutter the 
space :)

Looking forward to your review comments.
Thank you!

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemstoreItrCountissue.patch, 
> NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-11 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416955#comment-15416955
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

Submitted another patch that fixes some compilation errors, no need to update 
the review board

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> HBASE-14921-V09-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemstoreItrCountissue.patch, 
> NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-11 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416941#comment-15416941
 ] 

Hadoop QA commented on HBASE-14921:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 7 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s 
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s 
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s 
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s 
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 28s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s 
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 12s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 30s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_101. 
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 30s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
47s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m 10s 
{color} | {color:red} The patch generated 1 new + 705 unchanged - 1 fixed = 706 
total (was 706) {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m 2s 
{color} | {color:red} The patch generated 5 new + 507 unchanged - 5 fixed = 512 
total (was 512) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 4s 
{color} | {color:red} The patch causes 20 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 0s 
{color} | {color:red} The patch causes 20 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 56s 
{color} | {color:red} The patch causes 20 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 53s 
{color} | {color:red} The patch causes 20 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 48s 
{color} | {color:red} The patch causes 20 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 46s 
{color} | {color:red} The patch causes 20 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 44s 
{color} | {color:red} The patch causes 20 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 44s 
{color} | {color:red} The patch causes 20 errors with Hadoop v2.6.3. {color} |

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-11 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416924#comment-15416924
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

All done according to your requests :) , will soon come with a summary about 
where we stand and what we have 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemstoreItrCountissue.patch, 
> NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-11 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416904#comment-15416904
 ] 

Anoop Sam John commented on HBASE-14921:


Pls update the RB link with latest patch

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemstoreItrCountissue.patch, 
> NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-10 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416542#comment-15416542
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


[~anastas], [~eshcar]
Any updates/feedback here?  

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemstoreItrCountissue.patch, 
> NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-02 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403568#comment-15403568
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

THANK YOU SO MUCH !!!

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemstoreItrCountissue.patch, 
> NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-02 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403504#comment-15403504
 ] 

Hadoop QA commented on HBASE-14921:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 26s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 59s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m 23s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} 
|
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 23s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 59s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_80. 
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 59s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.7.0_80. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 58s 
{color} | {color:red} Patch causes 30 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 53s 
{color} | {color:red} Patch causes 30 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 41s 
{color} | {color:red} Patch causes 30 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 27s 
{color} | {color:red} Patch causes 30 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 13s 
{color} | {color:red} Patch causes 30 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 0s 
{color} | {color:red} Patch causes 30 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 12m 48s 
{color} | {color:red} Patch causes 30 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 14m 34s 
{color} | {color:red} Patch causes 30 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 16m 26s 
{color} | {color:red} Patch causes 30 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 46s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-02 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403444#comment-15403444
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


[~anastas] - JFYI.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, MemstoreItrCountissue.patch, 
> NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-01 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403357#comment-15403357
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


[~eshcar]
With default memstore impl with MSLAB ON and Chunk pool we were able to run 
with 32G. But with MSLAB and chunk pool are OFF we need atleast 50G to run with 
this load and other GC tunings. Because it generates lot of garbage. 
With 32G see if you could run with CompactingMemstore with MSLAB ON and Chunk 
pool. 


> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-01 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403355#comment-15403355
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


Ok. I will paste that UT in some time now. Yesterday was without internet. So 
could not do it.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-08-01 Thread Eshcar Hillel (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402087#comment-15402087
 ] 

Eshcar Hillel commented on HBASE-14921:
---

Hi [~anoop.hbase], [~ram_krish], 
Did you use ssd machines for the PE?
So far when running the suggested setting on an hdd machine I get an 
insufficient memory error even when running the default memstore, so I suspect 
this is due to IO bottleneck (flushes are simply too slow).
Can you share the configuration of the machine you run on?

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-31 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401503#comment-15401503
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


[~anastas]
My UT was very simple. I selected the testCompaction1Bucket or any such test 
case. I just added some keyvalues but ensured that there are 50 cols in each 
row. 
I called flattening manually and then checked the count of cells in that 
flattened segment. I think you will get that issue pretty directly. I don't 
have that UT now as i reverted my changes. 
Let me know if you still need them I can paste them here. Thank you.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-31 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401099#comment-15401099
 ] 

Anoop Sam John commented on HBASE-14921:


1. Ours is a write only workload.  So BC size not that relevant any way.. Ya it 
was way lower than 0.38.  Less than 10% I believe
2. writeToWAL = false

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-31 Thread Eshcar Hillel (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401076#comment-15401076
 ] 

Eshcar Hillel commented on HBASE-14921:
---

Questions on PE tests:
1. you set hbase.regionserver.global.memstore.size to 0.42. Did you also set 
hfile.block.cache.size to 0.38? or even lower?
2. which setting did you use for WAL writeToWAL=true/false?

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-31 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401013#comment-15401013
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

Hey [~ram_krish], thank you for your comment explaining the problem with 
MemStoreCompactorIterator!
You said that you have the UT exposing the bug. I fixed the problem (at least I 
think I do), but in order to be sure I would very much like to run your UT.
Can you please share the UT with me anyhow? I am OK with even copy-pasting the 
method here.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-27 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395945#comment-15395945
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


bq.. What is the exact cluster configuration you are running on? Is it a single 
machine hosting both the master and 1 RS? What is the setting of the underlying 
HDFS? Again NN and 1 DN on the same machine?
Yes this is a single node system. 1NN and 1DN all running on same machine.
{code}
Xmx - 32G

  hbase.regionserver.global.memstore.size
  0.42


hbase.hregion.memstore.mslab.enabled
true
  


hbase.hregion.memstore.chunkpool.maxsize
1
  


hbase.hregion.memstore.chunkpool.initialsize
0.5
  
{code}
Do you need any other information?
bq.--inmemoryCompaction flag you are using in PE is not in master? Can you open 
a Jira to share the patch with this supplement so we don't need to re-implement 
this or even better commit this code to master.
I think this is already in master.
{code}
final String inMemoryCompaction = "--inmemoryCompaction=";
{code}
bq.How much time the experiment is expected to run (approximately)?
It should run for approx 20mins. Because we load 150G of data. 
{code}
./hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=50 
--size=150 --columns=50 --valueSize=200 --writeToWAL=true --bloomFilter=NONE 
--inmemoryCompaction=false  randomWrite 50
{code}
Thanks for the updates.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-27 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395836#comment-15395836
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

Indeed, no SQM, both in ppt and in the code

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-27 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395835#comment-15395835
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

The ppt is meant for those unfamiliar with this patch :)

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-27 Thread Eshcar Hillel (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395713#comment-15395713
 ] 

Eshcar Hillel commented on HBASE-14921:
---

We are trying to reproduce the PE results you reported [~anoop.hbase] and 
[~ram_krish], and we need your help with the following questions:

1. What is the exact cluster configuration you are running on? Is it a single 
machine hosting both the master and 1 RS? What is the setting of the underlying 
HDFS? Again NN and 1 DN on the same machine?

2. --inmemoryCompaction flag you are using in PE is not in master? Can you open 
a Jira to share the patch with this supplement so we don't need to re-implement 
this or even better commit this code to master.

3. How much time the experiment is expected to run (approximately)? 

One major drawback of the PE tool is that it does not allow to benchmark 
concurrent reads and writes.
Once we are able to test 100% writes and have results for all settings we 
should also compare the 50%-50% workload. But we can discuss this when we get 
there.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-27 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395408#comment-15395408
 ] 

Anoop Sam John commented on HBASE-14921:


I read it again. I think it is fine. 
recreateCellArrayMapSet is doing a Segment#getSegementScanner only. Good.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-27 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395371#comment-15395371
 ] 

Anoop Sam John commented on HBASE-14921:


The ppt says in case of flatten, we scan the cells in the segment with out 
using SQM.  I am +1 for this.  But not sure code wise it happens this way !  
Pls check once.. We just need a simple iterator over the CSLM for this flatten. 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-27 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395368#comment-15395368
 ] 

Anoop Sam John commented on HBASE-14921:


Thanks for the ppt.. It is a very clear explanation. (Even though I knew it 
after reading the code many a times by now :-) )

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-27 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395276#comment-15395276
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

[~stack], I attached a presentation explaining current patch flow. I know it 
doesn't clarify all that we have in this patch, but this is what I can quickly 
do for now. I will probably add class diagram and more explanations latter. 
Will be happy to answer specific question if any. Thank you for taking a look.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-26 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394006#comment-15394006
 ] 

stack commented on HBASE-14921:
---

Let me do some more review of the posted patch. It is a bit messy.
St.Ack

On Tue, Jul 26, 2016 at 9:04 AM, Anoop Sam John (JIRA) 



> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-26 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394004#comment-15394004
 ] 

Anoop Sam John commented on HBASE-14921:


+1

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-26 Thread Edward Bortnikov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393993#comment-15393993
 ] 

Edward Bortnikov commented on HBASE-14921:
--

Thanks, all, for the last round ... Seems like we are converging. 

So, we seem to agree on the following:
1. At the end of the day, we want compaction (data de-duplication) where it 
helps, and don't want it where it doesn't. That might use a flag or some smart 
policy. Getting there might take 1-2 more (smaller) patches. 
2. We want to land the current patch because it's getting big. The mandatory 
condition for passing it is bug-freedom (via improved coverage), including the 
more stressful PE tests. Hopeful to finish this week or early next week. 
3. We want to release all the dependencies for [~anoop.hbase] and [~ram_krish] 
on their way to off-heap memory implementation. We'll switch to that as soon as 
the current patch is landed. We have a common goal (smile). 

Deal? 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-26 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393386#comment-15393386
 ] 

Anoop Sam John commented on HBASE-14921:


Am fine with committing the other jira first.. I can rework on this jira after 
that.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-25 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393204#comment-15393204
 ] 

stack commented on HBASE-14921:
---

[~anoop.hbase] What you think of the [~ebortnik] proposal.

I just want to iterate that it is priority that the 80% case, the case where we 
do not have much by way of Cell overlaps/duplicates, cannot suffer when we add 
in this feature. It is fine if there is a temporary performance regression. We 
can live with that in master branch, but it cannot go unaddressed. Just saying.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-25 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393193#comment-15393193
 ] 

Anoop Sam John commented on HBASE-14921:


Ya [~anastas] the 2 points I raised are those..  I wanted to make sure those 
are highlighted early rather than saying it after..  And I dont mean they 
should be addressed in a single jira. Ya this is already big patch and no need 
to do more work again.. That is the common practice we follow.. When a comment 
needs more work, the developer can suggest doing it later as part of another 
jira and reviewers mostly agree. I am fine for that..  Sorry if I was not 
saying it explicitly.  It is a practice for us so I missed. Sorry.
It is your wish..  In this form also am ok to get that in.. We can always make 
things better after.
The cost of scan may be more when we have CellChunkMap.. In this flattened 
form, we get rid of Cell objects and again this Scan will make these objects 
over us.  With CellArrayMap the overhead might be from SQM and StoreScanner 
heap.  With CellChunkMap it is more.. That is why I raised it early.  Ya let us 
do that later also. Its ok.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-25 Thread Edward Bortnikov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392671#comment-15392671
 ] 

Edward Bortnikov commented on HBASE-14921:
--

Let me re-iterate we are respectful of everyone's contribution, and are trying 
to do the right thing, as much by-consensus as possible. 

Here's a suggestion. For the sake of the current patch, let's decouple the 
in-memory flush configuration from compaction configuration. The latter is a 
special case of the former. With compaction protected by a explicit flag, we no 
more need the speculative scan to predict its worthiness. The code becomes 
simple. In the future, we can discuss smart policies to help us eliminate this 
flag. 

[~anastas] and [~anoop.hbase], can we agree on this as base for further 
discussion? 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-25 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392669#comment-15392669
 ] 

stack commented on HBASE-14921:
---

Ok. Left some notes on the review but any chance of a high-level overview on 
what the latest patch iteration delivers? Does it jibe w/ the attached design? 
If so, thats grand. I am asking because I presume it has morphed since my old 
reviews. Would it help [~anastas] if i ran another version of [~ram_krish]'s 
loading test? A YCSB say?

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-25 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392580#comment-15392580
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

All code review comments from review board were addressed. All replies are in 
the review board. As I have said above the main concerns are:

1. Correctness exceptions -- this is under investigation and is going to be 
fixed
2. The concern about how much the compaction-estimation costs -- we are going 
to run the PE tool ourselves
3. The problem with small flushes to disk due to lack of compaction -- no doubt 
this can be arranged, but probably not under this JIRA

This is my summary

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-25 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392571#comment-15392571
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

First, I agree that the sizing issue is ugly and need to be improved. Which is 
partially done in this patch and I planned to improve it further. 

However, I think it is unnecessary and not urgent, to open another JIRA for 
this fix. This is not an issue of rebase only, because we take the code in two 
different directions. We could live with the code as is (or at least could see 
the final outcome of 14921) and later we could agree how to arrange the sizes 
(if what we have is not good enough)... 

Your two concerns are very clear.
1. The flattening without compaction is causing many small segments in 
pipeline, and they are not flushed all together.
2. The issue of compaction prediction cost.

Please correct me if I am wrong.

We understand those concerns. There is no argument that your first concern will 
be fixed. For your second concern we are going to benchmark it ourselves with 
PE.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-25 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392525#comment-15392525
 ] 

stack commented on HBASE-14921:
---

Oh, and with [~ebortnik], what is shortest path to commit of this patch. 
Reviewing the RB comments, it seems like there are outstanding issues still. 
Can these be addressed or if not fundamental, removed and done in a separate 
issue?

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-25 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392497#comment-15392497
 ] 

stack commented on HBASE-14921:
---

Just did a reread of this whole issue.

First, what is going on in here is wonderful. True, the issue is going on too 
long and starting to run away from us but it is a shining example of the best 
of collaboration; informed, data-based compares, accommodating, smart, 
respectful back-and-forth, detailed reviews, actual testing (and fixes) of 
posted patches, etc. You can't beat it.

Second, all involved agree on the merit of these developments, their promise, 
and are trying to help land the patch. There is consensus that we should commit 
and then-address-outstanding-issues afterward but as I read it there seems to 
be a reluctance to take on the patch while it demonstrably slows down the 
default case -- i.e. when no duplicates -- and there is concern that we may not 
be able to recover the lost perf with the current approach. We could of course 
turn this feature 'off', by default, but most of us don't want to do that for 
reasons stated above (another is that [~ram_krish] and [~anoop.hbase] want to 
base some of their offheaping of write path on the work done here). Can I help 
in here? I can run some perf compares like [~ram_krish]'s?

[~anoop.hbase] Mind repeating what your two concerns were here just so the 
discussion is contained (this issue is long now).



> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-25 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392425#comment-15392425
 ] 

stack commented on HBASE-14921:
---

[~anastas] I like this reasoning. You've done this a few times in this issue. 
Please do not have these comments lost in the general back and forth. Can you 
hoist your thoughts into release notes/documentation for this feature?

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-25 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392298#comment-15392298
 ] 

Anoop Sam John commented on HBASE-14921:


HBASE-16229 trying to just make the size accounting in a better shape..  The 
accounting happens within each class like Segment/ CompactingMemstore.  Not 
like using setter some one else set a size and then some places we add some 
overhead and some other place minus it.. It was really confusing.
Pls see that change.. I said above that I can help with the rebase which might 
be needed because of this change.  Sorry for the rebase effort caused by other 
issue fixes.
See we all wanted to make sure that this feature is well accepted.  We feel 
that this has relevance not just in scenario where there are many 
duplicates/deletes.. But in a normal case also..  Or else we would not have 
given this much of our effort. 

I had raised 2 points of concerns on the general approach.  Am not saying that 
those has to be handled as part of this jira. We can get this in and then work 
on those also.. But I wanted to highlight those. I raised this at initial stage 
also.. But then there were counters.  And now those counter args can not stand 
at all time.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-25 Thread Edward Bortnikov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392213#comment-15392213
 ] 

Edward Bortnikov commented on HBASE-14921:
--

Suggest we install some order in this discussion, there are really multiple 
issues on the table.

TL;DR: Let's get this patch in shape and check it in without over-optimizing; 
it's already quite big. 

1. Bugs in the current PR. Thanks for reporting. Those must be fixed, period. 
We are working on reproducing and fixing.
2. Decoupling In-Memory Flush (with Flattening) from Compaction - either 
algorithmic or via configuration. IMHO, this is a matter of optimization, 
either approach has its pros and contras. For example, if flattening and 
compaction were always coupled, the too-many-open-files problem would not have 
emerged. In general, we're in favor of having a smart system with as few 
parameter knobs as possible, capable of figuring out the compaction benefits at 
a low cost. But again, this is a matter of policy. We suggest to defer it 
beyond the current commit. 
3. Concurrent development. Currently, there are at least two JIRA's 
(HBASE-16003 and HBASE-16229) that try to concurrently handle the same issues 
as this JIRA, which creates a lot of friction in the code. The prior consensus 
was that HBASE-14921 would be the umbrella for all memory optimizations, 
including the ultimate flattening (CellChunkMap). Failing to stick to this 
discipline slows us down a lot. [~anoop.hbase] and [~ram_krish], if you feel 
you've reached a more advanced stage of flat memory implementation for the 
ultimate off-heaping, and prefer to lead that charge - this is perfectly fine 
with us. But let us merge the 14921 patch first (it's already heavily 
invested), and start optimizing on top of it. 

Cheers,
Ed

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-22 Thread Edward Bortnikov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15389678#comment-15389678
 ] 

Edward Bortnikov commented on HBASE-14921:
--

Agreed. We all need more benchmarking. Let's proceed with the CR independently. 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-22 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15389649#comment-15389649
 ] 

Anoop Sam John commented on HBASE-14921:


Thanks Ram
[~anastas] Suggest you people also test this with large load as we are doing.  
A min of 50 client threads doing the writes and let it run for mins.  We are 
trying to write 150 GB of data so that it takes more than 10 mins always.  We 
can know the impact of flushes/compactions also.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-21 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388051#comment-15388051
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


bq. to move on to CellChunkMap (the real thing).
Ya fine to move to this. We have made necessary changes to avoid copy to get 
the chunk id. This change and the chunkId creation  can be committed as 
seperate patches. You can create the CellChunkMap based on that.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-21 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388017#comment-15388017
 ] 

Anoop Sam John commented on HBASE-14921:


When the compaction of 3 or more segments can not really remove 
duplicate/deleted cells, why you think we need to compact it?  Just to make the 
tail of the pipeline grow bigger?  
Ya rather than compact a segment with another in pipeline immediately after the 
second one comes in might not be needed.  The compaction can kick start at some 
interval (may be based on #segments in pipeline)

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-21 Thread Edward Bortnikov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387999#comment-15387999
 ] 

Edward Bortnikov commented on HBASE-14921:
--

Appreciate if we could get the new patch reviewed really quick, to move on to 
CellChunkMap (the real thing). 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-21 Thread Edward Bortnikov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387993#comment-15387993
 ] 

Edward Bortnikov commented on HBASE-14921:
--

[~ram_krish], thanks for all the feedback so far. My 2 cents:
1. On-disk fragmentation - agree we might need to merge upon flush (or maybe a 
bit before) if the pipeline becomes too big. We don't want to create too many 
files, obviously. 
2. Flattening without compaction - let's give this feature a second thought, 
I'm not all that sure it's worth separating between the two. At least that 
would be confusing for the administrator to deal with one more flag. Could you 
please bring some hard benchmark numbers that show that the overhead of 
compaction doesn't make sense? 

Maybe we can solve the fragmentation and compaction problems together? Like, a 
new immutable segment is automatically flattened but not merged with other 
segments. Once there are enough segments in the pipeline (3?), they are merged, 
without any speculative prediction check. Upon flush, the oldest segment goes 
to disk - same as now. No flags, no configuration parameters (other than just 
using the new Store type). 

Opinions? [~stack], please chime in :)

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-21 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387517#comment-15387517
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

The recent patch is in RB now

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-21 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387509#comment-15387509
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


bq.We can do it when we have like e.g. 10 segments in the pipeline. If we are 
going to simply flush it all to disk we are going to create many small files 
and their compaction is going to run on disk then...
[~anastas] - The idea was to flush all the segments into one file. So the 
snapshot will contain 10 segments and all the 10 will be flushed at one shot. 
Not that we will create 10 files - 1 out of each segment.
I am just reviewing this code. If the patch is not updated in RB can you update 
it? 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-21 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387470#comment-15387470
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

Thank you [~anoop.hbase] for your very reasonable comments!

bq. But when the use case is like some thing of time series data, where we 
really dont expect duplicates/updates, it might be better to turn off 
compaction and do only flatten.

Do you suggest to make an externally editable flag for turning compaction on 
and off? So what should be the default value for this flag? Didn’t we wanted 
sysadmins to work less with all those flags and settings (that we already 
have)? We can make this compaction-pre-check scan every second (Xth) flush to 
pipeline if it appears to decrease the performance.

bq. Again flatten to CellChunkMap would be ideal as that will release and 
reduce heap memory footprint for this memstore considerably. CellArrayMap, yes 
it reduces but not much. 

CellChunkMap is valuable because it can be taken off-heap, but CellChunkMap 
doesn’t significantly reduces the memory usage compared to CellArrayMap. All 
that you save memory-wise in CellChunkMap is that Cell object is now “embedded" 
as part of the array, and so you do not need the reference and the object 
overhead. So the difference between CellArrayMap and CellChunkMap is in 24 
bytes per Cell.

bq. In your usecase, the max adv you get because of the compaction as many 
cells will get removed.

I do not agree. In our experiments we (on purpose) use uniform distribution 
with small data size and we have little duplicates. We still see that the 
compaction has little impact on the performance.

bq. My another concern is regarding the fact that in this memstore only the 
tail of the pipeline getting flushed to disk when a flush request comes. In 1st 
version it was like always the compaction happens. So all chances that the tail 
of pipeline is much bigger sized and so that much data gets flushed. Now when 
compaction is not at all happening and we do have many small sized segments in 
pipeline, it would have been better to flush all the segments to disk that 
making small sized flushes. I raised this concern at first step also. But then 
the counter was that the compaction happens always but now it is not the case.

I remember this concern of yours from the code review. This is a valid concern 
and we are thinking about it. Apparently, this is one more reason to do 
compactions (at least for merge) once in a while. We can do it when we have 
like e.g. 10 segments in the pipeline. If we are going to simply flush it all 
to disk we are going to create many small files and their compaction is going 
to run on disk then...

bq. JFYI.. There is a periodic memstore flush checking. If we accumulate more 
than 30 million edits in memstore, we will flush

We know there is a flush to disk once about every hour. The main reason for 
that is WAL, right? Otherwise, why would we care how many cells are in memory? 
Actually, may be in this we do not want to flush absolutely everything to disk 
and to flush just the oldest part so the WAL can truncate a bit is enough?

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-21 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387315#comment-15387315
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

I think I set it all correct now. Please take a look on the last patch.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-21 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387309#comment-15387309
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

I have re-based everything for now. Reapplied the change, please take a look on 
the last patch.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-21 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387231#comment-15387231
 ] 

Hadoop QA commented on HBASE-14921:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 1s 
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 1s 
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 7 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 24s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 24s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 12s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
29m 31s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 32s 
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 59s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 127m 40s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 29s 
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
46s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 194m 35s {color} 
| {color:black}

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-21 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387229#comment-15387229
 ] 

Anoop Sam John commented on HBASE-14921:


I see your latest rebased patch.  Pls see HBASE-16229.  Doing some changes wrt 
size and heap size accounting.  I can help you with rebase once we get that in.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-20 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387214#comment-15387214
 ] 

Anoop Sam John commented on HBASE-14921:


[~anastas]  So you got the comment abt the heap sizing for the CellArrayEntry ? 
 I mean to say here just consider the extra heap space need for adding an item 
to this CellArray structure.  Do not consider the heap need for that object as 
such. That can be done within that object's heap size calc.
Finally when we have to find whole heap size of a given CellArray structure we 
can 
 + 
for(Each entry in CellArray)
   CellArray Entry overhead + Cell object heap

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-20 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387050#comment-15387050
 ] 

Anoop Sam John commented on HBASE-14921:


I got your argument abt dynamic decision making for compaction vs flatten only. 
 Worry was how costly it will be to do another round of scan.  It involve SQM 
and a Heap with many compares. It is not so cheap.  As of now you are not 
adding the CellChunkMap based flattening.  Things will be much worse, when we 
have that.  We get rid of Cell objects as such in that flattened form. But then 
this scan need Cells to be created again. Means lots of garbage.
May be in a use case where duplicates are possible, chances are there that 
there are not enough #duplicate records so that we get some real benefit out of 
compaction. So then flatten would be enough.  So an extra scan may make sense 
there.
But when the use case is like some thing of time series data, where we really 
dont expect duplicates/updates, it might be better to turn off compaction and 
do only flatten.
Again flatten to CellChunkMap would be ideal as that will release and reduce 
heap memory footprint for this memstore considerably. CellArrayMap, yes it 
reduces but not much.  In your usecase, the max adv you get because of the 
compaction as many cells will get removed.

My another concern is regarding the fact that in this memstore only the tail of 
the pipeline getting flushed to disk when a flush request comes.  In 1st 
version it was like always the compaction happens. So all chances that the tail 
of pipeline is much bigger sized and so that much data gets flushed.  Now when 
compaction is not at all happening and we do have many small sized segments in 
pipeline, it would have been better to flush all the segments to disk that 
making small sized flushes. I raised this concern at first step also. But then 
the counter was that the compaction happens always but now it is not the case.

Ya Ram will come up with al perf analysis.

bq.We are now holding more in the memory and thus having more possibility to 
let a cell "die" in memory. 
JFYI..  There is a periodic memstore flush checking. If we accumulate more than 
30 million edits in memstore, we will flush
{code}
if (this.maxFlushedSeqId > 0
  && (this.maxFlushedSeqId + this.flushPerChanges < 
this.mvcc.getReadPoint())) {
  whyFlush.append("more than max edits, " + this.flushPerChanges + ", since 
last flush");
  return true;
}
{code}
This flushPerChanges is configurable btw.
The second check here is time based. If we have not flushed memstore for quite 
some time, we will make a flush. This time def to 1 hr.
Just saying for your consideration.


> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-20 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387034#comment-15387034
 ] 

Hadoop QA commented on HBASE-14921:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 2s 
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 2s 
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 7 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 3s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 49s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
29m 17s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 44s 
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 44s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 118m 2s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 13s 
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
44s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 188m 34s {color} 
| {color:black}

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-20 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385794#comment-15385794
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

Hi Guys,

I really wanted to publish the patch today, but HBASE-16194 is causing a big 
problem to rebase. I should take it carefully as I have changed the way 
heapSize is measured and so did HBASE-16194.
I will try tonight once again to resolve all the conflicts.

Sorry,
Anastasia

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-20 Thread Edward Bortnikov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385748#comment-15385748
 ] 

Edward Bortnikov commented on HBASE-14921:
--

[~ramkrishna], I'd suggest to review the upcoming patch as-is, and discuss the 
enablement of compaction later on. I see your point about the cost vs value of 
compaction. But maybe we can build a cheaper predictor of overlap between sets, 
a-la Bloom filter. That might affect the worthiness of optimizing this point 
... First things first. 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-20 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385690#comment-15385690
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

bq. Am not very sure on this. You mean most of the cases will have duplicates? 
There are use cases we have seen where there is not much duplicates and each 
row is unique. Say in a time based row key impl. 

No, I do not mean most of the cases will have duplicates. I am sure there are 
cases where are no duplicates at all. I mean for example the cases where there 
are period of times with more duplicates and periods with less. When it is not 
clearly known ahead of time. Usually, the usecases with no duplicates at all 
and with lots of duplicates are rare. I just think that 10-15% of duplicates 
should worth compaction...

bq. Yes minor compaction on the disk is a bottleneck because of IO. But in the 
case where you have very less duplicates you are doing that operation twice, 
once in memory and once in disk. This patch is not going to say that since 
memory compaction has been done avoid disk minor compaction. Coming to deletes, 
there are use cases where the deletes are there but very rare. So even when the 
in memory compaction is going to remove such deletes ( if it is encountered) 
that is going to create a flush which is going to be slighly lesser in size but 
again the minor compaction will be performed on this file also.

I agree with you that without duplicates in-memory compaction is unnecessary. I 
just wanted to show that in case of little duplicates you gain more then space 
in memory.

The results are very interesting. On which version exactly was the estimation 
done? On my previous patch? Let me give you a new and updated pach today.
Thank you, Ramkrishna!

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-20 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385649#comment-15385649
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


[~anastas]
Thanks for your consolidated feedback and thoughts. I really appreciate your 
inputs and I totally agree with your above points but for
bq. Only in very rare cases, it is known ahead of time that keys never repeat 
or being deleted.
Am not very sure on this. You mean most of the cases will have duplicates? 
There are use cases we have seen where there is not much duplicates and each 
row is unique. Say in a time based row key impl. 
bq.Whether there are duplicates or not, you are going to do the same minor 
compaction anyway on the disk  just to reduce the number of files.
YEs minor compaction on the disk is a bottleneck because of IO. But in the case 
where you have very less duplicates you are doing that operation twice, once in 
memory and once in disk. This patch is not going to say that since memory 
compaction has been done avoid disk minor compaction.
Coming to deletes, there are use cases where the deletes are there but very 
rare. So even when the in memory compaction is going to remove such deletes ( 
if it is encountered) that is going to create a flush which is going to be 
slighly lesser in size but again the minor compaction will be performed on this 
file also. 
bq.At least you should do a homework and present me the clear cut performance 
evidences that the in-memory-compaction of flattened segments is not effective 
in the average case ("default case" as you call it).
The onus is on me for doing this and coming up with results. 
For now we have done the following things
-> After the first version of compacting memstore went in we started testing it 
and found some issues with which we thought pipeline creation was costly. LAter 
the bugs that were solved helped in overcoming this part and we got a better 
result.
-> With only flattening 'ON' by default and with offheap memstore we were able 
to clearly see a better G1GC mixed GC avg time. It got reduced from 0.6 sec to 
0.2 secs. So we are 100% sure flattening is needed. Remember in this impl the 
entire pipeline segments are flushed.
-> For your point of seeing whether default case with inmemory compaction and 
flattening if it is impacting or not, I can get you the numbers. If it is not 
impacting the perf then we can definitely go with your design and no problem 
with that. 
[~saint@gmail.com] and [~anoop.hbase] can comment more on the use case part 
and if anything is being missed here. 



> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-20 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385639#comment-15385639
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

Thank you [~anoop.hbase] and [~ram_krish]! You understand this project so well, 
made a thoroughly code review, not to say how much do I appreciate your deep 
HBase knowledge and experience!
I am calling to [~stack] to join our interesting discussions!

I understand your point. You would prefer to flatten the segments without 
compaction. This is because you believe that the compaction should in average 
eliminate little cells and even a scan doing the compaction is costly.
Let me disagree with this point of view and let me explain myself. Here are my 
points:

1. Whether compaction is going to eliminate cells or not is unknown till the 
run-time. We don't want to add (yet another) user configurable flag: whether to 
use compaction or not. Only in very rare cases, it is known ahead of time that 
keys never repeat or being deleted.
2. For the deletion case, if a key K was inserted and deleted, then we (at 
least) have 2 versions of K were one can be eliminated. So it is not the 
compaction is useless for deletions.
3. The performance degradation due to "pre-compaction" scan is yet to be 
estimated.
4. Whether there are duplicates or not, you are going to do the same minor 
compaction anyway on the disk (!) just to reduce the number of files. And there 
it is going to cost you much more, due to write amplifications, HDFS file 
replications, networking, bounded number of threads for compaction, I/O 
multiplication, etc. 

So we can think about some compaction application policies, apply it once in a 
while, etc. 
But just to disregard the great possibility to compact in-memory... I don't 
think it is wise. 
At least you should do a homework and present me the clear cut performance 
evidences that the in-memory-compaction of flattened segments is not effective 
in the average case ("default case" as you call it).

Please note that in-memory-compaction is actually more than saving some RAM 
space, especially coming together with flattening.
We are now holding more in the memory and thus having more possibility to let a 
cell "die" in memory. 
As I explained in Point 4, this saves much more resources then just space in 
RAM.
Please convince me where I am wrong :)

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-19 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385330#comment-15385330
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


bq.Can we do anything better? It looks like we will never able to deal with 
deletion, because this compaction is always minor.
Yes. I think so. So better to leave it as is. That is why we were saying that 
if for a default case there are not many version available, doing a compaction 
may not be really needed. Instead if we can flatten the segments and directly 
flush all the segments in the pipeline will be good enough and to see we are 
reducing the heap overhead because of flattening. 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-19 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384110#comment-15384110
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

First, thanks everybody for the code review comments! And special thanks to 
[~anoop.hbase] for the thoroughly review!

One question to [~ram_krish], I am looking on your comment:
bq. I think it is already there. But this scanner will not deal with deletes, 
only if there are duplicate versions will be removed.
related to the following line in the MemStoreCompactorIterator:
{code}
   /**
   * Creates the scanner for compacting the pipeline.
   *
   * @return the scanner
   */
  private StoreScanner createScanner(Store store, KeyValueScanner scanner)   
throws IOException {
Scan scan = new Scan();
scan.setMaxVersions();  //Get all available versions
StoreScanner internalScanner =
new StoreScanner(store, store.getScanInfo(), scan, 
Collections.singletonList(scanner),
ScanType.COMPACT_RETAIN_DELETES, store.getSmallestReadPoint(),  
//< this is the line
HConstants.OLDEST_TIMESTAMP);
return internalScanner;
  }
{code}

Can we do anything better? It looks like we will never able to deal with 
deletion, because this compaction is always minor. Or am I wrong?


> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-19 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383748#comment-15383748
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

What is the default case, where we are sure we don't need to remove any 
duplicates?
Hereby, I add a summary of how the flattening is using scans.

When the size of active segment is above some threshold in CompactingMemStore, 
the active segment is pushed to pipeline (MutableSegment wrapped as 
ImmutableSegment). After that a single dedicated thread is doing the following:
1. Scan *all* segments in the pipeline (with ScanQueryMatcher) in order to 
understand whether compaction is needed. This is for now the only way to 
understand whether we have duplicates or not.
2. Decide whether to flatten or to compact
3. If to flatten, then scan the not-flat segment only (without 
ScanQueryMatcher) in order to flatten.

Can we have the real numbers showing what is the performance difference with 
and without the scan (in stage 1)? May be you ([~anoop.hbase], [~ram_krish]) 
can run this experiment on your big set up (while we have a simple 
configuration)?

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-19 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383727#comment-15383727
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

Got it. Removed the double counting.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-18 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383595#comment-15383595
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


bq.but I suggest first to proof this scan is causing the performance 
degradation. Any thoughts about that?
The point is just for the default case. In default cases we are sure that we 
don't need to remove any duplicates. In such a doing a compaction is not 
needed. Yes for flattening we need to do a scan but that should be like an 
iterator mode not with the ScanQueryMatcher. 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-18 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383594#comment-15383594
 ] 

ramkrishna.s.vasudevan commented on HBASE-14921:


I now got to which comment are you referring to wrt CELL_ARRAY_ENTRY. I checked 
the RB.
bq.CELL_ARRAY_MAP_ENTRY = align(OBJECT + 2*REFERENCE + 2*Bytes.SIZEOF_INT);
When you see the CELL_ARRAY_MAP_ENTRY it should only have the reference to the 
cell and one object right?  The 2 ints and the byte[] will be accounted in the 
KeyValue.heapSize().  Now if you add the ref and int here then the 
KeyValue.heapSize will again add it. So the comment in RB is valid I feel.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

2016-07-18 Thread Anastasia Braginsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382953#comment-15382953
 ] 

Anastasia Braginsky commented on HBASE-14921:
-

Also, got a comment saying that scan (being done for deciding whether to 
compact or to flatten) is too costly. However I empirical results we truly do 
not see any performance issues due to this scan. We probably can add some 
sophisticated logic to predict the duplicate ratio, but I suggest first to 
proof this scan is causing the performance degradation. Any thoughts about that?

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

1 2 >

1 - 100 of 192 matches

Mail list logo