[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15990440#comment-15990440
 ] 

Hudson commented on HBASE-16438:


ABORTED: Integrated in Jenkins build HBase-HBASE-14614 #190 (See 
[https://builds.apache.org/job/HBase-HBASE-14614/190/])
HBASE-16438 Create a cell type so that chunk id is embedded in it (Ram) 
(ramkrishna: rev c2c2178b2eebe4439eadec6b37fae2566944c16b)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java
* (add) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemstoreLABWithoutPool.java
* (add) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ByteBufferChunkCell.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ExtendedCell.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCellFlatSet.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFailedAppendAndSync.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALLockup.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLABImpl.java
* (add) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChunkCreator.java
* (add) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NoTagByteBufferChunkCell.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerFromBucketCache.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALMonotonicallyIncreasingSeqId.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingToCellArrayMapMemStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingMemStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRecoveredEdits.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHMobStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionArchiveConcurrentClose.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionIncrement.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Chunk.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionArchiveIOException.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OffheapChunk.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OnheapChunk.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
* (delete) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreChunkPool.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionPolicy.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLAB.java
Revert "HBASE-16438 Create a cell type so that chunk id is embedded in 
(ramkrishna: rev ecdfb82326035ad8221940919bbeb3fe16ec2658)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* (delete) 

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976548#comment-15976548
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

[~ram_krish], thank you very much!

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974943#comment-15974943
 ] 

Hudson commented on HBASE-16438:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2887 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2887/])
HBASE-16438 Create a cell type so that chunk id is embedded in it (Ram) 
(ramkrishna: rev 972e8c8c296d38507077b98c8fc2a33eda9fce66)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionIncrement.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OffheapChunk.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingToCellArrayMapMemStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHMobStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (add) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ByteBufferChunkCell.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestFSHLog.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALLockup.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLAB.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCellFlatSet.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingMemStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALMonotonicallyIncreasingSeqId.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRecoveredEdits.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionArchiveConcurrentClose.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OnheapChunk.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLABImpl.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ExtendedCell.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerFromBucketCache.java
* (add) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NoTagByteBufferChunkCell.java
* (add) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemstoreLABWithoutPool.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionArchiveIOException.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/AbstractTestFSWAL.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionPolicy.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFailedAppendAndSync.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Chunk.java
* (add) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChunkCreator.java
* (delete) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreChunkPool.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java


> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> 

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974377#comment-15974377
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


All are flaky tests that failed. Each of them has passed in my local run. So I 
will commit this once again.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974361#comment-15974361
 ] 

Hadoop QA commented on HBASE-16438:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 31 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 39s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
7s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 7s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 43s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
54m 48s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 59s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 193m 43s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 
45s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 284m 4s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.util.TestHBaseFsckTwoRS |
|   | hadoop.hbase.snapshot.TestExportSnapshot |
|   | hadoop.hbase.client.TestAsyncBalancerAdminApi |
|   | hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| Timed out junit tests | 
org.apache.hadoop.hbase.replication.regionserver.TestWALEntryStream |
|   | org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
|   | org.apache.hadoop.hbase.mapreduce.TestImportExport |
|   | org.apache.hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat |
|   | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
|   | org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery |
|   | org.apache.hadoop.hbase.util.TestHBaseFsckTwoRS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12863945/HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch
 |
| JIRA Issue | HBASE-16438 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974040#comment-15974040
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


>From the small tests 
I got these failures
{code}
Tests in error:
  TestBackupHFileCleaner.setUpBeforeClass:66 » IO Shutting down
org.apache.hadoop.hbase.master.balancer.TestRegionLocationFinder.org.apache.hadoop.hbase.master.balancer.TestRegionLocationFinder
  Run 1: TestRegionLocationFinder.setUpBeforeClass:58 » IO Shutting down
  Run 2: TestRegionLocationFinder.tearDownAfterClass:77 NullPointer

  TestLockManager.setupCluster:77 » IO Shutting down
  TestLockProcedure.setupCluster:103 » IO Shutting down
  TestProcedureManager.setupBeforeClass:54 » IO Shutting down
  TestHRegionFileSystem.testBlockStoragePolicy:77 » IO Shutting down
{code}
which seems to be unrelated.
>From the medium tests I got
{code}
Failed tests:
  TestRegionLoad.testRegionLoad:103->compareRegionLoads:110 No of regionLoads 
from clusterStatus and regionloads from RS doesn't match expected:<8> but 
was:<12>
{code}
>From large tests I got no failures.

The TestReversibleScanners and TestFSHLog that were failing have been fixed in 
the latest patch. It was nothing but it needed ChunkCreator to be initialized.
bq.Looks like the above assignment can be moved after currentInstance is 
created.
Nope. We need a new type of Chunkcreator so that it is able to retire the 
chunks immediately. So we nullify the current Instance and recreate the 
chunkCreator instance and again assign the original one back to the INSTANCE 
variable. If needed I can rename from 'currentInstance' to 'oldInstance'.


> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973144#comment-15973144
 ] 

Ted Yu commented on HBASE-16438:


Mind highlighting how you fixed the tests ?

Thanks

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972122#comment-15972122
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


Am not sure how to check these reports. So after I submitted the addendum, the 
precommit QA had run and there
https://builds.apache.org/job/PreCommit-HBASE-Build/6461/testReport/org.apache.hadoop.hbase.regionserver/TestReversibleScanners/
I see all these tests passed. How is that? That is why with the precommit QA 
report I got confused and went ahead with the commit.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971862#comment-15971862
 ] 

Hudson commented on HBASE-16438:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2877 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2877/])
Revert "HBASE-16438 Create a cell type so that chunk id is embedded in 
(ramkrishna: rev ecdfb82326035ad8221940919bbeb3fe16ec2658)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCellFlatSet.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionIncrement.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLABImpl.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OnheapChunk.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
* (delete) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChunkCreator.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFailedAppendAndSync.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionArchiveIOException.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java
* (delete) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemstoreLABWithoutPool.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Chunk.java
* (add) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreChunkPool.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRecoveredEdits.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALLockup.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHMobStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionArchiveConcurrentClose.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OffheapChunk.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerFromBucketCache.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingToCellArrayMapMemStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionPolicy.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALMonotonicallyIncreasingSeqId.java
* (delete) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NoTagByteBufferChunkCell.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ExtendedCell.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingMemStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLAB.java
* (delete) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ByteBufferChunkCell.java


> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> 

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971457#comment-15971457
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


I will revert the commit. Not sure why the precommit QA does not point to this 
failure directly and shows other tests as failures. Ya I think i may have to 
run tests individually. I need some time to do it. Will do it over this week.
Thanks [~chia7712].

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971382#comment-15971382
 ] 

Ted Yu commented on HBASE-16438:


{code}
250 } finally {
251   ChunkCreator.INSTANCE = currentInstance;
{code}
Is the finally block necessary ?
Looks like the above assignment can be moved after currentInstance is created.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Chia-Ping Tsai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971109#comment-15971109
 ] 

Chia-Ping Tsai commented on HBASE-16438:


Some failed tests are shown below. We should make sure that all unit tests pass 
if we run them individually. (It may be a boring job...)
{noformat}
testReversibleStoreScanner(org.apache.hadoop.hbase.regionserver.TestReversibleScanners)
 Time elapsed: 0.512 sec <<< ERROR!
java.lang.NullPointerException: null
at 
org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.getOrMakeChunk(MemStoreLABImpl.java:242)
at 
org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.copyCellInto(MemStoreLABImpl.java:118)
at 
org.apache.hadoop.hbase.regionserver.Segment.maybeCloneWithAllocator(Segment.java:168)
at 
org.apache.hadoop.hbase.regionserver.AbstractMemStore.maybeCloneWithAllocator(AbstractMemStore.java:268)
at 
org.apache.hadoop.hbase.regionserver.AbstractMemStore.add(AbstractMemStore.java:107)
at 
org.apache.hadoop.hbase.regionserver.TestReversibleScanners.writeMemstoreAndStoreFiles(TestReversibleScanners.java:676)
at 
org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreScanner(TestReversibleScanners.java:252)
{noformat}

{noformat}
testUnflushedSeqIdTracking(org.apache.hadoop.hbase.regionserver.wal.TestFSHLog) 
 Time elapsed: 0.357 sec  <<< ERROR!
java.lang.NullPointerException: null
at 
org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.getOrMakeChunk(MemStoreLABImpl.java:242)
at 
org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.copyCellInto(MemStoreLABImpl.java:118)
at 
org.apache.hadoop.hbase.regionserver.Segment.maybeCloneWithAllocator(Segment.java:168)
at 
org.apache.hadoop.hbase.regionserver.AbstractMemStore.maybeCloneWithAllocator(AbstractMemStore.java:268)
at 
org.apache.hadoop.hbase.regionserver.AbstractMemStore.add(AbstractMemStore.java:107)
at 
org.apache.hadoop.hbase.regionserver.AbstractMemStore.add(AbstractMemStore.java:101)
at org.apache.hadoop.hbase.regionserver.HStore.add(HStore.java:701)
at 
org.apache.hadoop.hbase.regionserver.HRegion.applyToMemstore(HRegion.java:3952)
at 
org.apache.hadoop.hbase.regionserver.HRegion.applyFamilyMapToMemstore(HRegion.java:3935)
at 
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3392)
at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3084)
at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3026)
at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3030)
at 
org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:3778)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:2903)
at 
org.apache.hadoop.hbase.regionserver.wal.TestFSHLog.testUnflushedSeqIdTracking(TestFSHLog.java:161)
{noformat}
Shall we declare the LOG as a static member ?
{noformat}
public class ChunkCreator {
private final Log LOG = LogFactory.getLog(ChunkCreator.class);
{noformat}

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971006#comment-15971006
 ] 

Hadoop QA commented on HBASE-16438:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 8s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 59s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
17s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 143m 22s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.snapshot.TestExportSnapshot |
|   | hadoop.hbase.snapshot.TestMobExportSnapshot |
|   | hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
|   | hadoop.hbase.snapshot.TestSecureExportSnapshot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12863630/HBASE-16438_addendum.patch
 |
| JIRA Issue | HBASE-16438 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 11e73a4dadc6 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / c2c2178 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6461/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/6461/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6461/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6461/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was 

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970769#comment-15970769
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


checking the test failure.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970720#comment-15970720
 ] 

Hudson commented on HBASE-16438:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2873 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2873/])
HBASE-16438 Create a cell type so that chunk id is embedded in it (Ram) 
(ramkrishna: rev c2c2178b2eebe4439eadec6b37fae2566944c16b)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALMonotonicallyIncreasingSeqId.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCellFlatSet.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLAB.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLABImpl.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OffheapChunk.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerFromBucketCache.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionArchiveIOException.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionIncrement.java
* (add) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NoTagByteBufferChunkCell.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRecoveredEdits.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingMemStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingToCellArrayMapMemStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionPolicy.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHMobStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java
* (add) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChunkCreator.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
* (add) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemstoreLABWithoutPool.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFailedAppendAndSync.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionArchiveConcurrentClose.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Chunk.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OnheapChunk.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ExtendedCell.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALLockup.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
* (add) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ByteBufferChunkCell.java
* (delete) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreChunkPool.java


> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> 

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970620#comment-15970620
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


I did not see that code. Initially there were two versions onheap and offheap 
Cells. So we needed that. Once we have BBChunkCell that extends BBKV then it is 
not needed. Thanks.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970242#comment-15970242
 ] 

Anoop Sam John commented on HBASE-16438:


No need to have the hasArray() based special case. These cells work over BB 
directly and so considering arrayOffset() is wrong.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969988#comment-15969988
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

Hey [~ram_krish], how is it going?

I have one question regarding CellChunkMap I see in your version of 
CellChunkMap you have the following code:
{code}
if (buf.hasArray()) {
  // so extract out the seqid from here.
  // Also should indicate if there were any tags or not
  return new ByteBufferChunkCell(buf, offsetOfCell + buf.arrayOffset(), 
lengthOfCell)
} else {
  return new ByteBufferChunkCell(buf, offsetOfCell, lengthOfCell);
}
{code}

where buf is the ByteBuffer where the cell data is stored. Can you please 
explain about buf.hasArray() ?

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967308#comment-15967308
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


There is only one failure and seems to be unrelated.  Will verify once again 
locally. I shall commit this by EOD my time.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965628#comment-15965628
 ] 

Hadoop QA commented on HBASE-16438:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 28 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 42s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 49s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 107m 35s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
30s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 153m 35s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12862980/HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch
 |
| JIRA Issue | HBASE-16438 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 525fecfa32a4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / cf3215d |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6398/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6398/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6398/console |
| 

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962778#comment-15962778
 ] 

Hadoop QA commented on HBASE-16438:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 28 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 44s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 4s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
54m 42s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 5s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 211m 39s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 2m 
6s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 302m 5s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestAsyncBalancerAdminApi |
|   | hadoop.hbase.client.TestMobSnapshotCloneIndependence |
|   | hadoop.hbase.client.TestAsyncTableAdminApi |
|   | hadoop.hbase.client.TestBlockEvictionFromClient |
|   | hadoop.hbase.util.TestHBaseFsckReplicas |
| Timed out junit tests | org.apache.hadoop.hbase.backup.TestFullRestore |
|   | org.apache.hadoop.hbase.backup.TestBackupDelete |
|   | org.apache.hadoop.hbase.backup.master.TestBackupLogCleaner |
|   | org.apache.hadoop.hbase.backup.TestBackupMultipleDeletes |
|   | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12862655/HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch
 |
| JIRA Issue | HBASE-16438 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 07ebc6072e2e 4.8.3-std-1 #1 SMP Fri Oct 21 

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961236#comment-15961236
 ] 

Ted Yu commented on HBASE-16438:


NPE from MemStoreLABImpl.java appeared in several failed tests:
{code}
testRegionObserverScanTimeStacking(org.apache.hadoop.hbase.coprocessor.TestRegionObserverScannerOpenHook)
  Time elapsed: 0.354 sec  <<< ERROR!
java.lang.NullPointerException: null
at 
org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.getOrMakeChunk(MemStoreLABImpl.java:242)
at 
org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.copyCellInto(MemStoreLABImpl.java:118)
at 
org.apache.hadoop.hbase.regionserver.Segment.maybeCloneWithAllocator(Segment.java:167)
{code}
Please fix.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961160#comment-15961160
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

+1 to commit this once QA is passing

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960573#comment-15960573
 ] 

Hadoop QA commented on HBASE-16438:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 24 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 50s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 46s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 46s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
56m 55s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 5s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 215m 56s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 
12s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 308m 37s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 
|
|   | hadoop.hbase.client.TestAsyncBalancerAdminApi |
|   | hadoop.hbase.client.TestAsyncTableAdminApi |
|   | hadoop.hbase.master.TestMasterBalanceThrottling |
| Timed out junit tests | 
org.apache.hadoop.hbase.master.TestGetLastFlushedSequenceId |
|   | org.apache.hadoop.hbase.client.TestMultiRespectsLimits |
|   | org.apache.hadoop.hbase.master.TestMasterShutdown |
|   | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
|   | org.apache.hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12862433/HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch
 |
| JIRA Issue | HBASE-16438 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 27693a24aee7 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Edward Bortnikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959577#comment-15959577
 ] 

Edward Bortnikov commented on HBASE-16438:
--

[~anastas] - is this a +1? 

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959565#comment-15959565
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

OK guys, I understand that seqID is not in ByteBuffer and this is how it was 
before this JIRA. If you don't like to write it on data-chunk I accept it. I 
will write seqID as part of the cell-representation in the CellChunkMap.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959127#comment-15959127
 ] 

Anoop Sam John commented on HBASE-16438:


We were never doing this way of writing seqId bytes onto MSLAB chunks (Data 
chunks)..  we keep it as state.  And FYI, when we flush cells to HFiles, then 
also we write seqId but as a VLong.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959036#comment-15959036
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

Hey once again!

I got now the idea why the seqID shouldn't probably be in the data-chunk's 
byte-buffer upon Cells creation. If upon snapshot the BBKV is streamed out 
directly to the hfileblocks so seqIDs must move to disk as well, then this is 
bad enough to place it in the CellChunkMap and probably decrease its 
performance. So if this is the issue I agree to accommodate the seqIDs in the 
CellChunkMap. Please let me know what do you think...

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958938#comment-15958938
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

Hi Guys!

I really apologize for this late comment. If you haven't yet committed, please 
consider this comment. 
I just had time to went over the last version of the patch once again. I left 
some comment in the RB, please take a look there, but most of them are not very 
important. 
The very important thing is that I have just realized that seqID is not written 
anywhere in the ByteBuffer! Please correct me if I am wrong...

I was sure (that even before this patch) the serialization of the Cell included 
writing of the seqID to the ByteBuffer and we are talking only about 
CellChunkMap. Obviously, if seqID is not written in MSLAB Chunk upon cell 
creation (when the cell is still indexed with CSLM) it can not be added there 
later when we have the transfer to CellChunkMap. So the only two options are 
(1) to add seqID in the index-chunk when CellChunkMap is created 
(in-memory-flush) or (2) to write seqID into the data-chunk when a cell is 
added to the memstore IN ADDITION to seqID continue existing in the Cell Object.

I strongly suggest to do the second option.
First, if seqID remains in the Cell object, there is completely no new work 
being done in the DefaultMemStore or active segment. The seqID in the data 
chunk are not going to be accessed by anyone except CellChunkMap.
Second, addition of another 8 bytes to key+value, which is anyway big sized is 
not significant. While adding those 8 bytes to cell-representation in the 
ChunkMap is increasing the metadata per cell almost twice.

I really appreciate all the hard work that you are doing! You were talking 
about this issue even before, I am sorry I didn't understand that. But I think 
this task is not complete before we finalize this seqID issue...

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957330#comment-15957330
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

OK. Let's commit and we can fix things together with moving to CellChunkMap, if 
needed.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956329#comment-15956329
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


[~anastas]
Pls have a look at RB and tell what you think. We can close this JIRA and move 
on to CellChunkMap.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955691#comment-15955691
 ] 

Hadoop QA commented on HBASE-16438:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 37s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 24 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
34m 51s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 0s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 108m 40s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
28s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 162m 45s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12861912/HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch
 |
| JIRA Issue | HBASE-16438 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 31f1d2b2b4cf 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / e916b79 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6312/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6312/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> 

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955403#comment-15955403
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


[~anastas]
Some how I missed your updates. The latest patch here has the change from long 
to int for the chunkid. And adds the constant of how many bytes we need for a 
cell that will be added to CellChunkMap.


> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954867#comment-15954867
 ] 

Hadoop QA commented on HBASE-16438:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 24 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
26s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
30m 14s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 56s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 128m 40s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
29s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 178m 54s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | 
org.apache.hadoop.hbase.client.TestAvoidCellReferencesIntoShippedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12861835/HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch
 |
| JIRA Issue | HBASE-16438 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux c8a0a3120ea5 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / e916b79 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| whitespace | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6309/artifact/patchprocess/whitespace-eol.txt
 |
| unit | 

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954726#comment-15954726
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

[~ram_krish], are there any differences in the new patch that worth to add a 
new version to the RB?

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954724#comment-15954724
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

Guys,

I think we are all confused. What I am talking about Cell-reference 
(chunkID+length+offset) need to be only part of CellChunkMap it is accessed 
only by CellChunkMap (written and read only by CellChunkMap). We can even not 
to create any new cell type. CellChunkMap when created is going to write 
(chunkID+length+offset) per cell in its index-chunk. Later when it is asked to 
return a Cell, CellChunkMap is going to read the relevant cell-reference from 
the chunk-index, then access the data-chunk and read the "true" cell data 
(key,value,seqID, etc.) and return it to anyone who needs.

What is the problem with that?

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954573#comment-15954573
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


IMHO the whole new cell should come only when we have CellChunkMap. The cell in 
the current patch is fine as it does not add much overhead except for adding an 
entry in the ChunkID map. But the new cell where seqId has to be embedded in 
the cell has to happen only while moving over to CellChunk representation. With 
default memstore case it is unwanted.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954534#comment-15954534
 ] 

Anoop Sam John commented on HBASE-16438:


On seqId being in which chunk.. We should be carefully taking a call..  One 
disadv of having it in data chunk is that, when the Cell is in active segment 
(Not ChunkMapped) and a parallel read happens, we will have to match its seqId 
against the read pnt of that Read request.  For this, if the seqId is in Cell 
object itself as a long state, we have that value ready for compare. If at that 
time itself, we moved that as 8 bytes after key and value bytes in data chunk, 
we will have to decode the seqId from the 8 bytes. This will happen for every 
cell.   More over, this MSLAB copy is a top level step which happens even for 
DefaultMemstore.  So we may end up doing this way of encoding 8 bytes in chunk 
for all cases. (There is no in memory compaction or no chunk map based flush).. 
All such cases, this impact will come.   So pls be careful abt making such a 
choice.  

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953981#comment-15953981
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

[~anoop.hbase] and [~ram_krish], 

it looks like we all agree to put the seqID in the data-chunk (not in index) 
together with the key, value and etc. This can be done by creating a new type 
of Cell that doesn't include seqID as a field.

bq. Am just getting confused here. When you say ChunkCell are you telling the 
BBChunkCell in current patch? In current patch there is no extra overhead at 
all. But if you are talking about the cell to be moved to CellChunkMap - yes 
then it will have some overhead in terms of serialization and not in terms of 
heap overhead

I am talking about the Cell-representation that is going to be part of 
CellChunkMap. I am confused here by myself and don't know what is exactly the 
name of such cell. But I need a constant saying how many bytes such a Cell 
takes...

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953244#comment-15953244
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


bq.In ByteBufferChunkCell, please explain me why to add this new class? Why can 
not the existing BBKV just have a new method - getChunkId() - to return the 
chunk id in the 0th offset of the backing BB?
We now have BBKV every where in write path and we can also make use of it in 
read path to form cells coming out of hfileblocks. Since we have added 
getChunkId() to the ExtendedCell any cell can make use of this getchunkId. 
(though it was not generic it was added to make things simpler).
Since we deal with ExtendedCells we create a specific impl of BBKV that returns 
the chunkId alone and by default it will be returning -1.
bq.In ByteBufferKeyValue or in MSLAB or anywhere else, please add constant 
saying what is the size in bytes of the ChunkCell or what I call 
cell-representation (chunkId + offset + length + seqId), so I can use it later.
Ok.
bq.OK. So lets have a new cell representation.
Ok fine we can make use of it.
bq.This is not a desired situation. We are increasing from 12 bytes to 20 
bytes, almost twice... We should not do it unless it is very very necessary...
Some where we need the seqId of every cell getting written to the cellChunkMap. 
How do you think you can avoid it? You have some idea on that?
bq.. In ByteBufferKeyValue or in MSLAB or anywhere else, please add constant 
saying what is the size in bytes of the ChunkCell 
Am just getting confused here. When you say ChunkCell are you telling the 
BBChunkCell in current patch? In current patch there is no extra overhead at 
all. But if you are talking about the cell to be moved to CellChunkMap - yes 
then it will have some overhead in terms of serialization and not in terms of 
heap overhead. 

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953207#comment-15953207
 ] 

Anoop Sam John commented on HBASE-16438:


bq.Do you mean the seqID is going to be written in index-chunk only and is not 
going to be written in the main-chunk, holding key, value and etc.? So no 
duplication? Are you sure? If so, then already little better, but still I would 
like to keep the Cell meta data smaller.
Yes. Either to main chunk or to meta data chunk. As of now, it has to be in the 
meta chunk.. Ya we can change that if we make a new Cell impl for representing 
the cells added to the MSLAB. ( Seems we have one already). One more issue is u 
will need some more refactoring at the base classes level so that u have a 
class to extend here. BBKV u can not as that is already having a long state for 
seqId (wasting 8 bytes on heap space per cell).
bq.Why can not the existing BBKV just have a new method - getChunkId() - to 
return the chunk id in the 0th offset of the backing BB
The cell impl BBKV might be used elsewhere also where its data is NOT in MSLAB 
chunks.  So blindly reading 1st 4 bytes of the chunk's backing buffer is not 
correct. You need a new Cell impl with this way of impl for the getChunkId() 
method.   I see this method is added to ExtendedCell now.  So other impls of 
the ExtendedCell (Like KV) will throw Exception?

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953177#comment-15953177
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

bq. What specific question in RB are you looking out for? 

OK. I will write here the questions that bother me and I don't see responses:
1.In ByteBufferChunkCell, please explain me why to add this new class? Why can 
not the existing BBKV just have a new method - getChunkId() -  to return the 
chunk id in the 0th offset of the backing BB?
2. In ByteBufferKeyValue or in MSLAB or anywhere else, please add constant 
saying what is the size in bytes of the ChunkCell or what I call 
cell-representation (chunkId + offset + length + seqId), so I can use it later.
I will review the existing patch once again

bq. ChunkId is per ByteBuffer backing the chunk. I can change the chunkId to be 
an int.

You got it yourself, I also thought so for a moment. I am talking about ChunkID 
of where each cell is located, which is saved per cell. 
Please do change chunkID to int, but check for overflow (at least log some 
error). 
I believe we should strive to decrease number of bytes the cell representation 
is taking, because this is the reason why are we doing the CellChunkMap...

bq. My Q was, this Cell meta data (ChunkId, offset, length) also we planned to 
write to chunks. So what is the difference? In this chunk or that chunk?

Do you mean the seqID is going to be written in index-chunk only and is not 
going to be written in the main-chunk, holding key, value and etc.? So no 
duplication? Are you sure? If so, then already little better, but still I would 
like to keep the Cell meta data smaller.
The smaller the Cell meta data is (hopefully only chunkId, offset, length and 
only 12 bytes) the less is the meta-data-overhead per cell is and the more we 
can squeeze into single index-chunk (CellChunkMap). The smaller CellChunkMap is 
we all enjoy the locality for scans and the binary search can hit the 
processor-cache easily.

bq. The only thing is we should go with fixed 8 bytes for that. 

This is not a desired situation. We are increasing from 12 bytes to 20 bytes, 
almost twice... We should not do it unless it is very very necessary...

bq. So now if you are going to write the seqId in the BB backing every cell, 
then the seqId as the state variable is not needed at all and hence you may 
need a new cell representation for it. 

OK. So lets have a new cell representation.

bq. Otherwise we should still go with it and use the seqID as a caching value 
in addition to having it in the BB. 

Why to have the duplication of the same?


> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953063#comment-15953063
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


Ok. Got it now. You are asking for the CellChunk representation. Yes we need 
chunkId + offset + length + seqId. Seqid if embedded with Cell data it is 
easier to retrieve it. Just doing getSeqId can decode the value from the 
backing BB. The only thing is we should go with fixed 8 bytes for that. 
So now if you are going to write the seqId in the BB backing every cell, then 
the seqId as the state variable is not needed at all and hence you may need a 
new cell representation for it. Otherwise we should still go with it and use 
the seqID as a caching value in addition to having it in the BB. 

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953048#comment-15953048
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


[~anastas]
What specific question in RB are you looking out for? If it is the length of 
every cell along with this meta data for the chunks? IF so I think I answered 
it and going thro the latest comments I can change the chunkId to be an int.
And as I replied in that RB the chunkId will be once per ByteBuffer and not per 
cell. 
So when you convert to chunk map you wll be needing the length, offset and the 
seqId only per cell. ChunkId is per ByteBuffer backing the chunk.


> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952686#comment-15952686
 ] 

Anoop Sam John commented on HBASE-16438:


U mean we can keep the seqId of cells in the original chunk where the cell key 
and value bytes are present. Yes we can do.  We can make a new Cell impl for 
this which is not having the seqId as a state.   My Q was, this Cell meta data 
(ChunkId, offset, length) also we planned to write to chunks.  So what is the 
difference? In this chunk or that chunk?

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952684#comment-15952684
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

[~anoop.hbase], we may read seqId from the byte buffer if needed... We keep 
offset and length, so we can read it there... also keys and values are there...

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952681#comment-15952681
 ] 

Anoop Sam John commented on HBASE-16438:


+1 for chunkId being int.  That is fine IMO.

We have seqId as a state variable in Cell objects.  Now when the cells are 
moved to ChunkMap, we lost the cell objects and we have to keep the seqId some 
where.  On a different way, we could make the seqId 8 bytes also sit after the 
Key, Value bytes in the original chunk.  But again, this cell meta data also 
added to chunks right? So what is the difference?

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952659#comment-15952659
 ] 

Ted Yu commented on HBASE-16438:


I think 4 bytes should be large enough to represent chunk Id. 

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952657#comment-15952657
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

As I am reviving the CellChunkMap now, having cell-representation of size of 24 
bytes, sounds way too much for me.

Per Cell we save: chunk ID (long - 8 bytes), offset (int - 4 bytes), length 
(int - 4 bytes), seqID (long - 8 bytes).

Can someone explain me why is it a must to keep the seqID there?

For chunk ID, can 4 bytes int be enough? If we are running out of 2^31 positive 
numbers we may start using the next 2^31 negative numbers... 
Having 2^32 chunks sounds quite enough. With the current chunk size (2MB=2^21 
bytes) we can cover 8 PetaBytes per RS, or we can increase size of chunk...
Thoughts?

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952591#comment-15952591
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

Hi [~ram_krish],

Can you please answer my comments on RB? I saw you put a new patch there and I 
just do not want to review it once again, before you address by previous 
questions...
Thanks!

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950463#comment-15950463
 ] 

Hadoop QA commented on HBASE-16438:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 24 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
29m 8s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 52s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 124m 11s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
30s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 172m 45s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12861377/HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch
 |
| JIRA Issue | HBASE-16438 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux f9b535d23927 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / d033cbb |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6275/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6275/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Create 

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15949018#comment-15949018
 ] 

Hadoop QA commented on HBASE-16438:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 13 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
28m 22s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
42s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 29s 
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 
total (was 0) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 48s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 106m 24s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
29s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 153m 3s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRecoveredEdits |
|   | hadoop.hbase.regionserver.TestWALLockup |
|   | hadoop.hbase.regionserver.TestStore |
|   | hadoop.hbase.regionserver.TestHRegionReplayEvents |
|   | hadoop.hbase.regionserver.wal.TestDurability |
|   | hadoop.hbase.io.hfile.TestScannerFromBucketCache |
|   | hadoop.hbase.regionserver.TestCompactionArchiveIOException |
|   | hadoop.hbase.regionserver.TestCompactionArchiveConcurrentClose |
|   | hadoop.hbase.regionserver.TestRegionIncrement |
|   | hadoop.hbase.regionserver.TestFailedAppendAndSync |
|   | hadoop.hbase.regionserver.TestHMobStore |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12861207/HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch
 |
| JIRA Issue | HBASE-16438 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | 

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948827#comment-15948827
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

Thanks [~ram_krish] for publishing a new patch!

Please pay attention that I have published some replies on your comments on the 
previous version of the patch. It just happened simultaneously. 
Please take a look there. I'll look on your new patch soon. Thanks!

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945040#comment-15945040
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

OK, I see from RB that many things are changed in the new patch. Let us see the 
new patch and review it again. In general, I am OK if we commit everything not 
related to performance here. Then all the issues with GC performance (as soft 
ref) we can do later. This commit should be correct, and performance issues we 
can deal with later.

And BIG BIG BIG thanks to [~ram_krish] for doing amazing work here! And to 
[~anoop.hbase] for great ideas and reviews!

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944707#comment-15944707
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


[~anastas]
If we are going with soft ref approach and then hardening it on creating 
CellChunkMap I can create a soft ref map version in ChunkCreator but the 
process of changing into hard ref has to happen only when CellChunkMap 
fllattening happens which will not be done by this JIRA.
So I thought we can just proceed with existing version and then in the 
CellChunkMap JIRA update the map to convert from soft to hard. Even if that 
requires a change in ChunkCreator map then we will do it there.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943235#comment-15943235
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

bq. Chance of getting GCed might be bit more when chunk is actually removed 
from Map (No refs to it at all) than still referring with one SoftRef.

I am sure you'll see no difference. Think what the chances are that all the 
cells in the (big sized) chunk are replaced till it is flushed in memory.

bq. So shall we do that also in this JIRA or should we just finish the patch in 
the current state and go with a new JIRA to solve the GC problem that we may 
face?

As we are designing the Creator now I think it should be written appropriately 
from the beginning. I mean with "convertible" references in the map, if we 
choose this way. Why to write it anyhow and then rewrite it?

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943210#comment-15943210
 ] 

Anoop Sam John commented on HBASE-16438:


New issue is ok. Ping [~carp84]..   Pls get his buy in as he fixed the other 
issue.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943174#comment-15943174
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


bq.Chance of getting GCed might be bit more when chunk is actually removed from 
Map (No refs to it at all) than still referring with one SoftRef. 
That is why I thought of going with the ref count way. May be it is heavy 
lifting but we can try the Soft ref way. So shall we do that also in this JIRA 
or should we just finish the patch in the current state and go with a new JIRA 
to solve the GC problem that we may face?

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943152#comment-15943152
 ] 

Anoop Sam John commented on HBASE-16438:


Chance of getting GCed might be bit more when chunk is actually removed from 
Map (No refs to it at all) than still referring with one SoftRef.  Ya we are 
not sure when it will get collected by GC.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943148#comment-15943148
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

bq. One doubt - the Soft ref will get cleared only when the OOME is hit?

I don't think so. This is what they say in the Oracle documentation:

All soft references to softly-reachable objects are guaranteed to have been 
cleared before the virtual machine throws an OutOfMemoryError. Otherwise no 
constraints are placed upon the time at which a soft reference will be cleared 
or the order in which a set of such references to different objects will be 
cleared. Virtual machine implementations are, however, encouraged to bias 
against clearing recently-created or recently-used soft references. 

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943144#comment-15943144
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


bq. and .
Yes this type will work. 
bq.All the remaining chunks seems of no use and can be immediate removed from 
the map.
Yes this will work provided we hit the limit to convert to CellChunkMap. In the 
case that was presented here we may not actually grow to that size? Then in 
that case we will have the chunks getting referenced?
So if we have this Soft reference way then may it is much easier so when we 
reallly convert from CSLM to CellChunkMap that time we can convert the soft ref 
to hard ref by having another map in the ChunkCreator. Any soft ref will 
automatically get removed when the GC tries to clear them for a GC cycle. 
One doubt - the Soft ref will get cleared only when the OOME is hit?

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943112#comment-15943112
 ] 

Anoop Sam John commented on HBASE-16438:


No I got what u were saying.  Change the ref (harden) on the go   Ya agree. 
This is kind of the way I was saying.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943108#comment-15943108
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

bq. But am not sure how you mean this hardening it again. 

public class SoftReference extends Reference
So I think you can use just reference in the map type and later just to update 
the map with new mapping (delete old mapping from some chunkID to soft 
reference and add a new mapping from same chunkID to hard reference). If this 
doesn't work for some reason we may have two maps  and 
.

bq. Now when a segment is getting converted from CSLM based into CellChunkMap 
based, can we just track the chunkIds actually getting used? (Which cells 
getting moved into ChunkMap). All the remaining chunks seems of no use and can 
be immediate removed from the map.

Nice idea. I think the result of this idea is the same as combining soft and 
hard pointers in the Creator's Map. However, with combining soft and hard 
pointers in the Creator's Map the chunks can be GCed also in active segment.

bq. Do you feel that can work? I have implemented the same in a patch but not 
posted it as I need to see if there is really a perf penalty.

I saw this idea, I feel it can work, I think it is a bit complicated and may 
cost us some performance. I believe we can achieve the same for less. But if 
you want to go for it, I am OK with that.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943054#comment-15943054
 ] 

Anoop Sam John commented on HBASE-16438:


Ya may be with every cell keeping the ref counted will have a -ve impact.  Need 
to test and prove any way.  We might not need a strict way. As much as 
possible, as early as possible,  allow the chunks to be GCed.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943042#comment-15943042
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


Pls see the comment on the idea  how to track the chunk id that is not really 
getting used. 
Do you feel that can work?  I have implemented the same in a patch but not 
posted it as I need to see if there is really a perf penalty.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943037#comment-15943037
 ] 

Anoop Sam John commented on HBASE-16438:


I can think of some way. Not sure whether it is crazy or not.
We have the id vs chunk Mapping globally in ChunkCreator.   Also we will have 
chunk ids tracked in MSLABImpl which this mslab deals with.  Now when a segment 
is getting converted from CSLM based into CellChunkMap based,  can we just 
track the chunkIds actually getting used? (Which cells getting moved into 
ChunkMap). All the remaining chunks seems of no use and can be immediate 
removed from the map.   Still while the segment is active those chunks can not 
get GCed, which is not the case right now..  But once the in memory flush 
happens, there can get GCed.   Just throwing out some thoughts.

Any way as said, there is no possibility any OOME which was fixed by Yu Li.   
Only thing is better GC possibility we are lossing.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942980#comment-15942980
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


bq.We can "harden" the references in the map in the process of transferring to 
CellChunkMap (flattening).
I agree to Anoop's point here. But am not sure how you mean this hardening it 
again. So considering the fact that if the ChunkID map was having soft ref to 
the Chunks and we added the cells from these chunks to CSLM as in the above use 
case of duplicate cells if the cells are removed from the CSLM and remove the 
ref to these chunks they can be GCed. So they are soft references here. 
So as per your idea, we create a CellChunkMap from these items in CSLM and that 
time we convert those chunk reference to harden references. How can that be 
done? The chunkId map will have a  signature. So while 
converting to CellChunkMap every soft ref of this chunk should now be converted 
to a direct reference. Sorry I get your idea but not sure on the impl thing 
that you are suggesting here.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942943#comment-15942943
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

bq. I dont think Soft ref can be used. Once the CSLM is converted to 
CellChunkMap, we dont have any hard refs to Chunks from any where (Cells are 
gone). Now the ChunkMap contain chunkId and offset , length info. Now assume a 
GC picked and removed these chunks (Ya there are no hard refs to it), and after 
a read comes and we want to read back that Cell, what to do? 

We can "harden" the references in the map in the process of transferring to 
CellChunkMap (flattening). The chunks that can not be reached will not be a 
part of this process and their IDs won't be found so their references are not 
going to be "harden", which is OK.

bq. So the Chunk that is put in the map will have a soft reference?

Yes. The chunk that is used for CSLM or CellArrayMap is going to have soft 
reference the chunk that is used for CellChunkMap is going to have hard 
reference.

bq. Can we say CellChunkMap to be used along with off heap MSLAB pool usage 
only?

This is the first question to answer. I think we should not limit the 
CellChunkMap to off-heap right now. As we have seen with CompactingMemStore 
only when you implement it all and make a good performance testing you can see 
what is benefit. As now we see the merge is a good thing to do, although in the 
past we already thought to remove this code. So if we implement CellChunkMap 
now without any possibility to use it on-heap, we are limiting ourselves 
without any experimental evidence. I believe we should write the code generally 
enough.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942907#comment-15942907
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


Sorry missed these latest questions/comments.
bq. When CellChunkMap is not in use, may be we dont need to keep this id vs 
chunk map at all.. How abt we enable this feature iff MSLAB pool is in place? 
Just asking
I was infact first thinking we could do this CellChunkMap only if there is pool 
enabled so that there is no need for all this chunkCreation and maintenance 
overhead. But later as per our discussion it evolved that chunk creations is 
there when ever MSLAB is on.

bq.In short, Soft References are used in Java to implement caches and objects 
pointed by soft references can be GCed if no other (hard reference) is pointing 
to this object.
So the Chunk that is put in the map will have a soft reference? 

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942903#comment-15942903
 ] 

Anoop Sam John commented on HBASE-16438:


I dont think Soft ref can be used.  Once the CSLM is converted to CellChunkMap, 
we dont have any hard refs to Chunks from any where (Cells are gone).  Now the 
ChunkMap contain chunkId and offset , length info.  Now assume a GC picked and 
removed these chunks (Ya there are no hard refs to it), and after a read comes 
and we want to read back that Cell, what to do? 

Ok if u also think CellChunkMap should be better used by off heap usage, can we 
say CellChunkMap to be used along with off heap MSLAB pool usage only?  In case 
of off heap MSLAB all chunks will get pooled. There wont be any on demand chunk 
creation.  So we will keep this  id vs chunk info in map iff CellChunkMap is in 
place and so off heap MSLAB pool in place? 
Any way the above mentioned use case ideally should not use MSLAB pool at all.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942878#comment-15942878
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

OK, I understand what you are saying. Then how about using Java's Soft 
References in the ChunkCreator?

https://docs.oracle.com/javase/7/docs/api/java/lang/ref/SoftReference.html

In short, Soft References are used in Java to implement caches and objects 
pointed by soft references can be GCed if no other (hard reference) is pointing 
to this object. 

This sounds to me as good (and again simple) solution. If any cell is not 
reachable from any CSLM/CellArrayMap/CellChunkMap then we are not going to read 
its chunkID and to ask for translation, so we do not care that its reference is 
cleared from the ChunkCreator's map. Just need to be careful about those null 
pointers in the ChunkCreator's Map. What do you think?

bq. I mean how to pass the info whether the CSLM has to be converted to 
CellArrayMap or CellChunkMap.

I plan to have it user-configured as part of MemStore definition (at least as a 
first step). I mean once created, some CompactingMemStore is planed to work 
with CellArayMap and another CompactingMemStore is created to work with 
CellChunkMap (mostly for the off-heap). But this is of course can be changed.



> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942323#comment-15942323
 ] 

Anoop Sam John commented on HBASE-16438:


The problem in HBASE-16195 is this.
Say we have MSLAB in place and it is not pooled at all.
The data is duplicated so much. Means one cell is added and again same more 
cells might come in.  When we add a cell to CSLM, the existing same one will 
get removed right. U can see the logic around sizing and all.
Now say a chunk is having 200 bytes size. cell1 is added which is of size 100 
and cell2 again with 100 bytes size.  Now that  chunk is full and we get 
another chunk of 200 bytes size.  Again cell1 and cell2 comes in.  CSLM will 
remove old added cell1 and cell2.  Means there are no refs to chunk1 from any 
cells..  (Forget abt concurrent scanners. In this case assume no such 
concurrent scan at all)..  chunk2 only active now.  Ideally GC can reclaim 
chunk1.   Now if we keep a ref to chunks (Any place other than from cells), 
chunk can not be reclaimed.  This was the issue.  Hope it is clear now.
As long as we have even one active cell within a chunk (The cell object is gone 
as it is converted to ChunkMap), we need its mapping.  Any time a scan can 
refer to this cell.  
How u plan to support ChunkMap feature?  I mean how to pass the info whether 
the CSLM has to be converted to CellArrayMap or CellChunkMap.  When 
CellChunkMap is not in use, may be we dont need to keep this id vs chunk map at 
all..  How abt we enable this feature iff MSLAB pool is in place? Just asking

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942275#comment-15942275
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

Hi [~anoop.hbase], [~ram_krish], and [~carp84],

As I understand, in HBASE-16195, you are avoiding adding a chunk into MSLAB's 
list of chunks in order to make the garbage collection (GC) faster. So when 
there is no more references to this chunk from a SkipList/CellArayMap, the 
chunk's memory can be freed by GC. 

So now openScannerCount in MSLAB only serves for understanding when chunks can 
be returned back to pool (if they were allocated from pool).
And why do we care for those chunks allocated from pool and don't care for 
those taken care by GC (allocated by JVM)? The problem is as following:
When a Segment is removed (let's say due to flush to disk), it is already not 
referenced from MemStore and the Segment is closed, following close of its 
MSLAB.
However, there are might be still ongoing scans accessing the chunks of this 
segment. Those chunks cannot be de-allocated by GC because they have references 
from scan. But if we return the chunks to pool, they can be reused and the 
memory corrupted under scan's hands.

Now when we introduce the ChunkCreator (keeping chunkID to chunk map) you are 
afraid that we keep references to chunks for too long and delay the GC. I am 
saying all that so you can check me whether I understand it all right. If I am 
wrong please correct me. If I am right, then I have a suggestion for the 
following *simple* solution.

As Ram has suggested keep a boolean in chunk saying if from pool or not and... 
Just remove the chunkID to chunk mapping when segment is closed (for chunks 
that are not in pool)!

The scans (if they are still working) don't need the translation from chunk ID 
to chunk. This translation is needed only for flattening/compaction when the 
segment is still alive. How about that? :)



> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940016#comment-15940016
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


I have a way to solve this problem. LEt's discuss before I put up the patch. 
Most of the other RB comments are fixed.
-> Now since we need to if the chunk is from pool or not - the Chunk will have 
a boolean indicating whther the chunk was created for the pool. Say we have 
isFromPool() will return true for those chunks.
-> Every chunk will have an AtomicInteger ref count.
-> When the MSLAB does a copyToChunkCell - where we know that the cell has to 
have a chunk(comes out of chunkCreator) we do an increment of the refCount.
-> Now in the MemstoreImpl when we do getCellSet().add() ( we need to have a 
new API in CellSet which actually returns the cell that was already there in 
the CSLM which is returned by CSLM.put() returns. Now we only have 
cellSet#add() which return boolean).
-> On this returned cell (which is the actual duplicate cell) we get the 
chunkId from the Cell. remember we now have a BbChunkCell which can give the 
chunkid frm the 0th offset.
-> Use this chunkId to actually do a decrement of the reference count of this 
chunk. For this we need a decrementChunkRefCount in MSLAB interface. I think it 
is valid because MSLAB impl is nothing but Chunks.
-> Now on doing this decrementChunkRefCount  , we could check if the result is 
now 0 and if so just remove that chunk from the chunkCreator map. So by this 
way we are making sure that the reference to the chunk is released immediately.
-> Things to note is that in case the chunk is from Pool this 
increment/decrement will not have any impact. This will impact only when we 
have ondemand chunks.
-> There is an atomic ref count operation happening now which may add on to the 
write path overhead. May be need to see the impact. but remember this is going 
to happen only if there are lot of duplicates like in HBASE-16195. In a normal 
case this should not be a problem because the CSLM#put() is going to return a 
null as there is no duplicate and so there are no such problems. And infact in 
such a case the GC issue mentioned in HBASE-16195 will not happen as all the 
chunks are needed till the MSLAB is closed.
Thoughts!!!

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939526#comment-15939526
 ] 

Yu Li commented on HBASE-16438:
---

bq. Not clearing/removing from Q is not the real issue...That is why that jira 
changed it to keep only chunks from pool.
Exactly.

bq. But as such there wont be OOME for sure.
Yes, with fix on memstore size accounting in HBASE-16194, won't be OOME


bq. Now we will keep ref to Chunks as long as the MSLAB, which created it, is 
not closed... But to have a better GC in this scenario, HBASE-16195 would have 
helped , which will be broken by this jira.
Yes, this is a true concern...

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938089#comment-15938089
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


Yes got it now. Great point.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936900#comment-15936900
 ] 

Anoop Sam John commented on HBASE-16438:


Not clearing/removing from Q is not the real issue. Code wise there may not be 
a remove. But the MSLABImpl object keeps this Q and the MSLAB object become 
dead once the segment referring it gets flushed.  So some time we will remove 
it.  The issue was specific to use case where there are many updates to same 
Cell.  MSLAB was in use in that!   So we have cells from one chunk but all 
become irrelevant after some time as new cells comes in. Adding new cells to 
CSLM would remove the old.  If we were not referring to that chunk any where 
other than those original Cells, a GC can collect it.  But adding to the Q was 
preventing it.   That is why that jira changed it to keep only chunks from pool.
Now we keep refer to all chunks created out of ChunkCreator. So we are back to 
old problem

But as such there wont be OOME for sure.  Because as part of above jira 
[~carp84] fixed another issue where we account for the cell size (when its data 
bytes in chunk) and so we will end up in size breaching at region level/ global 
level and normal/forced flush will happen.  So I dont feel like any OOME issue. 
 But to have a better GC in this scenario, HBASE-16195 would have helped , 
which will be broken by this jira.

Am I making things clear?

cc [~carp84]

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936878#comment-15936878
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


Ok seeing this comment now. I think in HBASE-16195 there was a queue which was 
not getting cleared at all and hence there was problem?
{code}
 this.chunkQueue.add(c);
{code}
I could see the above line the patch V4 attached in that JIRA where this queue 
is now added only when there is pool. Previously there was no remove I believe 
from this queue. 
Now in the these patches we hold on to the Chunk but still we clear it on close 
or when the scanner close it right? Anyway these chunks are needed till the 
flush completes and so some where there is a reference still maintained. Or am 
i missing something?

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936708#comment-15936708
 ] 

Anoop Sam John commented on HBASE-16438:


Pls see HBASE-16193..   Now we will keep ref to Chunks as long as the MSLAB, 
which created it, is not closed.With that we will kind of break the fix in 
HBASE-16195. Thoughts!

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932513#comment-15932513
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


some test failures I have corrected due to the design change. Will look into 
others before I post the next patch. Will wait for the comments in RB too.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932334#comment-15932334
 ] 

Hadoop QA commented on HBASE-16438:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 9 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 4s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
33s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 26s 
{color} | {color:red} hbase-server generated 3 new + 1 unchanged - 0 fixed = 4 
total (was 1) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 48s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 9s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
18s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 16s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestBulkLoad |
|   | hadoop.hbase.regionserver.TestDateTieredCompactionPolicy |
|   | hadoop.hbase.master.locking.TestLockManager |
|   | hadoop.hbase.master.balancer.TestRegionLocationFinder |
|   | hadoop.hbase.procedure.TestProcedureManager |
|   | hadoop.hbase.regionserver.TestMemstoreLABWithoutPool |
|   | hadoop.hbase.regionserver.TestDefaultCompactSelection |
|   | hadoop.hbase.regionserver.TestWALMonotonicallyIncreasingSeqId |
|   | hadoop.hbase.regionserver.TestDateTieredCompactionPolicyOverflow |
|   | hadoop.hbase.master.TestCatalogJanitor |
|   | hadoop.hbase.regionserver.TestHRegionFileSystem |
|   | hadoop.hbase.master.locking.TestLockProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12859505/HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch
 |
| JIRA Issue | HBASE-16438 |
| 

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932277#comment-15932277
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

[~ram_krish] and [~anoop.hbase],

Thank you very much! Going to look on the new patch!

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, 
> HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927705#comment-15927705
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

Hi [~ram_krish],

I understand that you have already done some refactoring work for Chunks 
creation/allocation. I appreciate it a lot! I just thought may be it can be 
quicker and easier for you if I do the creator/allocator. But if you have the 
new design/patch already then just go ahead. Do you mind to publish the new 
code on RB? BTW if you are using the lock to synchronize the threads asking for 
the current chunk in one MSLAB, then why do you need CAS there as well?

[~anoop.hbase], I also looked on that code and thought there is a bug there! :) 
But then realized it is OK due to this UNINITIALIZED. This is why I am saying 
we do not add any new delay here in chunks allocation. Anyway if two or more 
threads need new chunk one is initializing and the rest are waiting...

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927549#comment-15927549
 ] 

Anoop Sam John commented on HBASE-16438:


U mean this?
{code}
int oldOffset = nextFreeOffset.get();
  if (oldOffset == UNINITIALIZED) {
// The chunk doesn't have its data allocated yet.
// Since we found this in curChunk, we know that whoever
// CAS-ed it there is allocating it right now. So spin-loop
// shouldn't spin long!
Thread.yield();
continue;
  }
{code}
Oh ya.. Seems will handle.. So all is good.
Am not sure with this light weight way with no locking, how the Chunk AND BB 
create in one way u plan to do.  I may be missing some thing and did not really 
thought abt it. But as said, any change in the way here, pls do JMH to prove no 
perf regression and then only change.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927545#comment-15927545
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


I think see the alloc() code. There again there is a CAS operation happening. 
So nextfReeOffset should be not be in UNINITIALIZED state as I send in the 
previous comment.
https://issues.apache.org/jira/browse/HBASE-16438?focusedCommentId=15927525=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15927525

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927541#comment-15927541
 ] 

Anoop Sam John commented on HBASE-16438:


The discussion here made to again read the code in MSLABImpl#getOrMakeChunk()
{code}
while (true) {
  // Try to get the chunk
  Chunk c = curChunk.get();
  if (c != null) {
return c;
  }

 .
c = new OnheapChunk(chunkSize);// When chunk is not from pool, always 
make it as on heap.
  ...
  if (curChunk.compareAndSet(null, c)) {
// we won race - now we need to actually do the expensive
// allocation step
c.init();
...
return c;
  }
  ..
  // someone else won race - that's fine, we'll try to grab theirs
  // in the next iteration of the loop.
}
{code}
I believe there is a possible multi thread case bug here.
2 Threads trying to make chunk. Both made OnheapChunk objects (Not init) and 
only one will succeed in CAS. That will do the init and so make BB in chunk. 
Unless this init() call is happened, the chunk's underlying data is null.  Now 
the second thread, after failed in CAS op, went to next iteration and see 
curChunk is not null and it will use that. But remember, if this thread getting 
chance 1st before actual init() op on this chunk, any try for copy on to it 
will fail with NPE !

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927540#comment-15927540
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


bq.ByteBufferChunkCell should be in hbase-server beside the MSLAB rather than 
out in hbase-common? It is a server-side only thing? Ditto with copyToChunkCell 
and ChunkCell? What you think?
Yes I agree. 
bq.private long id = -1;
It is passed on Construction. Can it change during lifetime of the Cell?
Can make it final. It won't change. It is id only not an offset. Just an unique 
number.
bq.Will it ever be the case that Cells from Chunks are persisted across 
restarts? Say, in a bucketcache that is persisted? Just wondering if the 
chunkid needs to be unique across restarts?
I don't think so. We want any chunk's life time to be available till the 
segment having the chunk is flushed. So even if it is from pool we need to know 
the chunkId till the flush happens.
bq.We have MSLABChunkCreator. So a Chunk is a 'piece' of a MSLAB? And a 
'MSLABChunkCreateor' creates chunks or allocates pieces of the MSLAB? Do we 
have to have MSLAB in the name? Is it MSLAB only?
I like the last question . MSLAB is created per segment and each MSLAB can have 
more than one Chunk. So it is not one to one here.
So now we try to designate a ChunkCreator who does the creation and management 
of these chunks.
bq.We keep a chunkIdMap? Is this of all chunks? How many chunks will there be? 
All threads will be banging on this Map?
In case of ChunkPool we are limited to the pool size. but if there is no 
chunkPool then I think we will be having more number of chunks created on 
demand. That is why we have the logic of removing these chunks Ids from the map 
when there is no pool when we do the close() of a segment so that we are sure 
that we no longer need those chunkIds. In case of pool we cannot do that as we 
reuse the chunks.
bq.Regards the below, who sets forceOnHeapOnly? Rather should we pass in the 
Cell and let the allocator figure where to allocate the memory?
>From the Cell we cannot tell because in case of ChunkPool once we run out of 
>its max size the MSLAB decides where to create the chunk and not the cell.
bq.So if no chunk pool, we keep chunk ids in a map elsewhere than in 
MSLABChunkCreateor?
am not sure if you have seen [~anastas]'s concern. Infact I thought one 
MSLABChunkCreator is enought and we can pass the singleton ref with MSLAB and 
the pool. So that chunkCreator decides on chunkCreation but the responsibility 
is with MSLAB to either as the pool or chunkCreator directly. But she feels 
that is not right and better to refactor fully in such a way that only 
ChunkCreator does chunkCreation and init(which is the costly one) and with that 
change all the CAS operation and the way it works out. 

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927525#comment-15927525
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


bq.I'll continue with the Allocator/Creator and all the rest... What do you 
think?
Oh I see. I already did some work here to impl as per your suggestion. If you 
want to continue I can reduce the scope of this patch.
I think lock unlock is needed along with CAS for the curChunk thing. And also 
if you see the other threds who is waiting for the curChunk though may get it 
but has to wait for the alloc to happen and then only it will succeed. I think 
so overall the same will happen here but only thing is the lock/unlock way may 
have some internal implication. 

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927523#comment-15927523
 ] 

stack commented on HBASE-16438:
---

I took a quick look at the patch.

ByteBufferChunkCell should be in hbase-server beside the MSLAB rather than out 
in hbase-common? It is a server-side only thing? Ditto with copyToChunkCell and 
ChunkCell? What you think?

NoTagByteBufferChunkCell <= You know what I'm going to say (smile).

Should this be final:

  private long id = -1;

It is passed on Construction. Can it change during lifetime of the Cell?

Is it an 'id' or is it an 'offset'?

Can we pass in the id when we init a chunk?

Will it ever be the case that Cells from Chunks are persisted across restarts?  
Say, in a bucketcache that is persisted? Just wondering if the chunkid needs to 
be unique across restarts?

What is the nomenclature here?

We have MSLABChunkCreator. So a Chunk is a 'piece' of a MSLAB?  And a 
'MSLABChunkCreateor' creates chunks or allocates pieces of the MSLAB? Do we 
have to have MSLAB in the name? Is it MSLAB only?

We keep a chunkIdMap? Is this of all chunks? How many chunks will there be? All 
threads will be banging on this Map?

Regards the below, who sets forceOnHeapOnly? Rather should we pass in the Cell 
and let the allocator figure where to allocate the memory?

Chunk createChunk(boolean forceOnheapOnly) {

So if no chunk pool, we keep chunk ids in a map elsewhere than in 
MSLABChunkCreateor? Should we do this accounting in one place only?

Otherwise, looking good.










> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926497#comment-15926497
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

The CAS can work similarly to how it works today in MemStoreLABImpl. One thread 
wins the race buy being able to swap (CAS) the null pointer to the pointer to 
chunk (not yet initialized). While the chunk is being initialized the other 
threads may be busy-waiting.

But I have a different suggestion. Initially, I actually thought about 
implementing myself the MemStoreAllocator (that you are calling ChunkCreator). 
So may be you can drop the ChunkCreator from this patch and just commit all the 
Cell related refactoring? I'll continue with the Allocator/Creator and all the 
rest... What do you think?

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926126#comment-15926126
 ] 

Anoop Sam John commented on HBASE-16438:


Not sure then CAS way can work.  Might need a lock unlock way only.  Because ya 
one thread succed in getting CAS success/lock for calling ChunkCreator. It will 
only make the call.. There is a time gap for the actual chunk create work. 
During this time also, the other threads have to wait.  So  ya tryLock () 
unlock way may be needed..  Need to carefully test for the perf implication of 
any such change (Using a JMH)

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925877#comment-15925877
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

I meant to let only one thread from any MSLAB to ask ChunkCreator for the chunk 
allocation. I was talking about CAS (or any other sunchronization) inside 
MSLAB. So this fight remains within MSLAB.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925863#comment-15925863
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


The cyclic reference is with the current code where if we use chunkCreator in 
MSLAB and wrap the pool inside chunkCreator. But i think that can be managed if 
we allow the pool itself to reside inside the ChunkCreator and the HRS also 
will only instantiate the ChunkCreator and internally the chunkCreator creates 
and manages the pool.
For the 2nd one, if we make the CAS inside chunkCreator then all the MSLAB 
should wait for getting a successful chunk. Where is now it is within one MSLAB 
there is a fight. So with in a MSLAB we get a chunk and two threads compete to 
make it as the curChunk.
If we move it to chunkCreator two different threads on two different MSLAB 
itself need to wait for one to get the chunk and then create for other one. Am 
saying for the non-pooled cases.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925785#comment-15925785
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

Hey! 

Here comes the interesting part! :) Pleasure working with you guys! :)

First, can you please explain why cyclic references are needed?

Second, regarding the concurrency within MSLAB. If many threads are requesting 
the creation of the new current chunk, you can let only one thread from any 
MSLAB to ask the ChunkCreator for a new chunk. It can be done using the same 
CAS as you use for chunk initialization. Or using lock() with try_lock() 
semantics. What do you think?

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925512#comment-15925512
 ] 

Anoop Sam John commented on HBASE-16438:


Chunks are being asked by MSLABs and each of the MSLAB at a time need one 
Chunk. That request also (on one MSLAB) raised by different handler threads.  
How can the CAS op happen within ChunkCreator?

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925507#comment-15925507
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


I think her idea is to move all those CAS allocations also inside the 
ChunkCreator when the MSLAB asks for it. That is why I thought it will be a big 
refactoring.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924365#comment-15924365
 ] 

Anoop Sam John commented on HBASE-16438:


No cyclic reference way pls.  
We dont do init (Actual BB allocation) at first step to avoid possible 
locks/synchronized way and do it with CAS.
{code}
if (curChunk.compareAndSet(null, c)) {
// we won race - now we need to actually do the expensive
// allocation step
c.init();
if (pooledChunk) {
  if (!this.closed && !this.pooledChunkQueue.offer(c)) {
if (LOG.isTraceEnabled()) {
  LOG.trace("Chunk queue is full, won't reuse this new chunk. 
Current queue size: "
  + pooledChunkQueue.size());
}
  }
}
return c;
  } else if (pooledChunk) {
chunkPool.putbackChunk(c);
  }
{code}
In multi threaded case, for sure for many threads, the CAS will fail and they 
can go with using the same Chunk allocated via calls from other threads.  If 
the init also happened as part of ChunkCreator's Chunk creation, that will be 
waste of allocating 2 MB memory right?  How u will avoid that as well as 
continue to use CAS way?

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924173#comment-15924173
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

[~ram_krish], thank you so much for considering it! 

If I can help anyhow - let me know!
At least you shouldn't care for CellChunkMap integration :)

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924095#comment-15924095
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


[~anastas]
Since you are very particular in this change and seeing your suggestions I will 
see how can I do the heavy weigh refactoring. Infact I was pretty sure that 
doing as per your suggestion will require lot of refactoring and including the 
cyclic reference. 
Now with your inputs in mind will see if we can do it something better. Wll be 
back here.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924009#comment-15924009
 ] 

Anastasia Braginsky commented on HBASE-16438:
-

Hi [~ram_krish]!

Thank you very much for the updated patch!
I see that you added the tests which is very good!
Are you going to address my other comments that I posted on the RB? (on your 
previous version of the patch)

I want to raise again the issue of the design and responsibilities between 
MemStoreChunkPool and MemStoreChunkCreator.
It is much clearer, understandable and easier to maintain when a class (e.g. 
MemStoreChunkCreator) has a clear responsibility (aka total allocation of the 
chunk including 2MB memory) and it is not spread over other (more than one) 
classes.
If something in this path is going to be changed in the future we have single 
point of maintenance. If there is a bug, it has only single appearance...
If we are already restructuring those relationships, let us make it in a better 
way than it was.

I suggest to make chunk.init() a part of the allocation in the 
MemStoreChunkCreator. So the chunk will leave the MemStoreChunkCreator already 
with the 2MB attached to it (whether from pool or from JVM).
The MemStoreChunkPool should also refer to MemStoreChunkCreator also when 
pre-allocating the chunks ahead of time.
More than that, in MemStoreLABImpl the code of getOrMakeChunk() is quite 
awkward. Chunk is allocated via pool or may be not, and then initiated 10 lines 
below after multiple if-statements. 
I understand that there can be multi-threaded concurrency, but we can deal with 
that. This is what exactly makes it interesting to rewrite this code 
creatively! :)

So if we are already talking about chunk allocation let's make it all much 
smoother...
I suggest to make the MemStoreChunkCreator the single entity responsible for 
chunk allocation AND initialization.
So the entire code-path of chunk allocation is going to be refactored and 
better looking...
What do you think?

Thanks,
Anastasia

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)