[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13940317#comment-13940317 ] haosdent commented on HBASE-10648: -- well done! [~carp84] > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648-0.94_v2.patch, > HBASE-10648-0.94_v3.patch, HBASE-10648.patch, HBASE-10648_V2.patch, > HBASE-10648_V3.patch, HBASE-10648_V4.patch, HBASE-10648_V5.patch, > HBASE-10648_V6.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13939439#comment-13939439 ] Lars Hofhansl commented on HBASE-10648: --- Looks good in principle. Doesn't change the behavior by default, so I have no problem with this. Nit: There are some references to Cells in some of the comments. Cells do not exist in 0.94 so we should change this. Two issues: # For this to be 0.94 it has to be in 0.96 and 0.98 as well. So [~stack] and [~apurtell] would have to agree as well. # I wonder if that breaks some users of coprocessors. [~giacomotaylor], mind having a quick lock to see if that break anything for Phoenix? I don't think unless Phoenix manually allocates a memstore, but should check. > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648-0.94_v2.patch, > HBASE-10648.patch, HBASE-10648_V2.patch, HBASE-10648_V3.patch, > HBASE-10648_V4.patch, HBASE-10648_V5.patch, HBASE-10648_V6.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935907#comment-13935907 ] Hudson commented on HBASE-10648: FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #118 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/118/]) HBASE-10648 Pluggable Memstore.(Anoop) (anoopsamjohn: rev 1577541) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFlusher.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFlusher.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch, > HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, > HBASE-10648_V5.patch, HBASE-10648_V6.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935337#comment-13935337 ] Hudson commented on HBASE-10648: FAILURE: Integrated in HBase-TRUNK #5010 (See [https://builds.apache.org/job/HBase-TRUNK/5010/]) HBASE-10648 Pluggable Memstore.(Anoop) (anoopsamjohn: rev 1577541) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFlusher.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFlusher.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch, > HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, > HBASE-10648_V5.patch, HBASE-10648_V6.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935079#comment-13935079
]
Yu Li commented on HBASE-10648:
---
Hi [~anoop.hbase],
{quote}
It won't be good to directly use some impl here. We use flushed variable for
logging purpose only right? In trunk it is. Can you use the size available in
MemStoreSnapshot?
{quote}
Yes, the flushedSize is for logging purpose, or say monitoring purpose.
Actually this is kind of a necessary metrics for our monitoring system and we
think it useful for analyzing online problem when there's any.
>From the code segment of the heapSizeChange method, it's for computing the
>heapsize change when adding/removing a previously-non-existing kv, so I don't
>think any size available in MemStoreSnapshot can be used here. If we decided
>to keep the flushedSize metrics, I think it's even better to put
>heapSizeChange method into the MemStore interface, since different impl of
>MemStore might have different heap size to store one kv, depending on the
>structure(not only CSLM) we use to store it. What's your opinion?
{code}
static long heapSizeChange(final KeyValue kv, final boolean notpresent) {
return notpresent ?
ClassSize.align(ClassSize.CONCURRENT_SKIPLISTMAP_ENTRY + kv.heapSize()):
0;
}
{code}
{quote}
Make snapshotSize = keySize().
Any way once HBASE-10514 is in to 94, will need some change here and
snapshotSize will come as an instance member.
{quote}
Make sense. Let me make the change in one go after HBASE-10514 goes into 94.
> Pluggable Memstore
> --
>
> Key: HBASE-10648
> URL: https://issues.apache.org/jira/browse/HBASE-10648
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.99.0
>
> Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch,
> HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch,
> HBASE-10648_V5.patch, HBASE-10648_V6.patch
>
>
> Make Memstore into an interface implementation. Also make it pluggable by
> configuring the FQCN of the impl.
> This will allow us to have different impl and optimizations in the Memstore
> DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935055#comment-13935055 ] Anoop Sam John commented on HBASE-10648: Committed to trunk. Thanks for the reviews. > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch, > HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, > HBASE-10648_V5.patch, HBASE-10648_V6.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935014#comment-13935014 ] Anoop Sam John commented on HBASE-10648: Test fail seems not related. Going to commit in some time. > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch, > HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, > HBASE-10648_V5.patch, HBASE-10648_V6.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934931#comment-13934931
]
Hadoop QA commented on HBASE-10648:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634686/HBASE-10648_V6.patch
against trunk revision .
ATTACHMENT ID: 12634686
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 26 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnDatanodeDeath(TestLogRolling.java:368)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//console
This message is automatically generated.
> Pluggable Memstore
> --
>
> Key: HBASE-10648
> URL: https://issues.apache.org/jira/browse/HBASE-10648
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch,
> HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch,
> HBASE-10648_V5.patch, HBASE-10648_V6.patch
>
>
> Make Memstore into an interface implementation. Also make it pluggable by
> configuring the FQCN of the impl.
> This will allow us to have different impl and optimizations in the Memstore
> DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934850#comment-13934850
]
Anoop Sam John commented on HBASE-10648:
Gone through the 94 patch. Looks good.
{code}
-flushed += this.memstore.heapSizeChange(kv, true);
+flushed += DefaultMemStore.heapSizeChange(kv, true);
{code}
It won't be good to directly use some impl here. We use flushed variable for
logging purpose only right (?) In trunk it is. Can you use the size available
in MemStoreSnapshot (?)
DefaultMemStore#snapshot()
{code}
+ long snapshotSize = 0;
...
return new MemStoreSnapshot(this.snapshotId, snapshot.size(),
+ snapshotSize, this.snapshotTimeRangeTracker,
+ new CollectionBackedScanner(snapshot, this.comparator));
{code}
Make snapshotSize = keySize().
Any way once HBASE-10514 is in to 94, will need some change here and
snapshotSize will come as an instance member.
> Pluggable Memstore
> --
>
> Key: HBASE-10648
> URL: https://issues.apache.org/jira/browse/HBASE-10648
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch,
> HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch,
> HBASE-10648_V5.patch, HBASE-10648_V6.patch
>
>
> Make Memstore into an interface implementation. Also make it pluggable by
> configuring the FQCN of the impl.
> This will allow us to have different impl and optimizations in the Memstore
> DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934846#comment-13934846
]
Hadoop QA commented on HBASE-10648:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634670/HBASE-10648_V6.patch
against trunk revision .
ATTACHMENT ID: 12634670
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 26 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnDatanodeDeath(TestLogRolling.java:368)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//console
This message is automatically generated.
> Pluggable Memstore
> --
>
> Key: HBASE-10648
> URL: https://issues.apache.org/jira/browse/HBASE-10648
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch,
> HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch,
> HBASE-10648_V5.patch, HBASE-10648_V6.patch
>
>
> Make Memstore into an interface implementation. Also make it pluggable by
> configuring the FQCN of the impl.
> This will allow us to have different impl and optimizations in the Memstore
> DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934844#comment-13934844 ] Anoop Sam John commented on HBASE-10648: bq.Sorry but what's preventing the change to go into 94?The CSLM issue (when there are too many KVs in it) also exists in v94, and we also want to get relieved from that Of course I'm not expecting the whole HBASE-10191 solution go into 94, but after making Memstore impl pluggable, at least in-memory flush impl would fit well in 0.94, right? Ping [~larsh] abt the decision. > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch, > HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, > HBASE-10648_V5.patch, HBASE-10648_V6.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934833#comment-13934833
]
Yu Li commented on HBASE-10648:
---
{quote}
Sorry I will need some more time to post a patch for HBASE-10713
{quote}
I see, it's ok, thanks for the update. In this case I might start writing some
code by myself (you know, the deadline thing :-)), and I'll appreciate a lot if
you can send your drafted patch to me through email, before it's ready for
public. Just want to avoid duplicated work as much as possible.
{quote}
BTW I am not sure whether this kind of a change can go into an older version
like 94
{quote}
Sorry but what's preventing the change to go into 94? The CSLM issue (when
there are too many KVs in it) also exists in v94, and we also want to get
relieved from that:-) Of course I'm not expecting the whole HBASE-10191
solution go into 94, but after making Memstore impl pluggable, at least
in-memory flush impl would fit well in 0.94, right?
{quote}
You want to internally maintain that?
{quote}
It's not an easy decision to upgrade to a new version in product env, and I'm
afraid we'll stay on 94 for some while, so yes we will maintain this
internally. But of course I'll try to contribute back to 94 branch after fully
test, so others using 94 can also benefit.
> Pluggable Memstore
> --
>
> Key: HBASE-10648
> URL: https://issues.apache.org/jira/browse/HBASE-10648
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch,
> HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch,
> HBASE-10648_V5.patch, HBASE-10648_V6.patch
>
>
> Make Memstore into an interface implementation. Also make it pluggable by
> configuring the FQCN of the impl.
> This will allow us to have different impl and optimizations in the Memstore
> DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934819#comment-13934819
]
Hadoop QA commented on HBASE-10648:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634676/HBASE-10648-0.94_v1.patch
against trunk revision .
ATTACHMENT ID: 12634676
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8991//console
This message is automatically generated.
> Pluggable Memstore
> --
>
> Key: HBASE-10648
> URL: https://issues.apache.org/jira/browse/HBASE-10648
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch,
> HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch,
> HBASE-10648_V5.patch, HBASE-10648_V6.patch
>
>
> Make Memstore into an interface implementation. Also make it pluggable by
> configuring the FQCN of the impl.
> This will allow us to have different impl and optimizations in the Memstore
> DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934806#comment-13934806 ] Anoop Sam John commented on HBASE-10648: Sorry I will need some more time to post a patch for HBASE-10713. Before that I want to make the MsLAB also as pluggable. (On heap or off heap impls can come in) This change I am doing now. BTW I am not sure whether this kind of a change can go into an older version like 94. You want to internally maintain that (?) Will review the attached patch any way. > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch, > HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, > HBASE-10648_V5.patch, HBASE-10648_V6.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934691#comment-13934691
]
Hadoop QA commented on HBASE-10648:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634649/HBASE-10648_V5.patch
against trunk revision .
ATTACHMENT ID: 12634649
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 26 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ KeyValue.COMPARATOR.compareRows(left.getRowArray(),
left.getRowOffset(), (int) left.getRowLength(), row1, 0, row1.length) == 0);
+System.out.println(iteration + " took usec: " + (System.nanoTime() -
nanos)/1000 + " for: " + cnt);
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.client.TestHCM
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnDatanodeDeath(TestLogRolling.java:368)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//console
This message is automatically generated.
> Pluggable Memstore
> --
>
> Key: HBASE-10648
> URL: https://issues.apache.org/jira/browse/HBASE-10648
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-10648.patch, HBASE-10648_V2.patch,
> HBASE-10648_V3.patch, HBASE-10648_V4.patch, HBASE-10648_V5.patch
>
>
> Make Memstore into an interface implementation. Also make it pluggable by
> configuring the FQCN of the impl.
> This will allow us to have different impl and optimizations in the Memstore
> DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932786#comment-13932786 ] Anoop Sam John commented on HBASE-10648: May be better I will wait for ur commit (HBASE-10514) > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Attachments: HBASE-10648.patch, HBASE-10648_V2.patch, > HBASE-10648_V3.patch, HBASE-10648_V4.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932784#comment-13932784 ] Anoop Sam John commented on HBASE-10648: bq.You looking for more compliments? Don't be greedy now... Hey no boss.. ;) The SnapshotInfo prt comment is reworked and related changes. So thought you will be interested in reviewing those changes. Thanks for the review Stack. Will commit later today IST. Before that if HBASE-10514 can go in, I will make the necessary change in this patch also. > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Attachments: HBASE-10648.patch, HBASE-10648_V2.patch, > HBASE-10648_V3.patch, HBASE-10648_V4.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932424#comment-13932424 ] stack commented on HBASE-10648: --- [~anoop.hbase] Whats up? I already said lgtm where the 'g' means great. You looking for more compliments? Don't be greedy now... > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Attachments: HBASE-10648.patch, HBASE-10648_V2.patch, > HBASE-10648_V3.patch, HBASE-10648_V4.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13931418#comment-13931418 ] Anoop Sam John commented on HBASE-10648: Ping [~stack], [~apurtell] . > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Attachments: HBASE-10648.patch, HBASE-10648_V2.patch, > HBASE-10648_V3.patch, HBASE-10648_V4.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13930980#comment-13930980 ] Ted Yu commented on HBASE-10648: +1 on v4 > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Attachments: HBASE-10648.patch, HBASE-10648_V2.patch, > HBASE-10648_V3.patch, HBASE-10648_V4.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925716#comment-13925716
]
Hadoop QA commented on HBASE-10648:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12633671/HBASE-10648_V4.patch
against trunk revision .
ATTACHMENT ID: 12633671
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 21 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8937//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8937//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8937//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8937//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8937//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8937//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8937//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8937//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8937//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8937//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8937//console
This message is automatically generated.
> Pluggable Memstore
> --
>
> Key: HBASE-10648
> URL: https://issues.apache.org/jira/browse/HBASE-10648
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-10648.patch, HBASE-10648_V2.patch,
> HBASE-10648_V3.patch, HBASE-10648_V4.patch
>
>
> Make Memstore into an interface implementation. Also make it pluggable by
> configuring the FQCN of the impl.
> This will allow us to have different impl and optimizations in the Memstore
> DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925191#comment-13925191 ] Anoop Sam John commented on HBASE-10648: I will give the reworked patch soon. bq.When you start to do the second implementation, you will have a better idea of what changes are needed in the Interface. I am already in the process of doing that.. Till now all looks good. Ya as the need comes we can refine the interface more. Any way this is a Private internal interface and we can change as per the needs. > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Attachments: HBASE-10648.patch, HBASE-10648_V2.patch, > HBASE-10648_V3.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925003#comment-13925003 ] stack commented on HBASE-10648: --- All of the above is good by me Anoop including the bit about more than one snapshot. Yeah, current implementation only allows one. I was thinking of an implementation that might carry N ... but this fact can be contained inside the actual implementation. It does not need to escape. In fact, it could be argued that this 'snapshotting' is an implementation detail of the default memstore and it should really be more generic... prepareForFlush instead of snapshot but we can do this in later issues. Important thing is make what we have is pluggable. When you start to do the second implementation, you will have a better idea of what changes are needed in the Interface. > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Attachments: HBASE-10648.patch, HBASE-10648_V2.patch, > HBASE-10648_V3.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924710#comment-13924710
]
Anoop Sam John commented on HBASE-10648:
Thanks Stack for the review and comments.
bq.Should it be CellScanner?
We have to create StoreScanner from the scanner on snapshot. So this can not be
a CellScanner. We will need changes in many upper layers. May be later when
we make changes of KV- Cell in write later, we can think of higher level changes
bq.What happens if a MemStore implementation has N snapshots?
Can it be?
{code}
void snapshot() {
// If snapshot currently has entries, then flusher failed or didn't call
// cleanup. Log a warning.
if (!this.snapshot.isEmpty()) {
LOG.warn("Snapshot called again without clearing previous. " +
"Doing nothing. Another ongoing flush or did we fail last attempt?");
}
{code}
We dont take a new snapshot.
bq.Is the snapshot info a good idea? We need to pass it alongside a scanner?
Should the two be combined; a class that implements CellScanner and from which
you can get snapshot info? You ask SnapshotInfo for the CellScanner when you
need one?
Will do some changes here. Snapshot Scanner "has a" snapshot info ? Or it may
be better we have a Snapshot class which "has a" snapshotScanner and "has" some
info. Info includes cellsCount, size, TimeRangeTracker etc.. The snapshot()
call creates a snapshot instance and there needs to be a getSnapshot() this
returns a Snapshot which has all the things abt the snapshot. Sounds
bq.We need to pass it alongside a scanner?
So we pass the Snapshot. Get scanner or info wherever needed.
bq.And oh, it took me a while... yeah, a class in an Interface as per Andrew is
ahem, 'unorthodox' ... smile. It should be an Interface too?
Fine. I will move out the class from the interface. What I thought was this
SnapshotInfo is relevant only within the MemStore scope and not a top level
thing. This is a POJO and so an interface might not be needed. Now as per the
above comment fix there wont be some thing like SnapshotInfo class at all.
bq.Can this be about Cells rather than KVs since you are making a change:
You mean in TestHRegion.java ? Yes it can be
bq.WHy does this go away? testUpsertMSLAB
This was testing with usage of MemStore#updateColumnValue() This and
HStore#updateColumnValue() were used only in tests and I could see a TODO for
removal. We still need to keep this? If so it can be added to the interface also
bq.Is MSLAB an implementation detail on DefaultMemStore?
Kind of. It might not to be added in MemStore interface. But this MSLAB also
will be extracted to an interface impl way. The default impl will be on heap
MSLAB. May be some one can change to OffHeap MSLAB and plugin to
DefaultMemStore. Andy was also having this same question/suggestion. But
this will not be in this Jira but in a follow on issue :)
bq.MemStore#size is not heapSize or is? FYI, HBASE-10514 adds a method to
MemStore.
For DefaultMemStore both are same. But the size is supposed to return the size
of the memory the MemStore takes. (Whether on heap or off heap) The heapSize
is on heap size alone. After this patch, the heapSize() as such not having
much use. It is used in HRegion#heapSize() and this is not getting used any
where above the line. Oh let me see the patch for HBASE-10514. Any changes
as per that I can do the necessary change in this patch also (Once it goes in)
> Pluggable Memstore
> --
>
> Key: HBASE-10648
> URL: https://issues.apache.org/jira/browse/HBASE-10648
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-10648.patch, HBASE-10648_V2.patch,
> HBASE-10648_V3.patch
>
>
> Make Memstore into an interface implementation. Also make it pluggable by
> configuring the FQCN of the impl.
> This will allow us to have different impl and optimizations in the Memstore
> DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924402#comment-13924402
]
stack commented on HBASE-10648:
---
bq. KeyValueScanner snapshotScanner
Should it be CellScanner?
MemStore#size is not heapSize or is? FYI, HBASE-10514 adds a method to
MemStore.
Is the snapshot info a good idea? We need to pass it alongside a scanner?
Should the two be combined; a class that implements CellScanner and from which
you can get snapshot info? You ask SnapshotInfo for the CellScanner when you
need one? What happens if a MemStore implementation has N snapshots? This
scanner + info implementation will allow hiding these details? (It should be
able to...).
And oh, it took me a while... yeah, a class in an Interface as per Andrew is
ahem, 'unorthodox' ... smile. It should be an Interface too?
Can this be about Cells rather than KVs since you are making a change:
- for (KeyValue kv : ((HStore) region.getStore(fam1)).memstore.kvset) {
+ for (KeyValue kv : memstore.kvset) {
WHy does this go away? testUpsertMSLAB
Is MSLAB an implementation detail on DefaultMemStore?
Patch lgtm where the 'g' in the previous is 'great', not 'good'.
> Pluggable Memstore
> --
>
> Key: HBASE-10648
> URL: https://issues.apache.org/jira/browse/HBASE-10648
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-10648.patch, HBASE-10648_V2.patch,
> HBASE-10648_V3.patch
>
>
> Make Memstore into an interface implementation. Also make it pluggable by
> configuring the FQCN of the impl.
> This will allow us to have different impl and optimizations in the Memstore
> DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923882#comment-13923882
]
Hadoop QA commented on HBASE-10648:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12633365/HBASE-10648_V3.patch
against trunk revision .
ATTACHMENT ID: 12633365
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 22 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAdmin
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8922//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8922//console
This message is automatically generated.
> Pluggable Memstore
> --
>
> Key: HBASE-10648
> URL: https://issues.apache.org/jira/browse/HBASE-10648
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-10648.patch, HBASE-10648_V2.patch,
> HBASE-10648_V3.patch
>
>
> Make Memstore into an interface implementation. Also make it pluggable by
> configuring the FQCN of the impl.
> This will allow us to have different impl and optimizations in the Memstore
> DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923806#comment-13923806
]
Hadoop QA commented on HBASE-10648:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12633329/HBASE-10648_V2.patch
against trunk revision .
ATTACHMENT ID: 12633329
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 22 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 5
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnDatanodeDeath(TestLogRolling.java:354)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8920//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8920//console
This message is automatically generated.
> Pluggable Memstore
> --
>
> Key: HBASE-10648
> URL: https://issues.apache.org/jira/browse/HBASE-10648
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-10648.patch, HBASE-10648_V2.patch
>
>
> Make Memstore into an interface implementation. Also make it pluggable by
> configuring the FQCN of the impl.
> This will allow us to have different impl and optimizations in the Memstore
> DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923576#comment-13923576 ] Anoop Sam John commented on HBASE-10648: Sure I can add those java docs also. Thanks Ted. Will give updated patch. > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Attachments: HBASE-10648.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13921802#comment-13921802
]
Ted Yu commented on HBASE-10648:
There're a few methods in MemStore interface which lack javadoc. e.g.:
{code}
+ long timeOfOldestEdit();
...
+ void getRowKeyAtOrBefore(final GetClosestRowBeforeTracker state);
{code}
> Pluggable Memstore
> --
>
> Key: HBASE-10648
> URL: https://issues.apache.org/jira/browse/HBASE-10648
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-10648.patch
>
>
> Make Memstore into an interface implementation. Also make it pluggable by
> configuring the FQCN of the impl.
> This will allow us to have different impl and optimizations in the Memstore
> DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13919117#comment-13919117
]
Andrew Purtell commented on HBASE-10648:
Static class in an interface is weird? Just me?
{code}
+public interface MemStore extends HeapSize {
...
static class SnapshotInfo {
...
{code}
Can we extract the MemstoreLAB to an interface and default implementation also?
Maybe as a follow on issue?
> Pluggable Memstore
> --
>
> Key: HBASE-10648
> URL: https://issues.apache.org/jira/browse/HBASE-10648
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-10648.patch
>
>
> Make Memstore into an interface implementation. Also make it pluggable by
> configuring the FQCN of the impl.
> This will allow us to have different impl and optimizations in the Memstore
> DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918996#comment-13918996 ] Anoop Sam John commented on HBASE-10648: bq.Should there be an interface for Memstore so that the default implementation and other implementations have a common contract ? Yes. We have an interface now in the patch. The main changes in this patch 1. Removed updateColumnValue() from HStore and Memstore. This was used only in tests and there is a TODO for removal 2. Extracted MemStore as an interface and current Memstore impl moved to an DefaultMemStore impl. 3. For dealing with flush, we had snapshot(), getSnapshot() and clearSnapshot(). This is changed to snapshot() , getSnapshotScanner() and clearSnapshot(). getSnapshotScanner() hides the snapshot DS details within MemStore. 4. MemStore impl can be plugged in by configuring the FQCN against "hbase.regionserver.memstore.class" > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Attachments: HBASE-10648.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918336#comment-13918336
]
Hadoop QA commented on HBASE-10648:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632288/HBASE-10648.patch
against trunk revision .
ATTACHMENT ID: 12632288
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 22 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 5
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8867//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8867//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8867//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8867//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8867//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8867//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8867//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8867//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8867//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8867//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8867//console
This message is automatically generated.
> Pluggable Memstore
> --
>
> Key: HBASE-10648
> URL: https://issues.apache.org/jira/browse/HBASE-10648
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-10648.patch
>
>
> Make Memstore into an interface implementation. Also make it pluggable by
> configuring the FQCN of the impl.
> This will allow us to have different impl and optimizations in the Memstore
> DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918288#comment-13918288 ] Ted Yu commented on HBASE-10648: Should there be an interface for Memstore so that the default implementation and other implementations have a common contract ? > Pluggable Memstore > -- > > Key: HBASE-10648 > URL: https://issues.apache.org/jira/browse/HBASE-10648 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Attachments: HBASE-10648.patch > > > Make Memstore into an interface implementation. Also make it pluggable by > configuring the FQCN of the impl. > This will allow us to have different impl and optimizations in the Memstore > DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
