[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527759#comment-15527759 ] Hudson commented on HBASE-16134: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1685 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1685/]) HBASE-16134 Introduce Cell extension for server side. (anoopsamjohn: rev b644e0fb8d725b9a09665d7b92544eb32155c689) * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/NoTagsKeyValue.java * (delete) hbase-common/src/main/java/org/apache/hadoop/hbase/Streamable.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/codec/KeyValueCodecWithTags.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/SettableSequenceId.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/SettableTimestamp.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/codec/KeyValueCodec.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/SizeCachedNoTagsKeyValue.java * (add) hbase-common/src/main/java/org/apache/hadoop/hbase/ExtendedCell.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/OffheapKeyValue.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch, HBASE-16134_V3.patch, HBASE-16134_V3.patch, > HBASE-16134_V4.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525598#comment-15525598 ] Anoop Sam John commented on HBASE-16134: Good catch. I did not check the QA results also.. Ya that also shows some fails.. And this is the issue.. The tagsLen is not serialized. Will fix. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch, HBASE-16134_V3.patch, HBASE-16134_V3.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[
https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525591#comment-15525591
]
ramkrishna.s.vasudevan commented on HBASE-16134:
{code}
if (withTags && this.tags != null) {
529 out.write(this.tags);
530 len += this.tags.length;
531 }
532 return len;
533 }
{code}
This tags array already has the tag len in it? Just asking.
Rest LGTM. +1.
> Introduce Cell extension for server side.
> -
>
> Key: HBASE-16134
> URL: https://issues.apache.org/jira/browse/HBASE-16134
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16134.patch, HBASE-16134.patch,
> HBASE-16134_V2.patch, HBASE-16134_V3.patch, HBASE-16134_V3.patch
>
>
> Came after the discussion under HBASE-15721 and HBASE-15879.
> InternalCell is a Cell extension. We do have Cell extensions across different
> interfaces now.
> SettableSeqId
> SettableTimestamp
> Streamable.
> And demand for this keep growing.
> So let us include everything into one Interface.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[
https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522709#comment-15522709
]
Hadoop QA commented on HBASE-16134:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 43s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 33s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 39s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 43s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 31s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 117m 14s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.security.visibility.TestVisibilityLabelReplicationWithExpAsString |
| | hadoop.hbase.client.TestBlockEvictionFromClient |
| | hadoop.hbase.security.visibility.TestVisibilityLabelsReplication |
| |
hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService
|
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide3 |
| | org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor |
| | org.apache.hadoop.hbase.filter.TestMultiRowRangeFilter |
| | org.apache.hadoop.hbase.client.TestHCM |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830266/HBASE-16134_V3.patch |
| JIRA Issue | HBASE-16134 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 72220ae16748 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[
https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522410#comment-15522410
]
Hadoop QA commented on HBASE-16134:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 30s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 50s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 33s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 14s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 51s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 29s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 55s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestSerialization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830258/HBASE-16134_V3.patch |
| JIRA Issue | HBASE-16134 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 9158abe5832c 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / da37fd9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3714/artifact/patchprocess/branch-findbugs-hbase-common-warnings.html
|
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/3714/artifact/patchprocess/whitespace-eol.txt
|
| unit |
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522194#comment-15522194 ] Anoop Sam John commented on HBASE-16134: Am doing a bit of change on the latest patch. U can see, now the write(OS) method in Cell will write the total length (4 bytes) followed by the cell itself. Write of this total length int should be done by the Codec itself. It is upto the Encoder and decoder to know how they want to pass the total cell length. Right now we dont use this write(OS) method while write the cells to HFiles. We can infact use that. Only diff is in case of write to HFiles, we dont write total length of cell where as in case of RPC and WAL, we will first write total length int. So my plan is to add an API to retrieve the total cell length also along with write(OS). The codec uses this API and writes the int to OS and then call Cell#write().. The write impl will just write the cell as such into the OS. Will post the patch today. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522184#comment-15522184 ] Anoop Sam John commented on HBASE-16134: bq.How you think it will work when a Cell has system and user tags in it. Will there be filtering out of system tags when we return to client? In case of super user, as we said, all tags will get send back. In normal user case, we will filter out system tags but the user tags will get serialized back to client. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519918#comment-15519918 ] stack commented on HBASE-16134: --- Sounds good. In a separate issue let us address tags current circumstance where we have to pass booleans everywhere. How you think it will work when a Cell has system and user tags in it. Will there be filtering out of system tags when we return to client? +1 on current patch. Has a bunch of value aggregating the myriad Interfaces we have server-side all up into one. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518793#comment-15518793 ] Anoop Sam John commented on HBASE-16134: Coming back to this. The Codec based strip of tags was done as a temp solution not to pass the critical system tags from server back to client. This also imposes the limitation that Tags can not be used by users. Tags are a system side feature alone. In the past there were some Qs in user@ for using custom tags. Now before 2.0 release we should fix this. We should allow users to set tags on Cell and pass them while write. Also these custom tags must be returned back to users (Irrespective of codec and all). The system tags (like ACL, visibility) should not get transferred btw client and server. And when the client is run by a super user, we should pass all tags (including system tags). This way we can make sure that all tags are passed while replication and also tool like Export gets all tags. I have some idea on this and plan to do this sooner. Now for this issue as such, pls let me add write API with boolean param withTags. The KVCodec will pass false and KVCodecWithTags will pass true. Once the tag related IA is done, we can get rid of this boolean. How is that [~saint@gmail.com]? Will post a new patch soon. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507183#comment-15507183 ] Anoop Sam John commented on HBASE-16134: Replicator reads Cells from WAL. bq.Do we create a Cell from MemSLAB when we send Cell back over RPC? No. During write to memstore, we copy to MSLAB and create a new Cell object around it and keep in CSLM. When read, we return back the same object. Or else we will have to make many garbage. bq.Nah. This is just boolean in another form. Ya.. Just asking whether this will be a bit better. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507150#comment-15507150 ] stack commented on HBASE-16134: --- Replication opens a new Reader, right, each time it replicates? bq. How MemSLAB can help? Do we create a Cell from MemSLAB when we send Cell back over RPC? bq. Will it be better if we have 2 methods in ExtendedCell ie. write(OS) and writeNoTags(OS)? Nah. This is just boolean in another form. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507134#comment-15507134 ] Anoop Sam John commented on HBASE-16134: #1 For replication we use this Codec. Also there is a config which can be used to specify the Codec to be used. So one can change to this Codec. #2 It is not Reader which create Cells by HFileScanner - Ya we will have new scanner object for the compaction read and that knows whether this is a compaction scan (Already it knows). That is kind of ok bq.Can MemSlab work here? Sorry did not get. How MemSLAB can help? bq.Just trying to have it so Tags are first-class. We just have to solve the issue that stops Cells being put on the RPC. A flag is easy. Just trying to explore if any other possibility. Understood.. Ya agree it is bit ugly. Will it be better if we have 2 methods in ExtendedCell ie. write(OS) and writeNoTags(OS)? > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507094#comment-15507094 ] stack commented on HBASE-16134: --- On #1, yeah, when is that Codec used? On #2, open a new Reader? One that includes Tags? On read from memstore, that is a problem. Can MemSlab work here? Just trying to have it so Tags are first-class. We just have to solve the issue that stops Cells being put on the RPC. A flag is easy. Just trying to explore if any other possibility. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507049#comment-15507049 ] Anoop Sam John commented on HBASE-16134: If we have to avoid passing a boolean 'withTags' we may have to either have 2 methods (write, writeNoTags) or make the agreement of the write method such that it will write key and value but no tags. And the KVCodecWithTags will write the tags on its own. I dont feel the latter is good. Tags are part of Cell only > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507015#comment-15507015 ] ramkrishna.s.vasudevan commented on HBASE-16134: We need tags for compaction and also those tags are needed for visibility and access controller right. Ya we don't emit but still we need those details. Different cells for read and write wrt to tags won't work out in all cases. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506978#comment-15506978 ] Anoop Sam John commented on HBASE-16134: 2 issues with this 1. We want to emit tags in any read time when the KVCodecWithTags is configured. Export tool when doing this read, we have to emit tags or else we will loose it. May be this area we need to see whether we have documented this correctly or not 2. WAL and write time go with diff Cell and read time diff.. The issue comes with compaction. Here there is read and write. We pass the read cells (From an HFile) to writer. May be we should decide based on isCompaction boolean. This means the Cells what we add to memstore (Those will go to writer finally) will need to have a logic which will write tags. So what abt this same cells when read. The scan can read cells from memstore also.. So we will end up in copying those to a new Cell of type which wont write tags? Sorry boss.. I fear this wont work out :-( > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506953#comment-15506953 ] stack commented on HBASE-16134: --- When do we ever want to emit tags? When debugging? WAL write time is 'write time' so when Cell comes in the front door, we use a write-time Cell that will include tags when we call write. Otherwise, when we pull from hfile, we use a Cell type that does not include tags? If debug, pass a flag and then use a different Cell type, one that will do debug things including emitting Tags? > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505902#comment-15505902 ] Anoop Sam John commented on HBASE-16134: Yes as u said the case is when cell is having tags but the RPC layer dont want to serialize it. So it uses a Codec which tries to strip the tags. The Codec trying to call single write method in Cell to do the write to OS. So it has to have a way to instruct the method that dont write tags. bq.When is this Cell created? It is created at read time only? Can we create a Cell at read-time that is of a type that never includes tags when it serializes? I was exactly thinking in this direction from today morning. :-) But we cannot do that is what I got as ans Cell is created way down the read layer. ie. in HFileScanner. At that layer we have no idea of Codec/ RPC/ User and it should not also. So we can not create a special Cell at this level which knows it should do the Tags truncate while write to OS. The only time the Cell will be asked to seriailze is when it is writing on the RPC to send back to the client? Are there other times? It is at RPC side as well as WAL write time. Now we dont use but I believe we can use this Cell write method while we flush also. (HFileWriter flow). > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505832#comment-15505832 ] stack commented on HBASE-16134: --- So, if no Tags, we don't serialize any out. We are talking about the case where a Cell has tags and it is asked to serialize itself. We want option of not serializing tags. When is this Cell created? It is created at read time only? Can we create a Cell at read-time that is of a type that never includes tags when it serializes? The only time the Cell will be asked to seriailze is when it is writing on the RPC to send back to the client? Are there other times? > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[
https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503954#comment-15503954
]
Hadoop QA commented on HBASE-16134:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 42s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 53s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 38s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12814579/HBASE-16134_V2.patch |
| JIRA Issue | HBASE-16134 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a22675a2544b 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c5b8aab |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3603/artifact/patchprocess/branch-findbugs-hbase-common-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3603/testReport/ |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3603/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Introduce Cell extension for server side.
> -
>
> Key: HBASE-16134
> URL: https://issues.apache.org/jira/browse/HBASE-16134
> Project: HBase
>
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503951#comment-15503951 ] Anoop Sam John commented on HBASE-16134: [~saint@gmail.com] Forgot abt this jira for some time.. Coming back to this as am working on subtasks for the parent jira. We do have diff Codec impls which us used at RPC edge. One for writing Tags and one for filtering. We can use those as per the need. Here the method write(OutputStream) is implemented in a Cell impl class. (Like KeyValue). The method contract is to write this Cell bytes in a KV serialization format. Codec just call this method on Cell object and wont do getXXXLength(), getXXXArray() and write individual parts one by one. Means this method impl in Cell objects has to know how to write itself in KV format and what all to write. In case of Codec which needs tags filter, it has to write . Where as when the Codec is the one which need tags the method has to write ..So the point is how to make the method know whether it has to write the tag or not. For this reason only the method was taking a boolean param. Or else how we will do this? We dont want the Codec to call diff getters on Cell and write each part by part. We want to do this in one shot. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
