[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706396#comment-15706396
]
Hudson commented on HBASE-17012:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2042 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2042/])
HBASE-17012 Handle Offheap cells in CompressedKvEncoder (Ram) (ramkrishna: rev
7c43a23c07d2af1c236b3153ba932234c3a80d13)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestRpcServer.java
* (edit)
hbase-common/src/main/java/org/apache/hadoop/hbase/io/TagCompressionContext.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SecureWALCellCodec.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALCellCodecWithCompression.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALReaderOnSecureWAL.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java
* (edit)
hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/Dictionary.java
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch, HBASE-17012_2.patch,
> HBASE-17012_3.patch, HBASE-17012_4.patch, HBASE-17012_5.patch,
> HBASE-17012_6.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705862#comment-15705862
]
ramkrishna.s.vasudevan commented on HBASE-17012:
Thanks for the review Anoop.
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch, HBASE-17012_2.patch,
> HBASE-17012_3.patch, HBASE-17012_4.patch, HBASE-17012_5.patch,
> HBASE-17012_6.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705105#comment-15705105
]
Anoop Sam John commented on HBASE-17012:
Added some minor comments in RB.
+1. Can fix those on commit.
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch, HBASE-17012_2.patch,
> HBASE-17012_3.patch, HBASE-17012_4.patch, HBASE-17012_5.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695634#comment-15695634
]
Hadoop QA commented on HBASE-17012:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 41s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 4m 24s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 49s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 40s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 88m 23s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 156m 20s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840481/HBASE-17012_4.patch |
| JIRA Issue | HBASE-17012 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux b166c0a9a555 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 86e1785 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4623/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4623/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Handle Offheap cells in CompressedKvEncoder
>
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695281#comment-15695281
]
ramkrishna.s.vasudevan commented on HBASE-17012:
https://reviews.apache.org/r/54075/. Added to RB.
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch, HBASE-17012_2.patch,
> HBASE-17012_3.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695107#comment-15695107
]
ramkrishna.s.vasudevan commented on HBASE-17012:
Ok. I think some recent commit caused compilation error. I need to take update
locally also in my local setup.
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch, HBASE-17012_2.patch,
> HBASE-17012_3.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695092#comment-15695092
]
Anoop Sam John commented on HBASE-17012:
To RB pls.. better to see diff and add comments..
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch, HBASE-17012_2.patch,
> HBASE-17012_3.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695058#comment-15695058
]
Hadoop QA commented on HBASE-17012:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 23s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 23s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 23s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 0s
{color} | {color:red} The patch causes 68 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 1s
{color} | {color:red} The patch causes 68 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 3s
{color} | {color:red} The patch causes 68 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 6s
{color} | {color:red} The patch causes 68 errors with Hadoop v2.6.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 7s
{color} | {color:red} The patch causes 68 errors with Hadoop v2.6.5. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 10s
{color} | {color:red} The patch causes 68 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 13s
{color} | {color:red} The patch causes 68 errors with Hadoop v2.7.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 16s
{color} | {color:red} The patch causes 68 errors with Hadoop v2.7.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 17s
{color} | {color:red} The patch causes 68 errors with Hadoop v3.0.0-alpha1.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 21s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 28s
{color} | {color:red} hbase-server generated 2 new + 1 unchanged - 0 fixed = 3
total (was 1) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 42s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 23s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} |
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15693725#comment-15693725
]
Anoop Sam John commented on HBASE-17012:
Should be min addition in CellUtil and no stuff like the Dict/Compress kind of
work. eg: compressCellForWal
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch, HBASE-17012_2.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15693659#comment-15693659
]
ramkrishna.s.vasudevan commented on HBASE-17012:
I think better not to add anything to CellUtil.
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch, HBASE-17012_2.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15693652#comment-15693652
]
ramkrishna.s.vasudevan commented on HBASE-17012:
Changing to DataOS is fine I think but I need to see.
The reason why I added a TODO is because I thought we can add a WALCellCodec
Util or something like that and just add all such APIs to it? Anyway there is
no need to expose in CellUtil.
That would make things easier?
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch, HBASE-17012_2.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15693567#comment-15693567
]
Anoop Sam John commented on HBASE-17012:
Did not read the patch fully.. First level comment
{code}
// TODO : Should this be moved to CellUtil? seems very specific to SecureWAL.
212 if (cell instanceof ByteBufferedCell) {
213 bbwos.write(((ByteBufferedCell) cell).getRowByteBuffer(),
214 ((ByteBufferedCell) cell).getRowPosition(),
cell.getRowLength());
215 } else {
216 bbwos.write(cell.getRowArray(), cell.getRowOffset(),
cell.getRowLength());
217 }
StreamUtils.writeRawVInt32(bbwos, cell.getFamilyLength());
219 if (cell instanceof ByteBufferedCell) {
220 bbwos.write(((ByteBufferedCell) cell).getFamilyByteBuffer(),
221 ((ByteBufferedCell) cell).getFamilyPosition(),
cell.getFamilyLength());
222 } else {
223 bbwos.write(cell.getFamilyArray(), cell.getFamilyOffset(),
cell.getFamilyLength());
224 }
..
{code}
We can avoid this.. CellUtil already having method like
writeRow(DataOutputStream out, Cell cell, short rlength),
writeFamily(DataOutputStream out, Cell cell, byte flength).. You can make use
of them here. If u see we can change DataOutputStream signature be
OutputStream.. It should be very much fine. I believe we added it in 2.0 and
there is no BC break also.
compressCellForWal -> Do we need to move this entire logic to here? Let the
logic be there in the original place.. You can add way to do this in CellUtil
write(out, ((ByteBufferedCell) cell).getRowByteBuffer(),
1957((ByteBufferedCell) cell).getRowPosition(),
cell.getRowLength(), rowDict);
Add API like writeRow(OS,Cell) .. And the actual logic of check the Dict and
add entry to Dict can be moved to some CompressionContext class.. We have
TagCompressionContext.. May be this should be renamed and made generic to be
CellComressionContext? Let CellUtil know abt this context only and do the
Dict specific work talking with this context. But the CellUtil impl can check
whether Cell is BBCell or not and can decide to pass BB or byte[]
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch, HBASE-17012_2.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692690#comment-15692690
]
Hadoop QA commented on HBASE-17012:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 29s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 41s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 14s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 40s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestAcidGuarantees |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840378/HBASE-17012_2.patch |
| JIRA Issue | HBASE-17012 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 5e5d44ecdfa2 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0b0e857 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4608/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4608/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670348#comment-15670348
]
Hadoop QA commented on HBASE-17012:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 13s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 45s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 83m 54s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 123m 47s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839150/HBASE-17012_1.patch |
| JIRA Issue | HBASE-17012 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d82879ae2443 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0f7a7f4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4491/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4491/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Handle Offheap cells in CompressedKvEncoder
>
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670275#comment-15670275
]
Anoop Sam John commented on HBASE-17012:
bq.cout.write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength());
Here the unconditional cal to getXXXArray() can be avoided? Or else we will
end up in making so many temp byte[] garbage. ( 5 per Cell !!)
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670165#comment-15670165
]
ramkrishna.s.vasudevan commented on HBASE-17012:
I think EncryptedKVEncoder is ok as such. Because even if we
{code}
ByteArrayOutputStream baos = new ByteArrayOutputStream();
OutputStream cout = encryptor.createEncryptionStream(baos);
{code}
wrap the BAOS with ByteBufferWriterOuputStream, still what we write is to the
cout (that is the encrypted stream).
{code}
StreamUtils.writeRawVInt32(cout, KeyValueUtil.keyLength(cell));
StreamUtils.writeRawVInt32(cout, cell.getValueLength());
// To support tags
StreamUtils.writeRawVInt32(cout, tlen);
// Write row, qualifier, and family
StreamUtils.writeRawVInt32(cout, cell.getRowLength());
cout.write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength());
StreamUtils.writeRawVInt32(cout, cell.getFamilyLength());
cout.write(cell.getFamilyArray(), cell.getFamilyOffset(),
cell.getFamilyLength());
{code}
So am not sure how to write the ByteBuffer content directly to cout. But I
could actually use the hbase's BAOS and try writing to it and then do a
getBuffer() and use that buffer to write to cout.
But finally we need that baos to actually write it to the actual outputstream
'out'.
So am not sure if this will be same as writing individual byte[] using cout.
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670110#comment-15670110
]
ramkrishna.s.vasudevan commented on HBASE-17012:
Oh. Just seeing that EncryptedKVEncoder.
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15632843#comment-15632843
]
Anoop Sam John commented on HBASE-17012:
As part of this handle EncryptedKvEncoder#write(Cell)
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
> Fix For: 2.0.0
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
