[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14184357#comment-14184357 ] Ted Yu commented on HBASE-12202: +1 on addendum. Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202-addendum.patch, HBASE-12202.patch, HBASE-12202_V2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14184392#comment-14184392 ] Hudson commented on HBASE-12202: SUCCESS: Integrated in HBase-1.0 #361 (See [https://builds.apache.org/job/HBase-1.0/361/]) HBASE-12202 Support DirectByteBuffer usage in HFileBlock - addendum (anoop.s.john: rev 37ac17f62638420430e1004aa48e29d5291e07b5) * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202-addendum.patch, HBASE-12202.patch, HBASE-12202_V2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14184393#comment-14184393 ] Hudson commented on HBASE-12202: SUCCESS: Integrated in HBase-TRUNK #5702 (See [https://builds.apache.org/job/HBase-TRUNK/5702/]) HBASE-12202 Support DirectByteBuffer usage in HFileBlock - addendum (anoop.s.john: rev 34f9962618c85ad041ca7eac4913453335a81647) * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202-addendum.patch, HBASE-12202.patch, HBASE-12202_V2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14178407#comment-14178407 ] Hudson commented on HBASE-12202: FAILURE: Integrated in HBase-1.0 #332 (See [https://builds.apache.org/job/HBase-1.0/332/]) HBASE-12202 Support DirectByteBuffer usage in HFileBlock. (anoopsamjohn: rev 67d1cee2d868d0924968dccfb93b1536a0265086) * hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferInputStream.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultDecodingContext.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java * hbase-common/src/test/java/org/apache/hadoop/hbase/io/TestByteBufferInputStream.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202.patch, HBASE-12202_V2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14178451#comment-14178451 ] Hudson commented on HBASE-12202: SUCCESS: Integrated in HBase-TRUNK #5678 (See [https://builds.apache.org/job/HBase-TRUNK/5678/]) HBASE-12202 Support DirectByteBuffer usage in HFileBlock. (anoopsamjohn: rev c01d9981d89ba3633377952d132c88536c86ebc5) * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultDecodingContext.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferInputStream.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * hbase-common/src/test/java/org/apache/hadoop/hbase/io/TestByteBufferInputStream.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202.patch, HBASE-12202_V2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177932#comment-14177932 ] Enis Soztutar commented on HBASE-12202: --- Skimmed the patch. Looks good for branch-1 as well. Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202.patch, HBASE-12202_V2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176319#comment-14176319 ] Anoop Sam John commented on HBASE-12202: Ping [~enis] for branch-1 Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202.patch, HBASE-12202_V2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176356#comment-14176356 ] ramkrishna.s.vasudevan commented on HBASE-12202: Ya good with the patch. And surely a later clean up for getting Read only BB. BB already has the facility to create actual read only buffer. We can do that in the clean up. Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202.patch, HBASE-12202_V2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176092#comment-14176092 ] stack commented on HBASE-12202: --- I see what Ram is saying. Looking, duplicate is effectively equal to a wrap (new instance plus setting internal pointers). You good w/ the patch then [~ram_krish]? Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202.patch, HBASE-12202_V2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174774#comment-14174774 ] Anoop Sam John commented on HBASE-12202: We keep it as it was in old code. Further cleanup may be as follow up issue? It depends also on whether we change to BR Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202.patch, HBASE-12202_V2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[
https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173634#comment-14173634
]
Hadoop QA commented on HBASE-12202:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12675256/HBASE-12202_V2.patch
against trunk revision .
ATTACHMENT ID: 12675256
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.http.TestHttpServerLifecycle.testStartedServerIsAlive(TestHttpServerLifecycle.java:71)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11375//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11375//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11375//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11375//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11375//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11375//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11375//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11375//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11375//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11375//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11375//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11375//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11375//console
This message is automatically generated.
Support DirectByteBuffer usage in HFileBlock
Key: HBASE-12202
URL: https://issues.apache.org/jira/browse/HBASE-12202
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 2.0.0, 0.99.2
Attachments: HBASE-12202.patch, HBASE-12202_V2.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[
https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173689#comment-14173689
]
ramkrishna.s.vasudevan commented on HBASE-12202:
Had a look at the patch.
- Now doing duplicate and a slice, we are doing more operations but they may
be minor I suppose.
- ByteBufferUtils.copyFromBufferToBuffer() can we reuse the one above the new
one. I mean can we refactor both of them.
- In HfileBlock.getBufferReadOnly()
{code}
-return ByteBuffer.wrap(buf.array(), buf.arrayOffset(),
-buf.limit() - totalChecksumBytes()).slice();
+ByteBuffer dup = this.buf.duplicate();
+dup.limit(buf.limit() - totalChecksumBytes());
+return dup.slice();
{code}
This will impact the limit and the capacity of the returning array. Will that
be a concern? When we say a read only will it be better to return a real read
only Buffer? In case of BR we tried to do achieve that by returning an
ImmutableBR unless and otherwise stated to return a MutableBR.
{code}
ByteBuffer inDup = this.buf.duplicate();
+ inDup.limit(inDup.limit() + headerSize());
{code}
Why is the limit after considering the headerSize()?
Support DirectByteBuffer usage in HFileBlock
Key: HBASE-12202
URL: https://issues.apache.org/jira/browse/HBASE-12202
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 2.0.0, 0.99.2
Attachments: HBASE-12202.patch, HBASE-12202_V2.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173756#comment-14173756 ] Anoop Sam John commented on HBASE-12202: No extra overhead. Previously we had wrap() call and then slice. This will create 2 new objects but no data copy. In patch also same way. The existing will move the position of the destination buffer and is being used already. This is different in which both src and destination ops are absolute and no pos changes. IMO better can have 2 methods. Not many lines of codes.. Than passing a boolean and based on that do op, better 2 APIs getBufferReadOnly - the ops will be same as the old way. We want to limit the new buffer to exclude the checksum bytes. bq.return a real read only Buffer? But we need new limit. So no new BB object is needed. bq.Why is the limit after considering the headerSize()? We increase the limit here because we have to copy the header data of the next block. In case of array based copying this was ok. But now it is buffer to buffer copy. We can not read/write after limit. I just wrote comments above this code. Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202.patch, HBASE-12202_V2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174034#comment-14174034 ] ramkrishna.s.vasudevan commented on HBASE-12202: bq.We increase the limit here because we have to copy the header data of the next block This is limit right? So it is fine. When I meant read only buffer I meant should we have way to return a read only byte buffer? Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202.patch, HBASE-12202_V2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172180#comment-14172180 ] ramkrishna.s.vasudevan commented on HBASE-12202: Pls give me a day's time. I will check this and get back on it. Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172343#comment-14172343 ] Anoop Sam John commented on HBASE-12202: Sure Ram. Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[
https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171743#comment-14171743
]
stack commented on HBASE-12202:
---
Interesting. ByteBufferInputStream seems to be a pretty common invention if I
poke on net. Thought there was one already in hadoop but no
Generally I think this unnecessary... it is going to happen anyways?
67 if (b == null) {
68throw new NullPointerException();
... and this?
} else if (off 0 || len 0 || len b.length - off) {
70throw new IndexOutOfBoundsException();
71 }
Looks good to me. You think it can work w/ offheap BBs now?
Support DirectByteBuffer usage in HFileBlock
Key: HBASE-12202
URL: https://issues.apache.org/jira/browse/HBASE-12202
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 2.0.0, 0.99.2
Attachments: HBASE-12202.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171948#comment-14171948 ] Anoop Sam John commented on HBASE-12202: The null checks and others I can remove. Agree with you. It can work with offheap within this class. But an e2e test can be possible only after we do similar work in other read flow areas. Me and Ram working on those already. Like the above checks they are there in BB impls. Andy used to share concerns over this wrt perf. Also these can result in code not getting inlined. BB to BR change was ib that regard so that we can have own simplied impls of BB which is backed by BR (?) Any way those area will touch after these changes. Thanks boss Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171971#comment-14171971 ] stack commented on HBASE-12202: --- Ok +1 to commit. Remove the checks on commit. Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12202.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[
https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169273#comment-14169273
]
Hadoop QA commented on HBASE-12202:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674496/HBASE-12202.patch
against trunk revision .
ATTACHMENT ID: 12674496
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 4
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11325//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11325//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11325//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11325//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11325//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11325//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11325//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11325//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11325//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11325//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11325//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11325//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11325//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11325//console
This message is automatically generated.
Support DirectByteBuffer usage in HFileBlock
Key: HBASE-12202
URL: https://issues.apache.org/jira/browse/HBASE-12202
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 2.0.0, 0.99.2
Attachments: HBASE-12202.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163493#comment-14163493 ] Anoop Sam John commented on HBASE-12202: Trying to avoid calls to BB#array() BB#arrayOffset(). Can give patch in a day or 2. Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163492#comment-14163492 ] Anoop Sam John commented on HBASE-12202: Trying to avoid calls to BB#array() BB#arrayOffset(). Can give patch in a day or 2. Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163625#comment-14163625 ] stack commented on HBASE-12202: --- Good one [~anoop.hbase] I was looking in these parts these last few days. I was trying to shoehorn in using the ByteBuffer API on DFSInputStream HBASE-11710 to try and see if speedup. If we got data from DFS in ByteBuffers, we might be able to make use of the native checksummers (probably not directly, not w/o work -- checksumming is from 20--30% of CPU used in simple tests). So, read into offheap BB and passing to checksummer offheap could be a win. Would mean we' need to change read to be ByteBuffer rather than byte [] -based. Seemed like substantial change but we are doing a bit of ByteBuffer slinging already in here in HFileBlock? WDYT? Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163826#comment-14163826 ] ramkrishna.s.vasudevan commented on HBASE-12202: We discussed about this. Already we raised a JIRA to make BRs to be used in HFileBlock. But not sure if that is needed now. If so we need to pursue that also. Should we try for BR? Also the thought is that we may not convert entire read path starting from DFS client passed byte [] to be converted to ByteBuffer at the first stage. Only when the block moves to L2 cache from there start serving the reads from Offheap byte buffer. What do you think? Do you feel we can address from the DFSClient API itself? Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163844#comment-14163844 ] stack commented on HBASE-12202: --- bq. Already we raised a JIRA to make BRs to be used in HFileBlock. But not sure if that is needed now. Lets do some experiments, POC, to see what is best fit? bq. Should we try for BR? Lets draw out the pipeline. Can mark where BR is better fit and ditto BB (BR does not preclude BB, right?) bq. Also the thought is that we may not convert entire read path starting from DFS client passed byte [] to be converted to ByteBuffer at the first stage. Only when the block moves to L2 cache from there start serving the reads from Offheap byte buffer. What do you think? Do you feel we can address from the DFSClient API itself? Not sure I follow [~ram_krish] Please say again. Would be cool if we could get from dfsclient into DBBs, checksum, and then put into BC. May necessitate a copy from the DFSClient DBB to our own BC DBB because DFSClient will want its DBB back. Offheap copy of DBBs should be fast though and no garbage? Anyways, just thinking out loud. Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163914#comment-14163914 ] Anoop Sam John commented on HBASE-12202: Current work/idea is to continue the read from DFSClient to on heap only.(This is to a byte[] now) The L2 cache will/can be offheap and then when HFileBlock is being got from L2 cache avoid the need to copy to on heap. DFSClient read part can be checked as next steps. (If you are experimenting with that great. Will wait to see your analysis and test results) Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164171#comment-14164171 ] stack commented on HBASE-12202: --- [~anoop.hbase] ok. Looking forward to patch. Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164612#comment-14164612 ] ramkrishna.s.vasudevan commented on HBASE-12202: bq.Lets do some experiments, POC, to see what is best fit? Then based on this there may be some API changes needed in HFileBlock. I have some patch for this but changing the entire code path to BR including the BC related code is quite challenging. Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164767#comment-14164767 ] Lars Hofhansl commented on HBASE-12202: --- Careful with DBBs. JVM will one for each size allocated and not reuse (unless same size), and in addition they not cleaned until we do a full GC (at least that used to be the case). So we need to make sure we allocate DBB of same size (maybe from a pool). There might be other issues too. Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164777#comment-14164777 ] Anoop Sam John commented on HBASE-12202: As part of this issue and coming sub tasks we wont be creating new DBBs. Only thing is in the read path , there is assumption of BB being HBB. Just will avoid that so that if DBB used to back up the data we don't get issues. When the data block is in L2 cache and that is offheap we don't need to copy it back to on heap again.Thanks for the advice Lars. Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164797#comment-14164797 ] Lars Hofhansl commented on HBASE-12202: --- I see. That makes sense. If the DFSClient already hands us a DBB, we should use that. Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
[ https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164800#comment-14164800 ] Anoop Sam John commented on HBASE-12202: Stack is working on that area in HBASE-11710 Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
