[jira] [Commented] (HBASE-22802) Avoid temp ByteBuffer allocation in FileIOEngine#read
[
https://issues.apache.org/jira/browse/HBASE-22802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925769#comment-16925769
]
Hudson commented on HBASE-22802:
Results for branch master
[build #1416 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1416/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1416//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1416//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1416//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Avoid temp ByteBuffer allocation in FileIOEngine#read
> -
>
> Key: HBASE-22802
> URL: https://issues.apache.org/jira/browse/HBASE-22802
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-22802-master-v1.patch, profile_mem_alloc.png,
> profile_mem_alloc_with_pool.png
>
>
> a temp ByteBuffer was allocated each time FileIOEngine#read was called
> {code:java}
> public Cacheable read(BucketEntry be) throws IOException {
> long offset = be.offset();
> int length = be.getLength();
> Preconditions.checkArgument(length >= 0, "Length of read can not be less
> than 0.");
> ByteBuffer dstBuffer = ByteBuffer.allocate(length);
> ...
> }
> {code}
> we can avoid this by use of ByteBuffAllocator#allocate(length) after
> HBASE-21879
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22802) Avoid temp ByteBuffer allocation in FileIOEngine#read
[
https://issues.apache.org/jira/browse/HBASE-22802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925763#comment-16925763
]
Hudson commented on HBASE-22802:
Results for branch branch-2
[build #2241 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2241/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2241//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2241//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2241//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Avoid temp ByteBuffer allocation in FileIOEngine#read
> -
>
> Key: HBASE-22802
> URL: https://issues.apache.org/jira/browse/HBASE-22802
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-22802-master-v1.patch, profile_mem_alloc.png,
> profile_mem_alloc_with_pool.png
>
>
> a temp ByteBuffer was allocated each time FileIOEngine#read was called
> {code:java}
> public Cacheable read(BucketEntry be) throws IOException {
> long offset = be.offset();
> int length = be.getLength();
> Preconditions.checkArgument(length >= 0, "Length of read can not be less
> than 0.");
> ByteBuffer dstBuffer = ByteBuffer.allocate(length);
> ...
> }
> {code}
> we can avoid this by use of ByteBuffAllocator#allocate(length) after
> HBASE-21879
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22802) Avoid temp ByteBuffer allocation in FileIOEngine#read
[
https://issues.apache.org/jira/browse/HBASE-22802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925533#comment-16925533
]
Zheng Hu commented on HBASE-22802:
--
Pushed to branch-2 & master , Thanks [~javaman_chen] for contributing, please
fill the release node about the feature, [~javaman_chen].
> Avoid temp ByteBuffer allocation in FileIOEngine#read
> -
>
> Key: HBASE-22802
> URL: https://issues.apache.org/jira/browse/HBASE-22802
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-22802-master-v1.patch, profile_mem_alloc.png,
> profile_mem_alloc_with_pool.png
>
>
> a temp ByteBuffer was allocated each time FileIOEngine#read was called
> {code:java}
> public Cacheable read(BucketEntry be) throws IOException {
> long offset = be.offset();
> int length = be.getLength();
> Preconditions.checkArgument(length >= 0, "Length of read can not be less
> than 0.");
> ByteBuffer dstBuffer = ByteBuffer.allocate(length);
> ...
> }
> {code}
> we can avoid this by use of ByteBuffAllocator#allocate(length) after
> HBASE-21879
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22802) Avoid temp ByteBuffer allocation in FileIOEngine#read
[
https://issues.apache.org/jira/browse/HBASE-22802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914107#comment-16914107
]
HBase QA commented on HBASE-22802:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
6s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} hbase-common: The patch generated 0 new + 48
unchanged - 2 fixed = 48 total (was 50) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} The patch passed checkstyle in hbase-server {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
4s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 26s{color} | {color:green} Patch does not cause any errors with Hadoop
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
52s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}223m 49s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}292m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestMetaTableAccessor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base:
https://builds.apache.org/job/PreCommit-HBASE-Build/815/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-22802 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12978368/HBASE-22802-master-v1.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux cb06aa0d1235 4.15.0-54-generic #58-Ubuntu
[jira] [Commented] (HBASE-22802) Avoid temp ByteBuffer allocation in FileIOEngine#read
[
https://issues.apache.org/jira/browse/HBASE-22802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913992#comment-16913992
]
chenxu commented on HBASE-22802:
bq. In your case the file based IO engine is backed by what type of Storage
device?
By SSD
> Avoid temp ByteBuffer allocation in FileIOEngine#read
> -
>
> Key: HBASE-22802
> URL: https://issues.apache.org/jira/browse/HBASE-22802
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
> Attachments: HBASE-22802-master-v1.patch, profile_mem_alloc.png,
> profile_mem_alloc_with_pool.png
>
>
> a temp ByteBuffer was allocated each time FileIOEngine#read was called
> {code:java}
> public Cacheable read(BucketEntry be) throws IOException {
> long offset = be.offset();
> int length = be.getLength();
> Preconditions.checkArgument(length >= 0, "Length of read can not be less
> than 0.");
> ByteBuffer dstBuffer = ByteBuffer.allocate(length);
> ...
> }
> {code}
> we can avoid this by use of ByteBuffAllocator#allocate(length) after
> HBASE-21879
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22802) Avoid temp ByteBuffer allocation in FileIOEngine#read
[
https://issues.apache.org/jira/browse/HBASE-22802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913967#comment-16913967
]
ramkrishna.s.vasudevan commented on HBASE-22802:
[~javaman_chen]
In your case the file based IO engine is backed by what type of Storage device?
> Avoid temp ByteBuffer allocation in FileIOEngine#read
> -
>
> Key: HBASE-22802
> URL: https://issues.apache.org/jira/browse/HBASE-22802
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
> Attachments: HBASE-22802-master-v1.patch, profile_mem_alloc.png,
> profile_mem_alloc_with_pool.png
>
>
> a temp ByteBuffer was allocated each time FileIOEngine#read was called
> {code:java}
> public Cacheable read(BucketEntry be) throws IOException {
> long offset = be.offset();
> int length = be.getLength();
> Preconditions.checkArgument(length >= 0, "Length of read can not be less
> than 0.");
> ByteBuffer dstBuffer = ByteBuffer.allocate(length);
> ...
> }
> {code}
> we can avoid this by use of ByteBuffAllocator#allocate(length) after
> HBASE-21879
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22802) Avoid temp ByteBuffer allocation in FileIOEngine#read
[
https://issues.apache.org/jira/browse/HBASE-22802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902638#comment-16902638
]
chenxu commented on HBASE-22802:
a PR was add, hope you can review this [~openinx] , [~anoop.hbase]
> Avoid temp ByteBuffer allocation in FileIOEngine#read
> -
>
> Key: HBASE-22802
> URL: https://issues.apache.org/jira/browse/HBASE-22802
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
> Attachments: profile_mem_alloc.png
>
>
> a temp ByteBuffer was allocated each time FileIOEngine#read was called
> {code:java}
> public Cacheable read(BucketEntry be) throws IOException {
> long offset = be.offset();
> int length = be.getLength();
> Preconditions.checkArgument(length >= 0, "Length of read can not be less
> than 0.");
> ByteBuffer dstBuffer = ByteBuffer.allocate(length);
> ...
> }
> {code}
> we can avoid this by use of ByteBuffAllocator#allocate(length) after
> HBASE-21879
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22802) Avoid temp ByteBuffer allocation in FileIOEngine#read
[
https://issues.apache.org/jira/browse/HBASE-22802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901815#comment-16901815
]
chenxu commented on HBASE-22802:
ok, will have a try, do my best
> Avoid temp ByteBuffer allocation in FileIOEngine#read
> -
>
> Key: HBASE-22802
> URL: https://issues.apache.org/jira/browse/HBASE-22802
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
>Reporter: chenxu
>Priority: Major
> Attachments: profile_mem_alloc.png
>
>
> a temp ByteBuffer was allocated each time FileIOEngine#read was called
> {code:java}
> public Cacheable read(BucketEntry be) throws IOException {
> long offset = be.offset();
> int length = be.getLength();
> Preconditions.checkArgument(length >= 0, "Length of read can not be less
> than 0.");
> ByteBuffer dstBuffer = ByteBuffer.allocate(length);
> ...
> }
> {code}
> we can avoid this by use of ByteBuffAllocator#allocate(length) after
> HBASE-21879
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22802) Avoid temp ByteBuffer allocation in FileIOEngine#read
[
https://issues.apache.org/jira/browse/HBASE-22802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901803#comment-16901803
]
Zheng Hu commented on HBASE-22802:
--
So seems we can have a check in wrapAsCacheable ? If the IOEngine is shared,
then need to pass the BucketEntry refCnt to the HFileBlock , otherwise can just
use the ByteBuff's (Read from FileIOEngine & allocate from ByteBuffAllocator)
RefCnt to construct the HFileBlock ?
Sounds reasonable, [~javaman_chen], want to try to prepare patch for this ? :-)
> Avoid temp ByteBuffer allocation in FileIOEngine#read
> -
>
> Key: HBASE-22802
> URL: https://issues.apache.org/jira/browse/HBASE-22802
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
>Reporter: chenxu
>Priority: Major
> Attachments: profile_mem_alloc.png
>
>
> a temp ByteBuffer was allocated each time FileIOEngine#read was called
> {code:java}
> public Cacheable read(BucketEntry be) throws IOException {
> long offset = be.offset();
> int length = be.getLength();
> Preconditions.checkArgument(length >= 0, "Length of read can not be less
> than 0.");
> ByteBuffer dstBuffer = ByteBuffer.allocate(length);
> ...
> }
> {code}
> we can avoid this by use of ByteBuffAllocator#allocate(length) after
> HBASE-21879
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22802) Avoid temp ByteBuffer allocation in FileIOEngine#read
[
https://issues.apache.org/jira/browse/HBASE-22802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901699#comment-16901699
]
chenxu commented on HBASE-22802:
{quote}Says we must consider the two issues for FileIOEngine : release the BB
from ByteBuffAllocator & when we can do the block eviction
{quote}
1. release the BB from ByteBuffAllocator
as Anoop mentioned, I think the BB’s release flow is already in place, At
least at the RPC level
2. when we can do the block eviction
in case of ByteBufferIOEngine, block eviction happened when the following
conditions are satisfied(may be not right, please correct me)
(1) block evicted from backingMap (BucketEntry#markAsEvicted called)
(2) no rpc reference to the shared memory(ByteBuff)
in case of FileIOEngine, the BB allocated from ByteBuffAllocator is not a
shared memory anymore, So we just need to satisfy condition 1.
> Avoid temp ByteBuffer allocation in FileIOEngine#read
> -
>
> Key: HBASE-22802
> URL: https://issues.apache.org/jira/browse/HBASE-22802
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
>Reporter: chenxu
>Priority: Major
> Attachments: profile_mem_alloc.png
>
>
> a temp ByteBuffer was allocated each time FileIOEngine#read was called
> {code:java}
> public Cacheable read(BucketEntry be) throws IOException {
> long offset = be.offset();
> int length = be.getLength();
> Preconditions.checkArgument(length >= 0, "Length of read can not be less
> than 0.");
> ByteBuffer dstBuffer = ByteBuffer.allocate(length);
> ...
> }
> {code}
> we can avoid this by use of ByteBuffAllocator#allocate(length) after
> HBASE-21879
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22802) Avoid temp ByteBuffer allocation in FileIOEngine#read
[
https://issues.apache.org/jira/browse/HBASE-22802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901688#comment-16901688
]
Anoop Sam John commented on HBASE-22802:
The FileIOEngine wont be a shared memory one anyways. So evict from there wont
need any changes and so no ref counting within FileIOEngine.
The BBs from Allocator release flow is already in place? I did not check in
detail.
> Avoid temp ByteBuffer allocation in FileIOEngine#read
> -
>
> Key: HBASE-22802
> URL: https://issues.apache.org/jira/browse/HBASE-22802
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
>Reporter: chenxu
>Priority: Major
> Attachments: profile_mem_alloc.png
>
>
> a temp ByteBuffer was allocated each time FileIOEngine#read was called
> {code:java}
> public Cacheable read(BucketEntry be) throws IOException {
> long offset = be.offset();
> int length = be.getLength();
> Preconditions.checkArgument(length >= 0, "Length of read can not be less
> than 0.");
> ByteBuffer dstBuffer = ByteBuffer.allocate(length);
> ...
> }
> {code}
> we can avoid this by use of ByteBuffAllocator#allocate(length) after
> HBASE-21879
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22802) Avoid temp ByteBuffer allocation in FileIOEngine#read
[
https://issues.apache.org/jira/browse/HBASE-22802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901674#comment-16901674
]
Zheng Hu commented on HBASE-22802:
--
bq. I think the ByteBuff will be released in HFileScannerImpl#close, is that
right?
No only closing, once we've sent the cells to rpc client then we can release
the ByteBuff. because the RPC is finished, the memory allocated from
ByteBuffAllocator won't be used any more. Of course, if close the scanner,
we must also release the BB.
bq. and there is no need to share the same refCnt with BucketEntry like what
ByteBufferIOEngine to do (code in BucketEntry#wrapAsCacheable)
Seems true, but my concerns is: we maintain the RefCnt for BucketEntry because
we want to know when we can free the memory area in IOEngine ( mostly in
ByteBufferIOEngine). Once the rpc reference count decrease to zero, we can try
to evict the block if neccessary. For FileIOEngine, if the ByteBuff is
allocated from ByteBuffAllocator, then its refCnt should be a seperate one, and
if won't pass the BucketEntry's refCnt to the ByteBuff in HFIleBlock, then how
we know when can do the block eviction ? Says we must consider the two issues
for FileIOEngine : release the BB from ByteBuffAllocator & when we can do the
block eviction.
> Avoid temp ByteBuffer allocation in FileIOEngine#read
> -
>
> Key: HBASE-22802
> URL: https://issues.apache.org/jira/browse/HBASE-22802
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
>Reporter: chenxu
>Priority: Major
> Attachments: profile_mem_alloc.png
>
>
> a temp ByteBuffer was allocated each time FileIOEngine#read was called
> {code:java}
> public Cacheable read(BucketEntry be) throws IOException {
> long offset = be.offset();
> int length = be.getLength();
> Preconditions.checkArgument(length >= 0, "Length of read can not be less
> than 0.");
> ByteBuffer dstBuffer = ByteBuffer.allocate(length);
> ...
> }
> {code}
> we can avoid this by use of ByteBuffAllocator#allocate(length) after
> HBASE-21879
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22802) Avoid temp ByteBuffer allocation in FileIOEngine#read
[
https://issues.apache.org/jira/browse/HBASE-22802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901597#comment-16901597
]
chenxu commented on HBASE-22802:
bq. so we need to read ByteBuffer one by one for a MultiByteBuff ?
yes, we can declare a method like read(FileChannel channel, long offset) in
ByteBuff
bq. When to release the ByteBuff ?
I think the ByteBuff will be released it in HFileScannerImpl#close, is that
right?
and there is no need to share the same refCnt with BucketEntry like what
ByteBufferIOEngine to do (code in BucketEntry#wrapAsCacheable)
> Avoid temp ByteBuffer allocation in FileIOEngine#read
> -
>
> Key: HBASE-22802
> URL: https://issues.apache.org/jira/browse/HBASE-22802
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
>Reporter: chenxu
>Priority: Major
> Attachments: profile_mem_alloc.png
>
>
> a temp ByteBuffer was allocated each time FileIOEngine#read was called
> {code:java}
> public Cacheable read(BucketEntry be) throws IOException {
> long offset = be.offset();
> int length = be.getLength();
> Preconditions.checkArgument(length >= 0, "Length of read can not be less
> than 0.");
> ByteBuffer dstBuffer = ByteBuffer.allocate(length);
> ...
> }
> {code}
> we can avoid this by use of ByteBuffAllocator#allocate(length) after
> HBASE-21879
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22802) Avoid temp ByteBuffer allocation in FileIOEngine#read
[
https://issues.apache.org/jira/browse/HBASE-22802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901223#comment-16901223
]
Zheng Hu commented on HBASE-22802:
--
Sounds good to do this. Few questions:
1. The ByteBuffAllocator#allocate(length) will get a ByteBuff (composited by
multi nio ByteBuffers), while the fileChannel#read (from JDK) can only accept
nio ByteBuffers, so we need to read ByteBuffer one by one for a MultiByteBuff
? I think so.
2. When to release the ByteBuff ? Seems we can release the ByteBuff once the
Cacheable#release it.
Thanks.
> Avoid temp ByteBuffer allocation in FileIOEngine#read
> -
>
> Key: HBASE-22802
> URL: https://issues.apache.org/jira/browse/HBASE-22802
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
>Reporter: chenxu
>Priority: Major
> Attachments: profile_mem_alloc.png
>
>
> a temp ByteBuffer was allocated each time FileIOEngine#read was called
> {code:java}
> public Cacheable read(BucketEntry be) throws IOException {
> long offset = be.offset();
> int length = be.getLength();
> Preconditions.checkArgument(length >= 0, "Length of read can not be less
> than 0.");
> ByteBuffer dstBuffer = ByteBuffer.allocate(length);
> ...
> }
> {code}
> we can avoid this by use of ByteBuffAllocator#allocate(length) after
> HBASE-21879
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22802) Avoid temp ByteBuffer allocation in FileIOEngine#read
[
https://issues.apache.org/jira/browse/HBASE-22802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901180#comment-16901180
]
chenxu commented on HBASE-22802:
how do you think about this, ping [~openinx]
> Avoid temp ByteBuffer allocation in FileIOEngine#read
> -
>
> Key: HBASE-22802
> URL: https://issues.apache.org/jira/browse/HBASE-22802
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
>Reporter: chenxu
>Priority: Major
> Attachments: profile_mem_alloc.png
>
>
> a temp ByteBuffer was allocated each time FileIOEngine#read was called
> {code:java}
> public Cacheable read(BucketEntry be) throws IOException {
> long offset = be.offset();
> int length = be.getLength();
> Preconditions.checkArgument(length >= 0, "Length of read can not be less
> than 0.");
> ByteBuffer dstBuffer = ByteBuffer.allocate(length);
> ...
> }
> {code}
> we can avoid this by use of ByteBuffAllocator#allocate(length) after
> HBASE-21879
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
