[jira] [Commented] (HBASE-22504) Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
[
https://issues.apache.org/jira/browse/HBASE-22504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146161#comment-17146161
]
Hudson commented on HBASE-22504:
Results for branch branch-2.3
[build #157 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/157/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
> ---
>
> Key: HBASE-22504
> URL: https://issues.apache.org/jira/browse/HBASE-22504
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: HBASE-22504.HBASE-21879.v01.patch
>
>
> In HBASE-22483, we saw that the BucketCacheWriter thread was quite busy
> [^BucketCacheWriter-is-busy.png], the flame graph also indicated that the
> ByteBufferArray#internalTransfer cost ~6% CPU (see
> [async-prof-pid-25042-cpu-1.svg|https://issues.apache.org/jira/secure/attachment/12970294/async-prof-pid-25042-cpu-1.svg]).
> because we used the hbase.ipc.server.allocator.buffer.size=64KB, each
> HFileBlock will be backend by a MultiByteBuff: one 64KB offheap ByteBuffer
> and one small heap ByteBuffer.
> The path is depending on the MultiByteBuff#get(ByteBuffer, offset, len) now:
> {code:java}
> RAMQueueEntry#writeToCache
> |--> ByteBufferIOEngine#write
> |--> ByteBufferArray#internalTransfer
> |--> ByteBufferArray$WRITER
> |--> MultiByteBuff#get(ByteBuffer, offset, len)
> {code}
> While the MultiByteBuff#get impl is simple and crude now, can optimze this
> implementation:
> {code:java}
> @Override
> public void get(ByteBuffer out, int sourceOffset,
> int length) {
> checkRefCount();
> // Not used from real read path actually. So not going with
> // optimization
> for (int i = 0; i < length; ++i) {
> out.put(this.get(sourceOffset + i));
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22504) Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
[
https://issues.apache.org/jira/browse/HBASE-22504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145975#comment-17145975
]
Hudson commented on HBASE-22504:
Results for branch branch-2
[build #2719 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2719/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2719/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2719/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2719/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2719/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
> ---
>
> Key: HBASE-22504
> URL: https://issues.apache.org/jira/browse/HBASE-22504
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: HBASE-22504.HBASE-21879.v01.patch
>
>
> In HBASE-22483, we saw that the BucketCacheWriter thread was quite busy
> [^BucketCacheWriter-is-busy.png], the flame graph also indicated that the
> ByteBufferArray#internalTransfer cost ~6% CPU (see
> [async-prof-pid-25042-cpu-1.svg|https://issues.apache.org/jira/secure/attachment/12970294/async-prof-pid-25042-cpu-1.svg]).
> because we used the hbase.ipc.server.allocator.buffer.size=64KB, each
> HFileBlock will be backend by a MultiByteBuff: one 64KB offheap ByteBuffer
> and one small heap ByteBuffer.
> The path is depending on the MultiByteBuff#get(ByteBuffer, offset, len) now:
> {code:java}
> RAMQueueEntry#writeToCache
> |--> ByteBufferIOEngine#write
> |--> ByteBufferArray#internalTransfer
> |--> ByteBufferArray$WRITER
> |--> MultiByteBuff#get(ByteBuffer, offset, len)
> {code}
> While the MultiByteBuff#get impl is simple and crude now, can optimze this
> implementation:
> {code:java}
> @Override
> public void get(ByteBuffer out, int sourceOffset,
> int length) {
> checkRefCount();
> // Not used from real read path actually. So not going with
> // optimization
> for (int i = 0; i < length; ++i) {
> out.put(this.get(sourceOffset + i));
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22504) Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
[
https://issues.apache.org/jira/browse/HBASE-22504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142525#comment-17142525
]
Zheng Hu commented on HBASE-22504:
--
[~ndimiduk], Well, when i write this patch , I checked there's no other class
depending on the findCommonPrefix, while it's a public method. removing it will
indeed introduce the compatibility issues. Let me restore it. Thanks.
> Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
> ---
>
> Key: HBASE-22504
> URL: https://issues.apache.org/jira/browse/HBASE-22504
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: HBASE-22504.HBASE-21879.v01.patch
>
>
> In HBASE-22483, we saw that the BucketCacheWriter thread was quite busy
> [^BucketCacheWriter-is-busy.png], the flame graph also indicated that the
> ByteBufferArray#internalTransfer cost ~6% CPU (see
> [async-prof-pid-25042-cpu-1.svg|https://issues.apache.org/jira/secure/attachment/12970294/async-prof-pid-25042-cpu-1.svg]).
> because we used the hbase.ipc.server.allocator.buffer.size=64KB, each
> HFileBlock will be backend by a MultiByteBuff: one 64KB offheap ByteBuffer
> and one small heap ByteBuffer.
> The path is depending on the MultiByteBuff#get(ByteBuffer, offset, len) now:
> {code:java}
> RAMQueueEntry#writeToCache
> |--> ByteBufferIOEngine#write
> |--> ByteBufferArray#internalTransfer
> |--> ByteBufferArray$WRITER
> |--> MultiByteBuff#get(ByteBuffer, offset, len)
> {code}
> While the MultiByteBuff#get impl is simple and crude now, can optimze this
> implementation:
> {code:java}
> @Override
> public void get(ByteBuffer out, int sourceOffset,
> int length) {
> checkRefCount();
> // Not used from real read path actually. So not going with
> // optimization
> for (int i = 0; i < length; ++i) {
> out.put(this.get(sourceOffset + i));
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22504) Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
[
https://issues.apache.org/jira/browse/HBASE-22504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874112#comment-16874112
]
Hudson commented on HBASE-22504:
Results for branch branch-2
[build #2029 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2029/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2029//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2029//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2029//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
> ---
>
> Key: HBASE-22504
> URL: https://issues.apache.org/jira/browse/HBASE-22504
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Attachments: HBASE-22504.HBASE-21879.v01.patch
>
>
> In HBASE-22483, we saw that the BucketCacheWriter thread was quite busy
> [^BucketCacheWriter-is-busy.png], the flame graph also indicated that the
> ByteBufferArray#internalTransfer cost ~6% CPU (see
> [async-prof-pid-25042-cpu-1.svg|https://issues.apache.org/jira/secure/attachment/12970294/async-prof-pid-25042-cpu-1.svg]).
> because we used the hbase.ipc.server.allocator.buffer.size=64KB, each
> HFileBlock will be backend by a MultiByteBuff: one 64KB offheap ByteBuffer
> and one small heap ByteBuffer.
> The path is depending on the MultiByteBuff#get(ByteBuffer, offset, len) now:
> {code:java}
> RAMQueueEntry#writeToCache
> |--> ByteBufferIOEngine#write
> |--> ByteBufferArray#internalTransfer
> |--> ByteBufferArray$WRITER
> |--> MultiByteBuff#get(ByteBuffer, offset, len)
> {code}
> While the MultiByteBuff#get impl is simple and crude now, can optimze this
> implementation:
> {code:java}
> @Override
> public void get(ByteBuffer out, int sourceOffset,
> int length) {
> checkRefCount();
> // Not used from real read path actually. So not going with
> // optimization
> for (int i = 0; i < length; ++i) {
> out.put(this.get(sourceOffset + i));
> }
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22504) Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
[
https://issues.apache.org/jira/browse/HBASE-22504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870937#comment-16870937
]
Hudson commented on HBASE-22504:
Results for branch master
[build #1168 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1168/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1168//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1168//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1168//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
> ---
>
> Key: HBASE-22504
> URL: https://issues.apache.org/jira/browse/HBASE-22504
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Attachments: HBASE-22504.HBASE-21879.v01.patch
>
>
> In HBASE-22483, we saw that the BucketCacheWriter thread was quite busy
> [^BucketCacheWriter-is-busy.png], the flame graph also indicated that the
> ByteBufferArray#internalTransfer cost ~6% CPU (see
> [async-prof-pid-25042-cpu-1.svg|https://issues.apache.org/jira/secure/attachment/12970294/async-prof-pid-25042-cpu-1.svg]).
> because we used the hbase.ipc.server.allocator.buffer.size=64KB, each
> HFileBlock will be backend by a MultiByteBuff: one 64KB offheap ByteBuffer
> and one small heap ByteBuffer.
> The path is depending on the MultiByteBuff#get(ByteBuffer, offset, len) now:
> {code:java}
> RAMQueueEntry#writeToCache
> |--> ByteBufferIOEngine#write
> |--> ByteBufferArray#internalTransfer
> |--> ByteBufferArray$WRITER
> |--> MultiByteBuff#get(ByteBuffer, offset, len)
> {code}
> While the MultiByteBuff#get impl is simple and crude now, can optimze this
> implementation:
> {code:java}
> @Override
> public void get(ByteBuffer out, int sourceOffset,
> int length) {
> checkRefCount();
> // Not used from real read path actually. So not going with
> // optimization
> for (int i = 0; i < length; ++i) {
> out.put(this.get(sourceOffset + i));
> }
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22504) Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
[
https://issues.apache.org/jira/browse/HBASE-22504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852701#comment-16852701
]
Zheng Hu commented on HBASE-22504:
--
Pushed to HBASE-21879 branch, Thanks [~ram_krish] & [~anoop.hbase] for
reviewing.
> Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
> ---
>
> Key: HBASE-22504
> URL: https://issues.apache.org/jira/browse/HBASE-22504
> Project: HBase
> Issue Type: Sub-task
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Attachments: HBASE-22504.HBASE-21879.v01.patch
>
>
> In HBASE-22483, we saw that the BucketCacheWriter thread was quite busy
> [^BucketCacheWriter-is-busy.png], the flame graph also indicated that the
> ByteBufferArray#internalTransfer cost ~6% CPU (see
> [async-prof-pid-25042-cpu-1.svg|https://issues.apache.org/jira/secure/attachment/12970294/async-prof-pid-25042-cpu-1.svg]).
> because we used the hbase.ipc.server.allocator.buffer.size=64KB, each
> HFileBlock will be backend by a MultiByteBuff: one 64KB offheap ByteBuffer
> and one small heap ByteBuffer.
> The path is depending on the MultiByteBuff#get(ByteBuffer, offset, len) now:
> {code:java}
> RAMQueueEntry#writeToCache
> |--> ByteBufferIOEngine#write
> |--> ByteBufferArray#internalTransfer
> |--> ByteBufferArray$WRITER
> |--> MultiByteBuff#get(ByteBuffer, offset, len)
> {code}
> While the MultiByteBuff#get impl is simple and crude now, can optimze this
> implementation:
> {code:java}
> @Override
> public void get(ByteBuffer out, int sourceOffset,
> int length) {
> checkRefCount();
> // Not used from real read path actually. So not going with
> // optimization
> for (int i = 0; i < length; ++i) {
> out.put(this.get(sourceOffset + i));
> }
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22504) Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
[
https://issues.apache.org/jira/browse/HBASE-22504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852696#comment-16852696
]
Zheng Hu commented on HBASE-22504:
--
Checked all API in MBB, seems other API are optimized now. Let me commit this
firstly, will give a benchmark later.
> Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
> ---
>
> Key: HBASE-22504
> URL: https://issues.apache.org/jira/browse/HBASE-22504
> Project: HBase
> Issue Type: Sub-task
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Attachments: HBASE-22504.HBASE-21879.v01.patch
>
>
> In HBASE-22483, we saw that the BucketCacheWriter thread was quite busy
> [^BucketCacheWriter-is-busy.png], the flame graph also indicated that the
> ByteBufferArray#internalTransfer cost ~6% CPU (see
> [async-prof-pid-25042-cpu-1.svg|https://issues.apache.org/jira/secure/attachment/12970294/async-prof-pid-25042-cpu-1.svg]).
> because we used the hbase.ipc.server.allocator.buffer.size=64KB, each
> HFileBlock will be backend by a MultiByteBuff: one 64KB offheap ByteBuffer
> and one small heap ByteBuffer.
> The path is depending on the MultiByteBuff#get(ByteBuffer, offset, len) now:
> {code:java}
> RAMQueueEntry#writeToCache
> |--> ByteBufferIOEngine#write
> |--> ByteBufferArray#internalTransfer
> |--> ByteBufferArray$WRITER
> |--> MultiByteBuff#get(ByteBuffer, offset, len)
> {code}
> While the MultiByteBuff#get impl is simple and crude now, can optimze this
> implementation:
> {code:java}
> @Override
> public void get(ByteBuffer out, int sourceOffset,
> int length) {
> checkRefCount();
> // Not used from real read path actually. So not going with
> // optimization
> for (int i = 0; i < length; ++i) {
> out.put(this.get(sourceOffset + i));
> }
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22504) Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
[
https://issues.apache.org/jira/browse/HBASE-22504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852619#comment-16852619
]
Anoop Sam John commented on HBASE-22504:
Good find.. +1
Could check for the perf after this fix?
When the class was introduced, seems no one used it and so left with that impl.
The comment says so too. Could you see any other APIs (used now or even not
used) in MBB with this way of non optimized impl? Better we correct all of
those.
> Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
> ---
>
> Key: HBASE-22504
> URL: https://issues.apache.org/jira/browse/HBASE-22504
> Project: HBase
> Issue Type: Sub-task
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Attachments: HBASE-22504.HBASE-21879.v01.patch
>
>
> In HBASE-22483, we saw that the BucketCacheWriter thread was quite busy
> [^BucketCacheWriter-is-busy.png], the flame graph also indicated that the
> ByteBufferArray#internalTransfer cost ~6% CPU (see
> [async-prof-pid-25042-cpu-1.svg|https://issues.apache.org/jira/secure/attachment/12970294/async-prof-pid-25042-cpu-1.svg]).
> because we used the hbase.ipc.server.allocator.buffer.size=64KB, each
> HFileBlock will be backend by a MultiByteBuff: one 64KB offheap ByteBuffer
> and one small heap ByteBuffer.
> The path is depending on the MultiByteBuff#get(ByteBuffer, offset, len) now:
> {code:java}
> RAMQueueEntry#writeToCache
> |--> ByteBufferIOEngine#write
> |--> ByteBufferArray#internalTransfer
> |--> ByteBufferArray$WRITER
> |--> MultiByteBuff#get(ByteBuffer, offset, len)
> {code}
> While the MultiByteBuff#get impl is simple and crude now, can optimze this
> implementation:
> {code:java}
> @Override
> public void get(ByteBuffer out, int sourceOffset,
> int length) {
> checkRefCount();
> // Not used from real read path actually. So not going with
> // optimization
> for (int i = 0; i < length; ++i) {
> out.put(this.get(sourceOffset + i));
> }
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
