[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357269#comment-14357269 ] stack commented on HBASE-13142: --- Made a subtask for 0.98 backport. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: Liang Xie Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.addendum.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356403#comment-14356403 ] Hudson commented on HBASE-13142: FAILURE: Integrated in HBase-1.1 #277 (See [https://builds.apache.org/job/HBase-1.1/277/]) HBASE-13142 [PERF] Reuse the IPCUtil#buildCellBlock buffer; ADDENDUM (stack: rev 0fb1ad7db46b12b6489d57e048f3baab147f78ec) * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: Liang Xie Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.addendum.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356458#comment-14356458 ] Hudson commented on HBASE-13142: SUCCESS: Integrated in HBase-TRUNK #6239 (See [https://builds.apache.org/job/HBase-TRUNK/6239/]) HBASE-13142 [PERF] Reuse the IPCUtil#buildCellBlock buffer; ADDENDUM (stack: rev b436db7d70c8a90b0167dc0e0120f503efb37e3c) * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: Liang Xie Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.addendum.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356334#comment-14356334 ] stack commented on HBASE-13142: --- [~ram_krish] You probably need the addendum boss. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: Liang Xie Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.addendum.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356348#comment-14356348 ] ramkrishna.s.vasudevan commented on HBASE-13142: bq.You probably need the addendum Yes +1 for addendum. Thanks Stack. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: Liang Xie Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.addendum.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356324#comment-14356324 ] stack commented on HBASE-13142: --- Linking HBASE-13188, a bug found by Ram [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: Liang Xie Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356322#comment-14356322 ] stack commented on HBASE-13142: --- Addendum fixes possible leak discovered trying to backport this to 0.98. 0.98 is different -- has MetaEditor etc. -- and I was able to fix one issue where we were leaking ByteBuffers and just ran out of memory but now tests won't start a minicluster master stuck trying to assign a region, RS stuck trying to report for duty. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: Liang Xie Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[
https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351690#comment-14351690
]
Andrew Purtell commented on HBASE-13142:
With this patch applied TestMetaScanner times out consistently.
{noformat}
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 92.72 sec
FAILURE! - in org.apache.hadoop.hbase.client.TestMetaScanner
testConcurrentMetaScannerAndCatalogJanitor(org.apache.hadoop.hbase.client.TestMetaScanner)
Time elapsed: 85.416 sec ERROR!
org.apache.hadoop.hbase.client.ScannerTimeoutException: 60334ms passed since
the last invocation, timeout is currently set to 6
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException:
org.apache.hadoop.hbase.UnknownScannerException: Name: 372, already closed?
{noformat}
Odd, but there it is.
[PERF] Reuse the IPCUtil#buildCellBlock buffer
--
Key: HBASE-13142
URL: https://issues.apache.org/jira/browse/HBASE-13142
Project: HBase
Issue Type: Improvement
Components: Performance
Reporter: stack
Assignee: stack
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt,
13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png,
hits.png, net.png, traces.2.svg, traces.svg
Running some scan profiling, flight recorder was mildly fingering resize of
the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It
was half-hearted blaming it for a few hundreds of ms over a five minute
sampling with a few tens of instances showing.
I tried then w/ flamegraph/lightweight profiler and this reported the buffer
allocations taking 22% of our total CPU. See attachment trace.svg.
I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and
indeed every allocation was doing a resize from initial allocation of 16k --
the default up to 220k (this test returns ten randomly sized rows zipfian
sized between 0 and 8k).
Upping the allocation to 220k meant we now avoided the resize but the initial
allocation was now blamed for 10% of allocations (see trace.2.svg attached).
Lets do buffer reuse. Will save a bunch of allocation and CPU.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351702#comment-14351702 ] stack commented on HBASE-13142: --- Let me look into it [~apurtell] It might be also be issue in master. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351667#comment-14351667 ] Andrew Purtell commented on HBASE-13142: Trying out the 0.98 patch locally [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351213#comment-14351213 ] Hudson commented on HBASE-13142: FAILURE: Integrated in HBase-TRUNK #6218 (See [https://builds.apache.org/job/HBase-TRUNK/6218/]) HBASE-13142 [PERF] Reuse the IPCUtil#buildCellBlock buffer (stack: rev 9159c82fbf6bf0b31eba58095eeca33348e2c389) * hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferOutputStream.java * hbase-common/src/test/java/org/apache/hadoop/hbase/io/TestBoundedByteBufferPool.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestByteBufferOutputStream.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/BoundedByteBufferPool.java [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351209#comment-14351209 ] Andrew Purtell commented on HBASE-13142: +1 for 0.98 if you're up for a backport [~stack] [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351304#comment-14351304 ] Hudson commented on HBASE-13142: FAILURE: Integrated in HBase-1.1 #256 (See [https://builds.apache.org/job/HBase-1.1/256/]) HBASE-13142 [PERF] Reuse the IPCUtil#buildCellBlock buffer: ADDENDUM -- Fix build breakage (stack: rev 880215b728c7f5086705e37a0815074d54280a38) * hbase-common/src/test/java/org/apache/hadoop/hbase/io/TestBoundedByteBufferPool.java [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351074#comment-14351074 ] stack commented on HBASE-13142: --- bq. ...how ugly it can get when big rows going on To see, enable TRACE on IPUtils for a few seconds on prod regionserver. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351069#comment-14351069 ] stack commented on HBASE-13142: --- Might be candidate for 0.98 [~apurtell] or for 1.0.x [~enis] given how ugly it can get when big rows going on [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351127#comment-14351127 ] Hudson commented on HBASE-13142: FAILURE: Integrated in HBase-1.1 #255 (See [https://builds.apache.org/job/HBase-1.1/255/]) HBASE-13142 [PERF] Reuse the IPCUtil#buildCellBlock buffer (stack: rev 3e3276d7fa2d20815c35b72b386efb5796db7939) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestByteBufferOutputStream.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferOutputStream.java * hbase-common/src/test/java/org/apache/hadoop/hbase/io/TestBoundedByteBufferPool.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/BoundedByteBufferPool.java [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[
https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350880#comment-14350880
]
Hadoop QA commented on HBASE-13142:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703095/13142v5.txt
against master branch at commit 6d4a4a48fe90b889c39e3ed42448fc7597a46973.
ATTACHMENT ID: 12703095
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//console
This message is automatically generated.
[PERF] Reuse the IPCUtil#buildCellBlock buffer
--
Key: HBASE-13142
URL: https://issues.apache.org/jira/browse/HBASE-13142
Project: HBase
Issue Type: Improvement
Components: Performance
Reporter: stack
Assignee: stack
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt,
13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png,
net.png, traces.2.svg, traces.svg
Running some scan profiling, flight recorder was mildly fingering resize of
the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It
was half-hearted blaming it for a few hundreds of ms over a five minute
sampling with a few tens of instances showing.
I tried then w/ flamegraph/lightweight profiler and this reported the buffer
allocations taking 22% of our total CPU. See attachment trace.svg.
I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and
indeed every allocation was doing a resize from initial allocation of 16k --
the default up to 220k (this test returns ten randomly sized rows zipfian
sized between 0 and 8k).
Upping the
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350958#comment-14350958 ] stack commented on HBASE-13142: --- Any chance of review? [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350984#comment-14350984 ] Elliott Clark commented on HBASE-13142: --- +1 Ship it. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[
https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350910#comment-14350910
]
Hadoop QA commented on HBASE-13142:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703111/13142v5.txt
against master branch at commit 6d4a4a48fe90b889c39e3ed42448fc7597a46973.
ATTACHMENT ID: 12703111
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//console
This message is automatically generated.
[PERF] Reuse the IPCUtil#buildCellBlock buffer
--
Key: HBASE-13142
URL: https://issues.apache.org/jira/browse/HBASE-13142
Project: HBase
Issue Type: Improvement
Components: Performance
Reporter: stack
Assignee: stack
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt,
13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png,
net.png, traces.2.svg, traces.svg
Running some scan profiling, flight recorder was mildly fingering resize of
the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It
was half-hearted blaming it for a few hundreds of ms over a five minute
sampling with a few tens of instances showing.
I tried then w/ flamegraph/lightweight profiler and this reported the buffer
allocations taking 22% of our total CPU. See attachment trace.svg.
I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and
indeed every allocation was doing a resize from initial allocation of 16k --
the default up to 220k (this test returns ten randomly sized rows zipfian
sized between 0 and 8k).
Upping the
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351425#comment-14351425 ] ramkrishna.s.vasudevan commented on HBASE-13142: Nice. Will try to port this to use this in our internal branch also. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349320#comment-14349320 ] stack commented on HBASE-13142: --- Any chance of a review here? Let me try on cluster and see if can get pictures of benefit. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349426#comment-14349426 ] stack commented on HBASE-13142: --- [~eclark] bq, Do you think the functionality of returning buffers in strictly sorted order is worth that much locking? Not that important I'd say. BBs would be less likely to be same general size but can't see a real downside. Let me get graphs on current state then play with something dumber. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349423#comment-14349423 ] Elliott Clark commented on HBASE-13142: --- There's going to be a lot of contention on getting a bb and putting a bb. Do you think the functionality of returning buffers in strictly sorted order is worth that much locking? Seems like this could be done with a pretty light-weight queue. Additionally shrinking the amount of code under lock could help a lot here. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349400#comment-14349400 ] Elliott Clark commented on HBASE-13142: --- Looking now. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349510#comment-14349510 ] stack commented on HBASE-13142: --- Hmm... I love testing. My key implementation is actually broken. Am allocating every time. Even with this breakage, the fact that we are getting an average size makes it so we skip a bunch of resizing and this alone is making for increased throughput (in spite of upped GC). Let me fix or rather, do the @eclark suggestion [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348380#comment-14348380 ] Hudson commented on HBASE-13142: FAILURE: Integrated in HBase-TRUNK #6209 (See [https://builds.apache.org/job/HBase-TRUNK/6209/]) HBASE-13142 [PERF] Reuse the IPCUtil#buildCellBlock buffer (stack: rev 55f8f56ad28f71a893acb1e5993689499134a018) * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestByteBufferOutputStream.java * hbase-common/src/test/java/org/apache/hadoop/hbase/io/TestByteBufferResevoir.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferOutputStream.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/BoundedByteBufferPool.java Revert HBASE-13142 [PERF] Reuse the IPCUtil#buildCellBlock buffer (stack: rev 74837d9b1f13b42415395e08ff168aeed0c21676) * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestByteBufferOutputStream.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/BoundedByteBufferPool.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferOutputStream.java * hbase-common/src/test/java/org/apache/hadoop/hbase/io/TestByteBufferResevoir.java [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[
https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349212#comment-14349212
]
Hadoop QA commented on HBASE-13142:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702842/13142v3.txt
against master branch at commit c15e684bf91fffee4fb43e5a1440154a6db2d3db.
ATTACHMENT ID: 12702842
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//console
This message is automatically generated.
[PERF] Reuse the IPCUtil#buildCellBlock buffer
--
Key: HBASE-13142
URL: https://issues.apache.org/jira/browse/HBASE-13142
Project: HBase
Issue Type: Improvement
Components: Performance
Reporter: stack
Assignee: stack
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, traces.2.svg,
traces.svg
Running some scan profiling, flight recorder was mildly fingering resize of
the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It
was half-hearted blaming it for a few hundreds of ms over a five minute
sampling with a few tens of instances showing.
I tried then w/ flamegraph/lightweight profiler and this reported the buffer
allocations taking 22% of our total CPU. See attachment trace.svg.
I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and
indeed every allocation was doing a resize from initial allocation of 16k --
the default up to 220k (this test returns ten randomly sized rows zipfian
sized between 0 and 8k).
Upping the allocation to 220k meant we now avoided the resize but the initial
allocation was now blamed for 10%
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[
https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348035#comment-14348035
]
Hadoop QA commented on HBASE-13142:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702667/13142.txt
against master branch at commit 0bdab85b065bd0876152ac30c2ec6d08adae8006.
ATTACHMENT ID: 12702667
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestInterfaceAudienceAnnotations
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13088//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13088//artifact/patchprocess/patchReleaseAuditWarnings.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13088//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13088//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13088//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13088//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13088//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13088//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13088//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13088//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13088//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13088//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13088//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13088//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13088//console
This message is automatically generated.
[PERF] Reuse the IPCUtil#buildCellBlock buffer
--
Key: HBASE-13142
URL: https://issues.apache.org/jira/browse/HBASE-13142
Project: HBase
Issue Type: Improvement
Components: Performance
Reporter: stack
Assignee: stack
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: 13142.txt, traces.2.svg, traces.svg
Running some scan profiling, flight recorder was mildly fingering resize of
the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It
was half-hearted blaming it for a few hundreds of ms over a five minute
sampling with a few tens of instances showing.
I tried then w/ flamegraph/lightweight profiler and this reported the buffer
allocations taking 22% of our total CPU. See attachment trace.svg.
I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and
indeed every allocation was doing a resize from initial allocation of 16k --
the default
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[
https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348328#comment-14348328
]
Hadoop QA commented on HBASE-13142:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702732/13142v2.txt
against master branch at commit 5bd27af8b035eaf1ffa3d1e5e4a6cf35324ef8f7.
ATTACHMENT ID: 12702732
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestCheckTestClasses
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13097//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13097//artifact/patchprocess/patchReleaseAuditWarnings.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13097//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13097//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13097//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13097//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13097//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13097//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13097//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13097//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13097//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13097//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13097//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13097//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13097//console
This message is automatically generated.
[PERF] Reuse the IPCUtil#buildCellBlock buffer
--
Key: HBASE-13142
URL: https://issues.apache.org/jira/browse/HBASE-13142
Project: HBase
Issue Type: Improvement
Components: Performance
Reporter: stack
Assignee: stack
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: 13142.txt, 13142v2.txt, traces.2.svg, traces.svg
Running some scan profiling, flight recorder was mildly fingering resize of
the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It
was half-hearted blaming it for a few hundreds of ms over a five minute
sampling with a few tens of instances showing.
I tried then w/ flamegraph/lightweight profiler and this reported the buffer
allocations taking 22% of our total CPU. See attachment trace.svg.
I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and
indeed every allocation was doing a resize from initial allocation of 16k --
the
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348284#comment-14348284 ] stack commented on HBASE-13142: --- The release edit warning was fixed by HBASE-13156 [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346822#comment-14346822 ] Anoop Sam John commented on HBASE-13142: This, we were planning to do after the current tests on HBASE-11425. We might need to calc the size required for the cell block and reuse available (already created) buffers of enough size. We can keep the created buffers as soft ref or so. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Attachments: traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344575#comment-14344575 ] ramkrishna.s.vasudevan commented on HBASE-13142: Ya. This seems to be an important area once we cross the actual scan related bottle necks. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Attachments: traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344542#comment-14344542 ] Andrew Purtell commented on HBASE-13142: bq. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. Good stuff. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Attachments: traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344506#comment-14344506 ] stack commented on HBASE-13142: --- Oh, this is a test that puts up many concurrent scanners where all data is in blockcache. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Labels: beginner Attachments: traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
