[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827571#comment-13827571 ] Hudson commented on HBASE-5945: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #125 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/125/]) HBASE-5945 Reduce buffer copies in IPC server response path (stack: rev 1543463) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/branches/0.96/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * /hbase/branches/0.96/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * /hbase/branches/0.96/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/BufferChain.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestBufferChain.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestIPC.java > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.95.2 >Reporter: Todd Lipcon >Assignee: stack > Fix For: 0.96.1 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, 5945v2.txt, 5945v4.txt, 5945v4.txt, 5945v5.txt, > buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt, with_patch.png, > without_patch.png > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827023#comment-13827023 ] Hudson commented on HBASE-5945: --- SUCCESS: Integrated in HBase-TRUNK #4688 (See [https://builds.apache.org/job/HBase-TRUNK/4688/]) HBASE-5945 Reduce buffer copies in IPC server response path (stack: rev 1543458) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/BufferChain.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestBufferChain.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestIPC.java > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.95.2 >Reporter: Todd Lipcon >Assignee: stack > Fix For: 0.96.1 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, 5945v2.txt, 5945v4.txt, 5945v4.txt, 5945v5.txt, > buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt, with_patch.png, > without_patch.png > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13826987#comment-13826987 ] Hudson commented on HBASE-5945: --- SUCCESS: Integrated in hbase-0.96 #196 (See [https://builds.apache.org/job/hbase-0.96/196/]) HBASE-5945 Reduce buffer copies in IPC server response path (stack: rev 1543463) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/branches/0.96/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * /hbase/branches/0.96/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * /hbase/branches/0.96/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/BufferChain.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestBufferChain.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestIPC.java > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.95.2 >Reporter: Todd Lipcon >Assignee: stack > Fix For: 0.96.1 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, 5945v2.txt, 5945v4.txt, 5945v4.txt, 5945v5.txt, > buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt, with_patch.png, > without_patch.png > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13826844#comment-13826844 ] Hudson commented on HBASE-5945: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #844 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/844/]) HBASE-5945 Reduce buffer copies in IPC server response path (stack: rev 1543458) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/BufferChain.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestBufferChain.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestIPC.java > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.95.2 >Reporter: Todd Lipcon >Assignee: stack > Fix For: 0.96.1 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, 5945v2.txt, 5945v4.txt, 5945v4.txt, 5945v5.txt, > buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt, with_patch.png, > without_patch.png > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825383#comment-13825383 ] stack commented on HBASE-5945: -- I will commit this later today unless objection. > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.95.2 >Reporter: Todd Lipcon >Assignee: stack > Fix For: 0.96.1 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, 5945v2.txt, 5945v4.txt, 5945v4.txt, 5945v5.txt, > buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt, with_patch.png, > without_patch.png > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[
https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825138#comment-13825138
]
Hadoop QA commented on HBASE-5945:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614339/5945v5.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
10 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7909//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7909//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7909//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7909//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7909//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7909//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7909//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7909//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7909//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7909//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7909//console
This message is automatically generated.
> Reduce buffer copies in IPC server response path
>
>
> Key: HBASE-5945
> URL: https://issues.apache.org/jira/browse/HBASE-5945
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 0.95.2
>Reporter: Todd Lipcon
>Assignee: stack
> Fix For: 0.96.1
>
> Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch,
> 5945-in-progress.patch, 5945v2.txt, 5945v4.txt, 5945v4.txt, 5945v5.txt,
> buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt, with_patch.png,
> without_patch.png
>
>
> The new PB code is sloppy with buffers and makes several needless copies.
> This increases GC time a lot. A few simple changes can cut this back down.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[
https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824339#comment-13824339
]
Hadoop QA commented on HBASE-5945:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614169/5945v4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 9
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface
org.apache.hadoop.hbase.util.TestHBaseFsck
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
{color:red}-1 core zombie tests{color}. There are 11 zombie test(s):
at
org.apache.hadoop.hbase.coprocessor.TestHTableWrapper.testHTableInterfaceMethods(TestHTableWrapper.java:139)
at
org.apache.hadoop.hbase.replication.TestMasterReplication.testCyclicReplication3(TestMasterReplication.java:243)
at
org.apache.hadoop.hbase.replication.TestMasterReplication.testCyclicReplication2(TestMasterReplication.java:206)
at
org.apache.hadoop.hbase.replication.TestMasterReplication.testCyclicReplication1(TestMasterReplication.java:150)
at
org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testReplayCmd(TestDistributedLogSplitting.java:704)
at
org.apache.hadoop.hbase.client.TestFromClientSide3.testHTableExistsMethodMultipleRegionsMultipleGets(TestFromClientSide3.java:348)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7891//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7891//console
This message is automatically generated.
> Reduce buffer copies in IPC server response path
>
>
> Key: HBASE-5945
> URL: https://issues.apache.org/jira/browse/HBASE-5945
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 0.95.2
>Reporter: Todd Lipcon
>Assignee: stack
> Fix For: 0.96.1
>
> Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch,
> 5945-in-progress.patch, 5945v2.txt, 5945v4.txt, 5945v4.txt,
> buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt, with_patch.png,
> without_patch.png
>
>
> The new PB code is sloppy with buffers and makes several needless copies.
> This increases GC time a lot. A few simple changes can cut this back down.
--
This message was sent by At
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[
https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824336#comment-13824336
]
Hadoop QA commented on HBASE-5945:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614164/5945v4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 9
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface
{color:red}-1 core zombie tests{color}. There are 12 zombie test(s):
at
org.apache.hadoop.hbase.coprocessor.TestHTableWrapper.testHTableInterfaceMethods(TestHTableWrapper.java:139)
at
org.apache.hadoop.hbase.replication.TestMasterReplication.testCyclicReplication3(TestMasterReplication.java:243)
at
org.apache.hadoop.hbase.replication.TestMasterReplication.testCyclicReplication2(TestMasterReplication.java:206)
at
org.apache.hadoop.hbase.replication.TestMasterReplication.testCyclicReplication1(TestMasterReplication.java:150)
at
org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testReplayCmd(TestDistributedLogSplitting.java:704)
at
org.apache.hadoop.hbase.client.TestFromClientSide3.testHTableExistsMethodMultipleRegionsMultipleGets(TestFromClientSide3.java:348)
at
org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers.testEncodedSeeker(TestEncodedSeekers.java:124)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7890//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7890//console
This message is automatically generated.
> Reduce buffer copies in IPC server response path
>
>
> Key: HBASE-5945
> URL: https://issues.apache.org/jira/browse/HBASE-5945
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 0.95.2
>Reporter: Todd Lipcon
>Assignee: stack
> Fix For: 0.96.1
>
> Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch,
> 5945-in-progress.patch, 5945v2.txt, 5945v4.txt, 5945v4.txt,
> buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt, with_patch.png,
> without_patch.png
>
>
> The new PB code is sloppy with buffers and makes several needless copies.
> This increases GC time a lot. A few simple changes can
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824271#comment-13824271 ] stack commented on HBASE-5945: -- [~devaraj] Just saw your comment. Thanks for review. Rough numbers attached. If hadoopqa passes I'll commit. There is more to be had here I'd say but will do in new issue -- then go read-side. Different approach on read-side I'd say --- look at keeping the request off-heap as long as possible.. we'll see. > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.95.2 >Reporter: Todd Lipcon >Assignee: stack > Fix For: 0.96.1 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, 5945v2.txt, 5945v4.txt, 5945v4.txt, > buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt, with_patch.png, > without_patch.png > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13823378#comment-13823378 ] Devaraj Das commented on HBASE-5945: Skimmed it. Looks okay to me. Any chance of getting perf numbers. If not, fine as well. Also, just wondering would it be possible to apply the pattern on the read side as well? > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.95.2 >Reporter: Todd Lipcon >Assignee: stack > Fix For: 0.96.1 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, 5945v2.txt, buffer-copies.txt, even-fewer-copies.txt, > hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[
https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822472#comment-13822472
]
Hadoop QA commented on HBASE-5945:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613551/5945v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:red}-1 hadoop2.0{color}. The patch failed to compile against the
hadoop 2.0 profile.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7862//console
This message is automatically generated.
> Reduce buffer copies in IPC server response path
>
>
> Key: HBASE-5945
> URL: https://issues.apache.org/jira/browse/HBASE-5945
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 0.95.2
>Reporter: Todd Lipcon
>Assignee: stack
> Fix For: 0.96.1
>
> Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch,
> 5945-in-progress.patch, 5945v2.txt, buffer-copies.txt, even-fewer-copies.txt,
> hbase-5495.txt
>
>
> The new PB code is sloppy with buffers and makes several needless copies.
> This increases GC time a lot. A few simple changes can cut this back down.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13821104#comment-13821104 ] Nicolas Liochon commented on HBASE-5945: +1 on the idea. I think we need to push byte buffers all over the place to save on all these copies. > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.95.2 >Reporter: Todd Lipcon >Assignee: stack > Fix For: 0.96.1 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, 5945v2.txt, buffer-copies.txt, even-fewer-copies.txt, > hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13821083#comment-13821083 ] stack commented on HBASE-5945: -- Stuff is very different now. The attached patch gets the big idea. The other fixups in hbase-5495.txt no longer apply. There is no more a HbaseObjectWritable nor Invocation. The hand-making of pbs I did not do but picked up the creation of CodedOutputStream using byte array to avoid it internally assigning buffer. The RPCServer changes make sense still a little complicated by sasl wrapping but we still get a win I believe. Will try and do a bit of benchmarking... > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.95.2 >Reporter: Todd Lipcon >Assignee: stack > Fix For: 0.96.1 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, 5945v2.txt, buffer-copies.txt, even-fewer-copies.txt, > hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609722#comment-13609722 ] Devaraj Das commented on HBASE-5945: Looked at this some. HBASE-7905 covered the improvements that have been discussed here. Maybe we can close this issue now.. > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: stack >Priority: Blocker > Fix For: 0.95.0 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, buffer-copies.txt, even-fewer-copies.txt, > hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13553004#comment-13553004 ] Devaraj Das commented on HBASE-5945: Okay, [~stack], this is fine. We can revisit this issue after HBASE-7533. > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, buffer-copies.txt, even-fewer-copies.txt, > hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13552350#comment-13552350 ] stack commented on HBASE-5945: -- bq. are you saying that we preserve most of the current RPC header/body structure, and commit the patch here with a focus on removing buffer copies? No. If we do that, we have to put rpc back together twice because it will have changed radically twice. We will also reduce buffer copies that we may not be of use in the final rpc incarnation. I was thinking we do the spec first over in 7533. This issue would be about saving buffer copies after we put rpc back together after 7533. > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, buffer-copies.txt, even-fewer-copies.txt, > hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551743#comment-13551743 ] Devaraj Das commented on HBASE-5945: [~stack], just trying to understand - are you saying that we preserve most of the current RPC header/body structure, and commit the patch here with a focus on removing buffer copies? (the RPC.proto in the last patch here has the description of the RPC header/body). Then follow it up on 7533 with the spec'ed out RPC header/body and a separate patch ? > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, buffer-copies.txt, even-fewer-copies.txt, > hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551632#comment-13551632 ] stack commented on HBASE-5945: -- [~devaraj] I'd say let this issue be about reducing buffer copies. hbase-7533 should be spec., yeah. New issue for implementation? Or, just leave that over in hbase-7533. Trying to implement the spec. will probably bring on a spec edit or two I'd say. > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, buffer-copies.txt, even-fewer-copies.txt, > hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551467#comment-13551467 ] Devaraj Das commented on HBASE-5945: [~stack], yes, there is overlap... I am currently trying to fix up HBASE-7213's test failures. I'll circle back on HBASE-7533 sometime by end of today or tomorrow. We can keep 7533 as a spec jira and use this for impl. Does it work. > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, buffer-copies.txt, even-fewer-copies.txt, > hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551460#comment-13551460 ] stack commented on HBASE-5945: -- [~devaraj] Looking at the patch, there is a bunch of overlap with HBASE-7533 where we are trying to spec out what goes on wire before coding it up. What you reckon? If we remove the 7533 stuff from this patch, what is left over? What buffer copies are we saving in particular? Good on you boss. > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, buffer-copies.txt, even-fewer-copies.txt, > hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[
https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548915#comment-13548915
]
Hadoop QA commented on HBASE-5945:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563981/5945-in-progress.2.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSide
org.apache.hadoop.hbase.TestLocalHBaseCluster
{color:red}-1 core zombie tests{color}. There are 8 zombie test(s):
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:833)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3951//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3951//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3951//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3951//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3951//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3951//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3951//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3951//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3951//console
This message is automatically generated.
> Reduce buffer copies in IPC server response path
>
>
> Key: HBASE-5945
> URL: https://issues.apache.org/jira/browse/HBASE-5945
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 0.96.0
>Reporter: Todd Lipcon
>Assignee: stack
>Priority: Blocker
> Fix For: 0.96.0
>
> Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch,
> 5945-in-progress.patch, buffer-copies.txt, even-fewer-copies.txt,
> hbase-5495.txt
>
>
> The new PB code is sloppy with buffers and makes several needless copies.
> This increases GC time a lot. A few simple changes can cut this back down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548833#comment-13548833 ] Devaraj Das commented on HBASE-5945: If we want to support BufferChain kind of request data for request as well, we'd need to put in a field in the requestheader to indicate as such to the server.. (thinking aloud..). > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, buffer-copies.txt, even-fewer-copies.txt, > hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13547049#comment-13547049 ] Devaraj Das commented on HBASE-5945: I swear that I had generated the patch correctly but it seems like I uploaded the incorrect one again.. I'll update the patch shortly with the comments. Thanks for looking, [~stack]. > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 5945-in-progress.2.patch, 5945-in-progress.patch, > buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13546671#comment-13546671 ] stack commented on HBASE-5945: -- Looked at Todds hackery again. Its lovely. I'd say lets try get it in. Looking at DD's patch, removing special response and request types looks right sending the request and response Messages instead w/ method name stuff moved up into header (other changes are coming down the pipe I'd say; e.g. hbase-7479 removes client protocol version altogether and error should be subsumed by response moved from response header .. but for later). EmptyMsg looks like it already exists. Is this saving a buffer copy or is it just doing what writeDelimitedTo does? - call.param.writeDelimitedTo(this.out); + CodedOutputStream cos = CodedOutputStream.newInstance(out,1); + cos.writeRawVarint32(call.rpcRequest.requestArg.getSerializedSize()); + call.rpcRequest.requestArg.writeTo(cos); + cos.flush(); There seem to be classes missing from the patch, ClientSideRpcRequest? The Todd BufferChain looks nice. Might as well use it if he has put up the code (and a test). We'll want it for sure when we need to pass encoded data blocks after protobuf request/response (HBASE-7233) > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 5945-in-progress.2.patch, 5945-in-progress.patch, > buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[
https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13544570#comment-13544570
]
Hadoop QA commented on HBASE-5945:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563396/5945-in-progress.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3873//console
This message is automatically generated.
> Reduce buffer copies in IPC server response path
>
>
> Key: HBASE-5945
> URL: https://issues.apache.org/jira/browse/HBASE-5945
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 0.96.0
>Reporter: Todd Lipcon
>Assignee: stack
>Priority: Blocker
> Fix For: 0.96.0
>
> Attachments: 5945-in-progress.2.patch, 5945-in-progress.patch,
> buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt
>
>
> The new PB code is sloppy with buffers and makes several needless copies.
> This increases GC time a lot. A few simple changes can cut this back down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538860#comment-13538860 ] Devaraj Das commented on HBASE-5945: [~stack], I am up for helping out with this. I'll try to do some analysis as well and get back... I was planning to do this as part of HBASE-6614. > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537592#comment-13537592 ] Todd Lipcon commented on HBASE-5945: Was reminded of this JIRA today. Anyone feel like picking it up? I'm a delinquent HBase contributor these days, so probably better if someone steals it from me (sorry guys!) > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Attachments: buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277639#comment-13277639 ] Hadoop QA commented on HBASE-5945: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525785/hbase-5495.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1908//console This message is automatically generated. > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: ipc >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Attachments: buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269722#comment-13269722 ] stack commented on HBASE-5945: -- /me hearts todd > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: ipc >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Attachments: buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269213#comment-13269213 ] Todd Lipcon commented on HBASE-5945: Yep, I think the next improvement is to use thread-local or connection-local buffers - perhaps even direct buffers. There's still one more copy from the heap byte buffer to a direct buffer within the SocketChannel.write() which would be avoided if we serialized directly into a DirectByteBuffer. Have to benchmark whether that's worth it. We should also do the same hacks on the client side of the connection - should get similar gains. > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: ipc >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Attachments: buffer-copies.txt, even-fewer-copies.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[
https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269128#comment-13269128
]
stack commented on HBASE-5945:
--
Looks great Todd.
Could we keep this DataOutputBuffer for reuse?
{code}
+ DataOutputBuffer buf = new DataOutputBuffer(size);
{code}
> Reduce buffer copies in IPC server response path
>
>
> Key: HBASE-5945
> URL: https://issues.apache.org/jira/browse/HBASE-5945
> Project: HBase
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 0.96.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Attachments: buffer-copies.txt, even-fewer-copies.txt
>
>
> The new PB code is sloppy with buffers and makes several needless copies.
> This increases GC time a lot. A few simple changes can cut this back down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[
https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268863#comment-13268863
]
Zhihong Yu commented on HBASE-5945:
---
Interesting finding.
Can I assume that the number of occurrences of Message's is higher than that of
Result's ?
If so, the following check can be lifted above that for Result.class:
{code}
+if (instance instanceof Message) {
{code}
> Reduce buffer copies in IPC server response path
>
>
> Key: HBASE-5945
> URL: https://issues.apache.org/jira/browse/HBASE-5945
> Project: HBase
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 0.96.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Minor
> Attachments: buffer-copies.txt
>
>
> The new PB code is sloppy with buffers and makes several needless copies.
> This increases GC time a lot. A few simple changes can cut this back down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
