[ https://issues.apache.org/jira/browse/HADOOP-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17637013#comment-17637013 ]
Shilun Fan edited comment on HADOOP-18536 at 11/22/22 6:04 AM: --------------------------------------------------------------- Thank you very much for your thoughts. From my personal point of view, I think the risk of this change is high, but the benefit is very small, and it is difficult for us to see performance optimization. For hadoop-client, the default configuration is 512MB of memory, 1024 Bytes copy, basically there will be no Impact, even if our client only has 2-3MB of memory, it has basically no impact. was (Author: slfan1989): >From my personal point of view, I think the risk of this change is high, but >the benefit is very small, and it is difficult for us to see performance >optimization. For hadoop-client, the default configuration is 512MB of memory, >1024 Bytes copy, basically there will be no Impact, even if our client only >has 2-3MB of memory, it has basically no impact. > RPC Client Improvement > ---------------------- > > Key: HADOOP-18536 > URL: https://issues.apache.org/jira/browse/HADOOP-18536 > Project: Hadoop Common > Issue Type: Improvement > Components: rpc-server > Reporter: xinqiu.hu > Priority: Minor > > In the RPC Client, before a request (including RpcRequestHeaderProto, > RequestHeaderProto, Message Payload) is sent, they will be copied to the three > CodedOutputStream internal byte arrays, and then aggregated to the > ResponseBuffer > internal byte array. Then the ResponseBuffer byte array is written to a > BufferedOutputStream and finally to a SocketOutputStream. > To simplify the writing process, Maybe we can copy them directly to a big > CodedOutputStream and send them directly to the IpcStreams#out. To achieve > this, I > propose HADOOP-18533. But it brings the following two side effects. > # The generic declaration of rpcRequestQueue inside Client has been changed > to Object > # The serialization of protobuf has been moved to rpcRequestThread, because > rpcRequestThread is a single thread for each connection, which may have a > performance impact. > For the above reasons, I propose this. This pr brings the following benefits > # For each rpc request, avoid creating a ResponseBuffer of 1024 bytes > # For each rpc request, reduce one copy > # For each rpc request, combine the three fragmented CodedOutputStreams into > one > # No side effects like HADOOP-18533 -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org