This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 055bf8e [SPARK-24938][CORE] Prevent Netty from using onheap memory for headers without regard for configuration 055bf8e is described below commit 055bf8ea1fe8dd4109772a282909ac3d69627f1b Author: Nihar Sheth <niharrsh...@gmail.com> AuthorDate: Tue Jan 22 08:41:42 2019 -0600 [SPARK-24938][CORE] Prevent Netty from using onheap memory for headers without regard for configuration ## What changes were proposed in this pull request? In MessageEncoder.java, the header would always be allocated on onheap memory regardless of whether netty was configured to use/prefer onheap or offheap. By default this made netty allocate 16mb of onheap memory for a tiny header message. It would be more practical to use preallocated buffers. Using a memory monitor tool on a simple spark application, the following services currently allocate 16 mb of onheap memory: netty-rpc-client netty-blockTransfer-client netty-external-shuffle-client With this change, the memory monitor tool reports all three of these services as using 0 b of onheap memory. The offheap memory allocation does not increase, but more of the already-allocated space is used. ## How was this patch tested? Manually tested change using spark-memory-tool https://github.com/squito/spark-memory Closes #22114 from NiharS/nettybuffer. Lead-authored-by: Nihar Sheth <niharrsh...@gmail.com> Co-authored-by: Nihar Sheth <nsh...@cloudera.com> Signed-off-by: Sean Owen <sean.o...@databricks.com> --- .../src/main/java/org/apache/spark/network/protocol/MessageEncoder.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java b/common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java index 997f74e..06dc447 100644 --- a/common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java +++ b/common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java @@ -79,7 +79,7 @@ public final class MessageEncoder extends MessageToMessageEncoder<Message> { // sent. int headerLength = 8 + msgType.encodedLength() + in.encodedLength(); long frameLength = headerLength + (isBodyInFrame ? bodyLength : 0); - ByteBuf header = ctx.alloc().heapBuffer(headerLength); + ByteBuf header = ctx.alloc().buffer(headerLength); header.writeLong(frameLength); msgType.encode(header); in.encode(header); --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org