[
https://issues.apache.org/jira/browse/TEZ-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168581#comment-15168581
]
Siddharth Seth commented on TEZ-3140:
-------------------------------------
The main intent here is to avoid creating huge buffers while sending data to
tasks - use the buffer created by the RPC layer itself, otherwise restrict to
8K at max ?
We really should move this over to the Protobuf engine - though that may end up
forcing the buffer creation as well.
The changes look good to me.
Minor: SERIALIZE_BUFFER_SIZE, ByteBufferDataInput.bb should be final
I think it's worthwhile to add a unit test for the serialization loop - a
simple test to write an EntityDescriptor with a payload less than 8K and
another > 8K. ByteBuffer changes have caused problems in the past.
> Reduce AM memory usage while serialization
> ------------------------------------------
>
> Key: TEZ-3140
> URL: https://issues.apache.org/jira/browse/TEZ-3140
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.7.1, 0.8.3
>
> Attachments: TEZ-3140-1.patch
>
>
> There is an unnecessary copy of userpayload byte array during
> serialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
