[
https://issues.apache.org/jira/browse/TEZ-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167433#comment-15167433
]
Rohini Palaniswamy edited comment on TEZ-3140 at 2/25/16 4:56 PM:
------------------------------------------------------------------
Avoided the need for a intermediate byte array by directly writing into the
DataOutputBuffer used by Hadoop. DataOutputBuffer is
{code}
@InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
@InterfaceStability.Unstable
{code}
but we use it directly in Tez in lots of places. So I hope that is fine. Added
an else clause logic just for completeness sake and changed it to reuse a 8K
buffer.
was (Author: rohini):
Avoided the need for a intermediate byte array by directly writing into the
DataOutputBuffer used by Hadoop. DataOutputBuffer is
@InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
@InterfaceStability.Unstable
but we use it directly in Tez in lots of places. So I hope that is fine. Added
an else clause logic just for completeness sake and changed it to reuse a 4K
buffer.
> Reduce AM memory usage while serialization
> ------------------------------------------
>
> Key: TEZ-3140
> URL: https://issues.apache.org/jira/browse/TEZ-3140
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.7.1, 0.8.3
>
> Attachments: TEZ-3140-1.patch
>
>
> There is an unnecessary copy of userpayload byte array during
> serialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
