[
https://issues.apache.org/jira/browse/PIG-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695549#comment-14695549
]
Rohini Palaniswamy commented on PIG-1472:
-----------------------------------------
Actually did some more verification and looks like the TINY, SMALL approach is
better than WritableUtils.writeVInt. Current uses unsigned max values for byte
and short (255 and 65535) and is able to represent better till 65535 with
lesser bytes than WritableUtils.writeVInt. And most of the data will fall into
that category. Also WritableUtils.writeVInt uses 2 bytes and 3 bytes for byte
and short respectively as 1 byte takes up length for some ranges. For eg: 32767
uses 3 bytes and not 2. So better to leave it at the current approach. One
thing that might be advantageous though is use WritableUtils.writeVLong to
serialize LONG instead of out.writeLong(). Though for values >= Math.pow(2,
56) it uses 9 bytes, for val > Math.pow(2, 32) and val < Math.pow(2, 48) it
uses 5 to 7 bytes which is good. timestamps which is the most used long uses 7
bytes instead of 8. Apart from the byte saving need to see the time taken to
serialize and deserialize to see if it is really advantageous.
> Optimize serialization/deserialization between Map and Reduce and between MR
> jobs
> ---------------------------------------------------------------------------------
>
> Key: PIG-1472
> URL: https://issues.apache.org/jira/browse/PIG-1472
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.8.0
>
> Attachments: PIG-1472.2.patch, PIG-1472.3.patch, PIG-1472.4.patch,
> PIG-1472.patch
>
>
> In certain types of pig queries most of the execution time is spent in
> serializing/deserializing (sedes) records between Map and Reduce and between
> MR jobs.
> For example, if PigMix queries are modified to specify types for all the
> fields in the load statement schema, some of the queries (L2,L3,L9, L10 in
> pigmix v1) that have records with bags and maps being transmitted across map
> or reduce boundaries run a lot longer (runtime increase of few times has been
> seen.
> There are a few optimizations that have shown to improve the performance of
> sedes in my tests -
> 1. Use smaller number of bytes to store length of the column . For example if
> a bytearray is smaller than 255 bytes , a byte can be used to store the
> length instead of the integer that is currently used.
> 2. Instead of custom code to do sedes on Strings, use DataOutput.writeUTF and
> DataInput.readUTF. This reduces the cost of serialization by more than 1/2.
> Zebra and BinStorage are known to use DefaultTuple sedes functionality. The
> serialization format that these loaders use cannot change, so after the
> optimization their format is going to be different from the format used
> between M/R boundaries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
