[ https://issues.apache.org/jira/browse/PIG-793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724594#action_12724594 ]
Alan Gates commented on PIG-793: -------------------------------- Using jmap, I've been toying around with our DefaultTuple implementation to see how much memory it takes. For a tuple with 3 elements, one int, one double, one 20 character string I see it taking: 16 bytes for the Tuple object 24 bytes for the ArrayList<Object> in the tuple ~26 bytes for pointers in the ArrayList 16 bytes for the Integer 16 bytes for the Double 24 bytes for the String overhead ~52 bytes for the String data Pointers in the ArrayList and character data in the String appear to be padded and vary somewhat depending on how I run the experiments. I played with changing the ArrayList<Object> in DefaultTuple to an Object. There are two advantages, the 24 bytes of ArrayList shrinks to 12 for the Object, and as I wrote it to always have the Object be exactly the right size there is no padding cost. The downside to this is append becomes a more expensive operation because it's growing the Object by one every time. However, after some investigation I believe that most places we use append can be changed to use set, thus alieviating this issue. I'm working on a patch to change this. Once I have that done I'll report on how that changes memory usage as well as any performance gains or losses. A related item I would like to look into is using Hadoop's Text instead of String to back chararray. Text takes 16 bytes of overhead + 36 bytes for string data to store 20 characters, versus the 24 / 52 of String. Obviously this would be a huge change and needs to have very impressive results to be considered. I'll play with it and report results here. > Improving memory efficiency of Tuple implementation > --------------------------------------------------- > > Key: PIG-793 > URL: https://issues.apache.org/jira/browse/PIG-793 > Project: Pig > Issue Type: Improvement > Reporter: Olga Natkovich > Assignee: Alan Gates > > Currently, our tuple is a real pig and uses a lot of extra memory. > There are several places where we can improve memory efficiency: > (1) Laying out memory for the fields rather than using java objects since > since each object for a numeric field takes 16 bytes > (2) For the cases where we know the schema using Java arrays rather than > ArrayList. > There might be more. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.