[ https://issues.apache.org/jira/browse/PIG-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772410#action_12772410 ]
Ashutosh Chauhan commented on PIG-1037: --------------------------------------- I am kinda late on this, but I would appreciate if someone can provide brief description of how this patch improves the memory layout and alleviates the spill problem. I took a quick look at the patch. According to my understanding, previously when memory is about to get exhausted Pig will start writing to the disk one tuple at a time. With this new patch, once the memory limit is hit whole bag is spilled to disk, at that point in-memory bag contains no tuples. If in-memory bag fills again, all of its content are spilled to disk in entirety again and so on.. So this patch ensures that we are not spilling one tuple at a time, but a full bag a time. Is this correct or am I missing something ? > better memory layout and spill for sorted and distinct bags > ----------------------------------------------------------- > > Key: PIG-1037 > URL: https://issues.apache.org/jira/browse/PIG-1037 > Project: Pig > Issue Type: Improvement > Reporter: Olga Natkovich > Assignee: Ying He > Fix For: 0.6.0 > > Attachments: PIG-1037.patch, PIG-1037.patch2, PIG-1037.patch3 > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.