[ https://issues.apache.org/jira/browse/PIG-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-3251: ------------------------------ Attachment: pig-3251-trunk-v01.patch In Bzip2TextInputFormat, it says {code} /** * Provide a bridge to get the bytes from the ByteArrayOutputStream without * creating a new byte array. */ private static class TextStuffer extends OutputStream { {code} However, in reality, Text just creates a new bytearray and copy the content. Attaching a patch that is similar to the approach taken by org.apache.hadoop.util.LineReader but with less changes since HADOOP-4012(added in 0.21) was a huge patch. This patch basically reads into the fixed-length-buffer and appends to Text whenever it gets full. Touching BZip2LineRecordReader makes me nervous so I wanted the changes to be small. I need to do more testings to see if this approach works or not. > Bzip2TextInputFormat requires double the memory of maximum record size > ---------------------------------------------------------------------- > > Key: PIG-3251 > URL: https://issues.apache.org/jira/browse/PIG-3251 > Project: Pig > Issue Type: Improvement > Reporter: Koji Noguchi > Assignee: Koji Noguchi > Priority: Minor > Attachments: pig-3251-trunk-v01.patch > > > While looking at user's OOM heap dump, noticed that pig's > Bzip2TextInputFormat consumes memory at both > Bzip2TextInputFormat.buffer (ByteArrayOutputStream) > and actual Text that is returned as line. > For example, when having one record with 160MBytes, buffer was 268MBytes and > Text was 160MBytes. > We can probably eliminate one of them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira