Koji Noguchi created PIG-3251:
---------------------------------
Summary: Bzip2TextInputFormat requires double the memory of
maximum record size
Key: PIG-3251
URL: https://issues.apache.org/jira/browse/PIG-3251
Project: Pig
Issue Type: Improvement
Reporter: Koji Noguchi
Assignee: Koji Noguchi
Priority: Minor
While looking at user's OOM heap dump, noticed that pig's Bzip2TextInputFormat
consumes memory at both
Bzip2TextInputFormat.buffer (ByteArrayOutputStream)
and actual Text that is returned as line.
For example, when having one record with 160MBytes, buffer was 268MBytes and
Text was 160MBytes.
We can probably eliminate one of them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
