[
https://issues.apache.org/jira/browse/PIG-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-3251:
------------------------------
Attachment: pig-3251-trunk-v05.patch
Thanks Daniel.
bq.is the patch ready?
Ah, forgot to flag it as patch available.
bq. can we just cache splittable?
Makes complete sense. Changing.
bq. Is it possible to wrap a codec deal with both bz2/bz?
As far as I understand, hadoop has 1-to-1 mapping for the codec and extension.
I don't know of a way to map multiple extensions to one codec.
Or, are you suggesting I create two silly wrappers instead of one?
> Bzip2TextInputFormat requires double the memory of maximum record size
> ----------------------------------------------------------------------
>
> Key: PIG-3251
> URL: https://issues.apache.org/jira/browse/PIG-3251
> Project: Pig
> Issue Type: Improvement
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Attachments: pig-3251-trunk-v01.patch, pig-3251-trunk-v02.patch,
> pig-3251-trunk-v03.patch, pig-3251-trunk-v04.patch, pig-3251-trunk-v05.patch
>
>
> While looking at user's OOM heap dump, noticed that pig's
> Bzip2TextInputFormat consumes memory at both
> Bzip2TextInputFormat.buffer (ByteArrayOutputStream)
> and actual Text that is returned as line.
> For example, when having one record with 160MBytes, buffer was 268MBytes and
> Text was 160MBytes.
> We can probably eliminate one of them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
