[
https://issues.apache.org/jira/browse/PIG-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-3251:
------------------------------
Attachment: pig-3251-trunk-v07.patch
Sorry for the delay.
{quote}
Then I realized that for the issue I previously reported on this jira,
testBlockHeaderEndingWithCR and testBlockHeaderEndingAtSplitNotByteAligned
failing, only the former is fixed in MAPREDUCE-5656 but not the latter.
{quote}
Spent some time only to remember that [~jlowe] did look at both test failures
at that time and was only able to reproduce the failure on one but not on
testBlockHeaderEndingAtSplitNotByteAligned.
Looking back again, I now understand that I had some ctag file which was
corrupting the test result for this specific test.
Fixing my test env did make the test successful.
bq. public static final String PIG_BZIP_USE_HADOOP_INPUTFORMAT =
"pig.bzip.use.hadoop.inputformat";
Changed.
Also changed the TestBZip.testCount method so that it runs with 2 mappers(2
splits) to make sure we're testing the right condition.
I'll run it through full unit/e2e test shortly.
> Bzip2TextInputFormat requires double the memory of maximum record size
> ----------------------------------------------------------------------
>
> Key: PIG-3251
> URL: https://issues.apache.org/jira/browse/PIG-3251
> Project: Pig
> Issue Type: Improvement
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: pig-3251-trunk-v01.patch, pig-3251-trunk-v02.patch,
> pig-3251-trunk-v03.patch, pig-3251-trunk-v04.patch, pig-3251-trunk-v05.patch,
> pig-3251-trunk-v06.patch, pig-3251-trunk-v07.patch
>
>
> While looking at user's OOM heap dump, noticed that pig's
> Bzip2TextInputFormat consumes memory at both
> Bzip2TextInputFormat.buffer (ByteArrayOutputStream)
> and actual Text that is returned as line.
> For example, when having one record with 160MBytes, buffer was 268MBytes and
> Text was 160MBytes.
> We can probably eliminate one of them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
