[
https://issues.apache.org/jira/browse/TEZ-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-3831:
---------------------------------
Attachment: TEZ-3831.001.patch
This approach assumes that we can avoid adding empty or null fetched inputs
into the completed inputs blocking queue. The inputs ready trigger is still
flagged in case all inputs are empty.
[~rohini], I think this is the scenario we were looking at earlier. [~ebadger],
can you have a look at this patch to see what I have missed.
> Reduce Unordered memory needed for storing empty completed events
> -----------------------------------------------------------------
>
> Key: TEZ-3831
> URL: https://issues.apache.org/jira/browse/TEZ-3831
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Attachments: Screen Shot 2017-09-13 at 4.55.11 PM.png,
> TEZ-3831.001.patch
>
>
> the completedInputs blocking queue is used to store inputs for the
> UnorderedKVReader to consume. With Auto-reduce parallelism enabled and nearly
> all empty inputs, the reader can't prune the empty events from the blocking
> queue fast enough to keep up. In my scenario, an OOM occurred.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
