[
https://issues.apache.org/jira/browse/TEZ-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-3831:
---------------------------------
Attachment: TEZ-3831.001-addendum.patch
Missed a case needed to wake up consumer thread in blocking take call. In this
case adding a poison pill end of input message to prevent hang in the case
where all input after take is called is NullFetchedInput and so isn't added to
blocking queue.
Instead of adding a new EndOfInputFetchedInput, I reused NullFetchedInput. Let
me know if it is worth it to add this separate class and work to support it.
> Reduce Unordered memory needed for storing empty completed events
> -----------------------------------------------------------------
>
> Key: TEZ-3831
> URL: https://issues.apache.org/jira/browse/TEZ-3831
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Fix For: 0.9.1
>
> Attachments: Screen Shot 2017-09-13 at 4.55.11 PM.png,
> TEZ-3831.001-addendum.patch, TEZ-3831.001.patch
>
>
> the completedInputs blocking queue is used to store inputs for the
> UnorderedKVReader to consume. With Auto-reduce parallelism enabled and nearly
> all empty inputs, the reader can't prune the empty events from the blocking
> queue fast enough to keep up. In my scenario, an OOM occurred.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
