[jira] [Created] (TEZ-3733) Auto-Reduce Parallelism needs to consider shuffle overhead to avoid OOM

[Jonathan Eagles (JIRA)]

Jonathan Eagles created TEZ-3733:
------------------------------------

             Summary: Auto-Reduce Parallelism needs to consider shuffle 
overhead to avoid OOM
                 Key: TEZ-3733
                 URL: https://issues.apache.org/jira/browse/TEZ-3733
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Jonathan Eagles
            Assignee: Jonathan Eagles


For each fetch there is associated memory overhead per partition which can be 
enough to affect the number of tasks needed to process the data. In some cases 
the reduction can be so severe the tasks can OOM due to overhead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)