Rohini Palaniswamy commented on TEZ-394:

Have been running into resource issues and slowness in some jobs because of 
this. In one particular job, where dataset A required 100 tasks and dataset X 
required 35000 tasks , the scheduler ran dataset X first which took ~2 hrs and 
then dataset A. This caused a lot of delay for the whole pipeline compared to 

> Better scheduling for uneven DAGs
> ---------------------------------
>                 Key: TEZ-394
>                 URL: https://issues.apache.org/jira/browse/TEZ-394
>             Project: Apache Tez
>          Issue Type: Sub-task
>            Reporter: Rohini Palaniswamy
>   Consider a series of joins or group by on dataset A with few datasets that 
> takes 10 hours followed by a final join with a dataset X. The vertex that 
> loads dataset X will be one of the top vertexes and initialized early even 
> though its output is not consumed till the end after 10 hours. 
> 1) Could either use delayed start logic for better resource allocation
> 2) Else if they are started upfront, need to handle failure/recovery cases 
> where the nodes which executed the MapTask might have gone down when the 
> final join happens. 

This message was sent by Atlassian JIRA

Reply via email to