Hitesh Shah commented on TEZ-3479:

Atleast for this scenario, I think we did not recover 
task_1476667862449_0031_1_07_000004 properly to a failed state which ends up 
leading to a hang as the vertex cannot complete.

2016-10-18 07:06:24,837 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: 
Task Completion: vertex_1476667862449_0031_1_07 [Map 3], tasks=29, failed=1, 
killed=24, success=3, completed=28, commits=0, err=OWN_TASK_FAILURE 

> DAG AM does not schedule any more containers in corner cases
> ------------------------------------------------------------
>                 Key: TEZ-3479
>                 URL: https://issues.apache.org/jira/browse/TEZ-3479
>             Project: Apache Tez
>          Issue Type: Improvement
>    Affects Versions: 0.7.1
>            Reporter: Rajesh Balamohan
>         Attachments: application_1476667862449_0031_not_complete.1.log.tar.gz
> Env: 3 node AWS cluster with data residing in S3. Tez version is 0.7.
> Some workloads end up generating lots of data that the tasks start throwing 
> "No space available" in local disks (e.g Q29 in TPCDS). DAG should fail after 
> enough number of retries which happens most of the time. Once in a while (~ 
> once in 20-30 runs), DAG AM gets into hung state and does not schedule any 
> more containers for the failed task attempts. Will attach the logs shortly. 

This message was sent by Atlassian JIRA

Reply via email to