Dmitry Sivachenko created YARN-3788:

             Summary: Application Master and Task Tracker timeouts are applied 
                 Key: YARN-3788
             Project: Hadoop YARN
          Issue Type: Bug
    Affects Versions: 2.4.1
            Reporter: Dmitry Sivachenko

I am running a streaming job which requires a big (~50GB) data file to run 
(file is attached via hadoop jar <...> -file BigFile.dat).

Most likely this command will fail as follows (note that error message is 
rather meaningless):
2015-05-27 15:55:00,754 WARN  [main] streaming.StreamJob 
( - -file option is deprecated, please use 
generic option -files instead.
packageJobJar: [/ssd/mt/lm/en_reorder.ylm,, 
/tmp/hadoop-mitya/hadoop-unjar3778165585140840383/] [] 
/var/tmp/streamjob633547925483233845.jar tmpDir=null
2015-05-27 19:46:22,942 INFO  [main] client.RMProxy 
( - Connecting to ResourceManager at
2015-05-27 19:46:23,733 INFO  [main] client.RMProxy 
( - Connecting to ResourceManager at
2015-05-27 20:13:37,231 INFO  [main] mapred.FileInputFormat 
( - Total input paths to process : 1
2015-05-27 20:13:38,110 INFO  [main] mapreduce.JobSubmitter 
( - number of splits:1
2015-05-27 20:13:38,136 INFO  [main] Configuration.deprecation 
( - mapred.reduce.tasks is 
deprecated. Instead, use mapreduce.job.reduces
2015-05-27 20:13:38,390 INFO  [main] mapreduce.JobSubmitter 
( - Submitting tokens for job: 
2015-05-27 20:13:38,689 INFO  [main] impl.YarnClientImpl 
( - Submitted application 
2015-05-27 20:13:38,743 INFO  [main] mapreduce.Job ( - 
The url to track the job:
2015-05-27 20:13:38,746 INFO  [main] mapreduce.Job 
( - Running job: job_1431704916575_2531
2015-05-27 21:04:12,353 INFO  [main] mapreduce.Job 
( - Job job_1431704916575_2531 running in 
uber mode : false
2015-05-27 21:04:12,356 INFO  [main] mapreduce.Job 
( - map 0% reduce 0%
2015-05-27 21:04:12,374 INFO  [main] mapreduce.Job 
( - Job job_1431704916575_2531 failed with 
state FAILED due to: Application application_1431704916575_2531 failed 2 times 
due to ApplicationMaster for attempt appattempt_1431704916575_2531_000002 timed 
out. Failing the application.
2015-05-27 21:04:12,473 INFO  [main] mapreduce.Job 
( - Counters: 0
2015-05-27 21:04:12,474 ERROR [main] streaming.StreamJob 
( - Job not Successful!
Streaming Command Failed!

This is because (defaults to 600 
sec) timeout expires before large data file is transferred.

Next step I increase  After that 
application is successfully initialized and tasks are spawned.

But I encounter another error: the default 600 seconds mapreduce.task.timeout 
expires before tasks are initialized and tasks fail.

Error message Task attempt_XXX failed to report status for 600 seconds is also 
misleading: this timeout is supposed to kill non-responsive (stuck) tasks but 
it rather strikes because auxiliary data files are copying slowly.

So I need to increase mapreduce.task.timeout too and only after that my job is 

At the very least error messages need to be tweaked to indicate that 
Application (or Task) is failing because auxiliary files are not copied during 
that time, not just generic "timeout expired".

Better solution would be not to account time spent for data files distribution.

This message was sent by Atlassian JIRA

Reply via email to