[
https://issues.apache.org/jira/browse/TEZ-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14736453#comment-14736453
]
Jeff Zhang commented on TEZ-2724:
---------------------------------
Steps to reproduce this issue:
* configuration requirements:
** yarn.timeline-service.generic-application-history.enabled=false
** yarn.resourcemanager.recovery.enabled=false
** ipc.client.connect.retry.interval=5000
** ipc.client.connect.max.retries=12
* Run command: "hadoop jar tez-tests/target/tez-tests-0.8.1-SNAPSHOT.jar
mrrsleep -m 5 -r 5 -mt 20000 -rt 10000"
* Kill the AM in the middle of job running
* Check the RM UI to wait for the yarn app finished, then restart RM
> Tez Client keeps on showing old status when application is finished but RM is
> shutdown
> --------------------------------------------------------------------------------------
>
> Key: TEZ-2724
> URL: https://issues.apache.org/jira/browse/TEZ-2724
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.5.4
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Attachments: TEZ-2724-1.patch, amrecovery_mutlipleamrestart.txt
>
>
> From the logs, it seems the ipc retry interval is set as 20 seconds and ipc
> max retries is 45. This means that the client will retry the RPC connection
> for total 900 (20*45) seconds. And in this period, the application may
> already complete and RM Restarting may be triggered as said in the jira
> description. And I think the RM recovery is not enabled, so even the new RM
> is restarted, the original application info is lost, that means the client
> can never get the correct application report which makes it showing the old
> status forever.
> {code}
> 15/05/07 19:13:43 INFO ipc.Client: Retrying connect to server:
> maint22-tez12/100.79.80.19:52822. Already tried 26 time(s); maxRetries=45
> Deleted /user/hadoopqa/Input1
> RUNNING: call D:\hdp\hadoop-2.6.0.2.2.6.0-2782\bin\hdfs.cmd dfs -ls
> /user/hadoopqa/Input2
> RUNNING: call D:\hdp\hadoop-2.6.0.2.2.6.0-2782\bin\hdfs.cmd dfs -rm -r
> -skipTrash /user/hadoopqa/Input2
> 15/05/07 19:14:03 INFO ipc.Client: Retrying connect to server:
> maint22-tez12/100.79.80.19:52822. Already tried 27 time(s); maxRetries=45
> {code}
> Configuration to reproduce this issue
> * disable generic application history
> (yarn.timeline-service.generic-application-history.enabled)
> * disable rm recovery (yarn.resourcemanager.recovery.enabled)
> * increase the ipc retry interval and max retry
> (ipc.client.connect.retry.interval & ipc.client.connect.max.retries)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
