[ https://issues.apache.org/jira/browse/YARN-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627341#comment-16627341 ]
Ayush Chauhan commented on YARN-8818: ------------------------------------- [~snemeth], Please take this and help me solve this issue > Yarn log aggregation of spark streaming job > ------------------------------------------- > > Key: YARN-8818 > URL: https://issues.apache.org/jira/browse/YARN-8818 > Project: Hadoop YARN > Issue Type: Bug > Reporter: Ayush Chauhan > Priority: Major > > By default, YARN aggregates logs after an application completes. But I am > trying to aggregate logs for spark streaming job which in theory will run > forever. I have set the following properties for log aggregation and > restarted yarn by restarting {{hadoop-yarn-nodemanager}} for core & task > nodes and {{hadoop-yarn-resourcemanager}} for master node on my emr cluster. > I can view my changes in [http://node-ip:8088/conf]. > {noformat} > yarn.log-aggregation-enable => true{noformat} > {noformat} > yarn.log-aggregation.retain-seconds => 172800{noformat} > {noformat} > yarn.log-aggregation.retain-check-interval-seconds => -1 {noformat} > {noformat} > yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds => > 3600{noformat} > All the articles and resources have only mentioned to include > {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} > property and yarn will start aggregating logs for running jobs. But it is not > working in my case. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org