[jira] [Commented] (YARN-8818) Yarn log aggregation of spark streaming job
[ https://issues.apache.org/jira/browse/YARN-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689796#comment-16689796 ] Szilard Nemeth commented on YARN-8818: -- Hi [~ayushChauhan]! Could you please elaborate on this a little bit more, making analyzing the issue more easier? # What was the symptoms you see? What was not working? # Could you please provide a way to reproduce the issue? How did you start a Spark job? Thanks! > Yarn log aggregation of spark streaming job > --- > > Key: YARN-8818 > URL: https://issues.apache.org/jira/browse/YARN-8818 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ayush Chauhan >Assignee: Szilard Nemeth >Priority: Major > > By default, YARN aggregates logs after an application completes. But I am > trying to aggregate logs for spark streaming job which in theory will run > forever. I have set the following properties for log aggregation and > restarted yarn by restarting {{hadoop-yarn-nodemanager}} for core & task > nodes and {{hadoop-yarn-resourcemanager}} for master node on my emr cluster. > I can view my changes in [http://node-ip:8088/conf]. > {noformat} > yarn.log-aggregation-enable => true{noformat} > {noformat} > yarn.log-aggregation.retain-seconds => 172800{noformat} > {noformat} > yarn.log-aggregation.retain-check-interval-seconds => -1 {noformat} > {noformat} > yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds => > 3600{noformat} > All the articles and resources have only mentioned to include > {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} > property and yarn will start aggregating logs for running jobs. But it is not > working in my case. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8818) Yarn log aggregation of spark streaming job
[ https://issues.apache.org/jira/browse/YARN-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627341#comment-16627341 ] Ayush Chauhan commented on YARN-8818: - [~snemeth], Please take this and help me solve this issue > Yarn log aggregation of spark streaming job > --- > > Key: YARN-8818 > URL: https://issues.apache.org/jira/browse/YARN-8818 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ayush Chauhan >Priority: Major > > By default, YARN aggregates logs after an application completes. But I am > trying to aggregate logs for spark streaming job which in theory will run > forever. I have set the following properties for log aggregation and > restarted yarn by restarting {{hadoop-yarn-nodemanager}} for core & task > nodes and {{hadoop-yarn-resourcemanager}} for master node on my emr cluster. > I can view my changes in [http://node-ip:8088/conf]. > {noformat} > yarn.log-aggregation-enable => true{noformat} > {noformat} > yarn.log-aggregation.retain-seconds => 172800{noformat} > {noformat} > yarn.log-aggregation.retain-check-interval-seconds => -1 {noformat} > {noformat} > yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds => > 3600{noformat} > All the articles and resources have only mentioned to include > {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} > property and yarn will start aggregating logs for running jobs. But it is not > working in my case. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8818) Yarn log aggregation of spark streaming job
[ https://issues.apache.org/jira/browse/YARN-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627337#comment-16627337 ] Szilard Nemeth commented on YARN-8818: -- Hi [~ayushChauhan]! Do you mind if I take this? > Yarn log aggregation of spark streaming job > --- > > Key: YARN-8818 > URL: https://issues.apache.org/jira/browse/YARN-8818 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ayush Chauhan >Priority: Major > > By default, YARN aggregates logs after an application completes. But I am > trying to aggregate logs for spark streaming job which in theory will run > forever. I have set the following properties for log aggregation and > restarted yarn by restarting {{hadoop-yarn-nodemanager}} for core & task > nodes and {{hadoop-yarn-resourcemanager}} for master node on my emr cluster. > I can view my changes in [http://node-ip:8088/conf]. > {noformat} > yarn.log-aggregation-enable => true{noformat} > {noformat} > yarn.log-aggregation.retain-seconds => 172800{noformat} > {noformat} > yarn.log-aggregation.retain-check-interval-seconds => -1 {noformat} > {noformat} > yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds => > 3600{noformat} > All the articles and resources have only mentioned to include > {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} > property and yarn will start aggregating logs for running jobs. But it is not > working in my case. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org