Ayush Chauhan created YARN-8818:
-----------------------------------
Summary: Yarn log aggregation of spark streaming job
Key: YARN-8818
URL: https://issues.apache.org/jira/browse/YARN-8818
Project: Hadoop YARN
Issue Type: Bug
Reporter: Ayush Chauhan
By default, YARN aggregates logs after an application completes. But I am
trying to aggregate logs for spark streaming job which in theory will run
forever. I have set the following properties for log aggregation and restarted
yarn by restarting {{hadoop-yarn-nodemanager}} for core & task nodes and
{{hadoop-yarn-resourcemanager}} for master node on my emr cluster. I can view
my changes in [http://node-ip:8088/conf].
{noformat}
yarn.log-aggregation-enable => true{noformat}
{noformat}
yarn.log-aggregation.retain-seconds => 172800{noformat}
{noformat}
yarn.log-aggregation.retain-check-interval-seconds => -1 {noformat}
{noformat}
yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds =>
3600{noformat}
All the articles and resources have only mentioned to include
{{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} property
and yarn will start aggregating logs for running jobs. But it is not working in
my case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
