Hi Rohith, Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us. Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory. During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?
Thanks, Smita From: Rohith Sharma K S [mailto:[email protected]] Sent: Monday, April 20, 2015 10:09 AM To: [email protected] Subject: RE: how to delete logs automatically from hadoop yarn Hi With below configuration , log deletion should be triggered. You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information. “INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds” But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured? <property> <description> Number of seconds after an application finishes before the nodemanager's DeletionService will delete the application's localized file directory and log directory. To diagnose Yarn application problems, set this property's value large enough (for example, to 600 = 10 minutes) to permit examination of these directories. After changing the property's value, you must restart the nodemanager in order for it to have an effect. The roots of Yarn applications' work directories is configurable with the yarn.nodemanager.local-dirs property (see below), and the roots of the Yarn applications' log directories is configurable with the yarn.nodemanager.log-dirs property (see also below). </description> <name>yarn.nodemanager.delete.debug-delay-sec</name> <value>0</value> </property> Thanks & Regards Rohith Sharma K S From: Sunil Garg [mailto:[email protected]] Sent: 20 April 2015 09:52 To: [email protected]<mailto:[email protected]> Subject: how to delete logs automatically from hadoop yarn How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working Is there any other way we can do this or am I doing something wrong !! <property> <name>yarn.log-aggregation-enable</name> <value>false</value> </property> <property> <name>yarn.nodemanager.log.retain-seconds</name> <value>3600</value> </property> Thanks Sunil Garg
