Ah, that is Hadoop logging, which is everywhere. Check where you can configure Hadoop's logging. M.
-----Original message----- > From:shubham.gupta <[email protected]> > Sent: Thursday 25th August 2016 6:27 > To: [email protected] > Subject: Re: Application creating huge amount of logs : Nutch 2.3.1 + Hadoop > 2.7.1 > > Hey > > Logs are created when spills of map job are created during the FETCH job > and are stored in /home/hadoop/nodelogs/usercache/root/appcache. The > total size of logs sums up to over 13GB which occupies a lot of disk > space of the datanode and I have to delete those logs for smooth > functioning of nutch. > > Also, I am unclear as to which parameter should be changed in the > log4j.properties to reduce this size. > > Shubham Gupta > > On 08/24/2016 05:20 PM, Markus Jelsma wrote: > > If it is Nutch logging, change its level in conf/log4j.properties. It can > > also be Hadoop logging. > > M. > > > > -----Original message----- > >> From:shubham.gupta <[email protected]> > >> Sent: Tuesday 23rd August 2016 8:15 > >> To: [email protected] > >> Subject: Application creating huge amount of logs : Nutch 2.3.1 + Hadoop > >> 2.7.1 > >> > >> Hey > >> > >> I have integrated Nutch 2.3.1 with Hadoop 2.7.1, and the fetcher.parse > >> property is set TRUE and the database used is MongoDB. While the map job > >> of nutch runs, it creates a huge size of nodelogs over 13GB in size. And > >> the cause of such huge amount of files in unknown. Any suggestion would > >> help. > >> > >> Thanks in advance. > >> > >> Shubham Gupta > >> > >
