Doh, originally just replied to Steve rather than to the list! Ross
> -----Original Message----- > From: linux-audit-boun...@redhat.com [mailto:linux-audit- > boun...@redhat.com] On Behalf Of Steve Grubb > Sent: 07 October 2010 20:51 > To: linux-audit@redhat.com > Subject: Re: Odd memory usage in auditd > > On Thursday, October 07, 2010 05:52:49 am Ross Kirk wrote: > > Has anybody got any advice for the following problem? As I'm seeing > some > > very odd behaviour with the auditd daemon in RHEL5.2 where under > heavy > > system load the auditd process doesn't free any resources until all > memory > > is consumed and the kernel kills the process with an Out Of Memory > error. > > I seem to recall something about disk flushing causing auditd to look > like its the culprit. Do you have barriers enabled on ext3? You might > also try setting the flushing to something else like none and see if > that does anything. Currently barriers are not enabled as it wasn't something I was aware of. However it does sound like something I may well want to be enabled so I will give the various flush settings a try and see if the barrier option affects anything. So if auditd is not the culprit what do you suspect the problem is? > > The system I have is a heavily customised RHEL5.2 with some fairly > > stringent auditing rules specified, the config is attached. In > addition to > > these rules there will be various SELinux AVCs being raised as well > as > > events from my own software so the auditing system is kept quite > busy, see > > the attached report.txt for the aureport summary . > > I don't see anything terribly unusual. The audit rules didn't make it, > but the backlog setting is the only thing I would be interested in > seeing. -b 16384 > > I can reproduce this behaviour consistently by generating a heavy > system > > load (CPU 100% usage) while also generating a significant number of > audit > > events. After about 20 minutes the auditd process will have grown > from 8Mb > > of memory to around 1Gb; > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > > 3037 root 16 -3 2763m 921m 16 S 3.7 91.2 0:26.49 auditd > > > > If the system is kept busy eventually auditd will consume all the > memory > > available on the system and the process be killed by the kernel with > an > > Out Of Memory error. > > Try playing with the disk flushing and let us know how that works out. > There > are no known memory leaks in recent version of auditd. I try to keep > malloc down to a minimum to prevent this and memory fragmentation to > creep in. With; * flush = none Completely different behaviour from previous, memory usage never changed for my entire test run * flush = data No increase in memory usage at all * flush = sync No increase in memory usage at all Changing the ext3 barrier setting didn't make any changes to the above results nor to the behaviour I saw when "incremental" was set. Well as the other settings are better behaved I can change to using "data" without any problems I believe. Arguably this is probably a better choice for me anyway! Thanks for your help! Ross -- Linux-audit mailing list Linux-audit@redhat.com https://www.redhat.com/mailman/listinfo/linux-audit
