Philippe, What does a perf top show?
Do you see get_task_cred or audit_filter_rules as high consumers? If they are high, then try turning off the monitoring of the /tmp, /dev/shm and /var/lock/lvm trees or if appropriate, switch to monitoring via a path directive if you don't need to monitor the entire tree. Steve, Paul, I have yet to put together a bug report, or researched to see if the problem exists upstream, but have discovered recursive directory rules can be expensive on the kernel. The rules below on a system running rabbitmq can see get_task_cred and audit_filter_rules above 10% each. -w /etc/pam.d -p wa -k PAM_Mods -w /boot -k BOOT_Mods -w /boot/grub/grub.conf -p war -k BOOT_Mods -w /etc/security -p wa -k Security_Mods -w /etc/sysconfig -p wa -k Sysconfig_Mods -w /etc/ld.so.conf.d -p wa -k Library_Mods -w /etc/inittab -p wa -k StartUp_Mods -w /etc/rc.d -p wa -k StartUp_Mods Regards On Tue, 2016-03-01 at 09:14 -0500, Steve Grubb wrote: > On Tuesday, March 01, 2016 02:57:45 PM Maupertuis Philippe wrote: > > The kernel is : 2.6.32-573.12.1.el6.x86_64 > > And the whole audit.rules file is : > > <snip> > > > During the hour preceding the fence we got these events from the passive > > node Key Summary Report > > =========================== > > total key > > =========================== > > 891 system_commands (ping) > > > > And on the active node : > > > > Key Summary Report > > =========================== > > total key > > =========================== > > 1330 system_commands > > 286 deletion > > > > I am going to follow your advice and to open a call with redhat. > > Anyway, I am interested in knowing if auditd has been reported to cause > > trouble without generating many events. > > Those numbers work out to 27 events per minute. That's not really a lot of > events. To see if its the rules or auditd causing the iowait, you might set > the logging format to NOLOG. This will discard events rather than log them. > If > you still have iowait, its something to do with the rules. If that cleared it > up, then auditd might be the source. Either way, put the format back to raw. > > I did some benchmarking of auditd over the holidays and posted some results > here: > > https://www.redhat.com/archives/linux-audit/2015-December/msg00061.html > > I'd recommend: > > flush = incremental > freq = 100 > > for a modest performance improvement. > > -Steve > > > > -----Message d'origine----- > > De : Paul Moore [mailto:[email protected]] > > Envoyé : mardi 1 mars 2016 14:25 > > À : Maupertuis Philippe > > Cc : [email protected] > > Objet : Re: auditd and redhat cluster > > > > On Mon, Feb 29, 2016 at 7:45 AM, Maupertuis Philippe > <[email protected]> wrote: > > > Hi list, > > > > > > One clusters fenced the passive node around two hours after auditd > > > was started. > > > > > > We have found that iowait has increased since auditd was started and > > > was unusually high. > > > > > > Auditd wasn’t generating many messages and there were no noticeable > > > added activity on the disk were the audit and syslog files were written. > > > > > > Besides watches, the only general rules were : > > > > > > # creation > > > -a exit,always -F arch=b32 -S creat -S mkdir -S mknod -S link -S > > > symlink -S mkdirat -S mknodat -S linkat -S symlinkat -F uid=root -F > > > success=1 -k creation -a exit,always -F arch=b64 -S creat -S mkdir -S > > > mknod -S link -S symlink -S mkdirat -S mknodat -S linkat -S symlinkat > > > -F uid=root -F success=1 -k creation > > > > > > # deletion > > > -a exit,always -F arch=b32 -S rmdir -S unlink -S unlinkat -F uid=root > > > -F > > > success=1 -k deletion > > > -a exit,always -F arch=b64 -S rmdir -S unlink -S unlinkat -F uid=root > > > -F > > > success=1 -k deletion > > > > > > After the rebot we deleted all rules and didn’t notice extra iowait > > > anymore. > > > > > > Could these rules be the cause of additional iowait even if not > > > generating many events (around 20 in two hours) ? > > > > > > Is there any other auditd mechanism that could explain this phenomenon ? > > > > > > I would appreciate any hints. > > > > Hi Philippe, > > > > First, as this is a RH cluster product, I would suggest contacting RH > > support with your question if you haven't already; this list is primarily > > for upstream development and support. > > > > If you are able to experiment with the system, or have a test environment, I > > would suggest trying to narrow down the list of audit rules/watches to see > > which rules/watches have the most affect on the iowait times. You've > > listed four rules, but you didn't list the watches you have configured. > > Also, what kernel version are you using? > > > > -- > > paul moore > > www.paul-moore.com > > > > !!!************************************************************************* > > ************ "Ce message et les pièces jointes sont confidentiels et > > réservés à l'usage exclusif de ses destinataires. Il peut également être > > protégé par le secret professionnel. Si vous recevez ce message par erreur, > > merci d'en avertir immédiatement l'expéditeur et de le détruire. > > L'intégrité du message ne pouvant être assurée sur Internet, la > > responsabilité de Worldline ne pourra être recherchée quant au contenu de > > ce message. Bien que les meilleurs efforts soient faits pour maintenir > > cette transmission exempte de tout virus, l'expéditeur ne donne aucune > > garantie à cet égard et sa responsabilité ne saurait être recherchée pour > > tout dommage résultant d'un virus transmis. > > > > This e-mail and the documents attached are confidential and intended solely > > for the addressee; it may also be privileged. If you receive this e-mail in > > error, please notify the sender immediately and destroy it. As its > > integrity cannot be secured on the Internet, the Worldline liability cannot > > be triggered for the message content. Although the sender endeavours to > > maintain a computer virus-free network, the sender does not warrant that > > this transmission is virus-free and will not be liable for any damages > > resulting from any virus transmitted.!!!" > > > > -- > > Linux-audit mailing list > > [email protected] > > https://www.redhat.com/mailman/listinfo/linux-audit > > > -- > Linux-audit mailing list > [email protected] > https://www.redhat.com/mailman/listinfo/linux-audit -- Linux-audit mailing list [email protected] https://www.redhat.com/mailman/listinfo/linux-audit
