Re: Audit overhead and default rules
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 02/10/2014 04:49 PM, Andrew Lutomirski wrote:
> On Mon, Feb 10, 2014 at 1:02 PM, Steve Grubb wrote:
>> On Monday, February 10, 2014 12:41:08 PM Andrew Lutomirski wrote:
> There are, indeed, many ways for me to fix this on my machine.
> I'm suggesting that Fedora change the default so that no one has
> experiences this overhead by default.
There are 3 levels of audit performance degradation. 1) audit is
disabled. You get full speed 2) audit is enabled and no rules. This
is the default for Fedora so that more information can be collected
when AVC's occur. 3) audit is enabled and rules loaded. This does get
a performance hit that can be measured. In this case, the person
wanted auditing and is willing to take any performance hit it may
incur.
The audit system has been set for #2 for the last 8 or 9 years as a
balance between getting information for avc's, not taking a big
performance hit, and keeping setup easy for when people want to add
auditing to their system.
>>>
>>> Right. I'm proposing changing the default from #2 to #1.
>>
>> I forgot to mention option 0) audit package not installed. I don't think
>> the audit package is mandatory and that would be the default. But if you
>> do install the audit package its assumed you want auditing in some
>> capacity and are willing to take the minimal hit. You also get more audit
>> events such as promiscuous socket use, user space events, and a couple
>> other kernel events that are security related.
>>
>>> I think that #2-by-default is a terrible tradeoff. I suspect I've
>>> debugged more selinux denials than the average user, and I have *never*
>>> *once* looked at a 'syscall' entry in the log.
>>
>> The selinux people wanted the syscall event. Once upon a time, you used
>> to have to add a rule to get the syscall information. But they decided
>> they want more information by default. I would suggest reverting that
>> patch as a test. I think the problem was that they needed a file path
>> sometimes and would ask people to add an audit rule like "-w /etc/shadow
>> -p w". But then the user wouldn't get a recurrence and they couldn't
>> really fix the problem very fast. The exact details may be different, but
>> I think this is the gist of it.
>>
>
> Here's an example from my logs:
>
> type=AVC msg=audit(1383816002.656:3662): avc: denied { execute } for
> pid=32707 comm="sh" name="ldconfig" dev="dm-2" ino=172883
> scontext=system_u:system_r:smoltclient_t:s0-s0:c0.c1023
> tcontext=system_u:object_r:ldconfig_exec_t:s0 tclass=file type=SYSCALL
> msg=audit(1383816002.656:3662): arch=c03e syscall=59 success=no
> exit=-13 a0=9c0d30 a1=9c0e00 a2=9bfd70 a3=0 items=0 ppid=32706 pid=32707
> auid=999 uid=999 gid=998 euid=999 suid=999 fsuid=999 egid=998 sgid=998
> fsgid=998 ses=415 tty=(none) comm="sh" exe="/usr/bin/bash"
> subj=system_u:system_r:smoltclient_t:s0-s0:c0.c1023 key=(null)
>
> The useful things (I think) are "dev" and "ino", both of which are in the
> AVC line, not the syscall line.
>
>>
>>> I think that subjecting every Fedora user by default to 20-40 ns of
>>> extra syscall latency for the sole benefit of getting those 'syscall'
>>> messages is a bad tradeoff.
>>
>> I don't think all Fedora users have audit installed and would not see the
>> hit.
>
>> From the F20 comps:
>
> core <_name>Core <_description>Smallest possible
> installation false
> false type="mandatory">audit
>
> audit is the very first mandatory package :-/
>
>>
>>> I'm willing to write kernel code to improve the situation. The problem
>>> is that getting rid of TIF_SYSCALL_AUDIT when there are no audit rules
>>> configured is messy. Better suggestions are welcome.
>>
>> The problem is getting TIF_SYSCALL_AUDIT back in all processes when
>> auditing is enabled. We cannot stop the OS and stab that flag into all
>> processes when audit gets re-enabled. Its best not to play with that
>> flag.
>
> Sure we can. I have patches to do that. They have other problems, though,
> but that's fixable.
>
>>
>> The kernel logic is supposed to be something like
>>
>> if (tif & TIF_SYSCALL_AUDIT) if (current->audit_context) if
>> (audit_ever_enabled) audit_syscall_entry()
>>
>> So, the overhead when disabled should only be an if statement or two.
>
> On my laptop it's up to 1/3 of *total* syscall time. Linux fast-path
> syscalls are fast, and audit disables the fast path.
>
> --Andy
>
Knowing the syscall was an execute versus and open call is valuable to knowing
if this is a leaked file descriptor versus an actual piece of code opening a
file. Syscall records are also used by kernel engineers and other programmers
to figure out why a strange AVC appeared. If there was a way to collect this
information only when an AVC happened, I would be fine with it.
I added a few of the SELinux kernel engineers to get their comment.
-BEGIN PGP
Re: Audit overhead and default rules
On Mon, Feb 10, 2014 at 1:02 PM, Steve Grubb wrote:
> On Monday, February 10, 2014 12:41:08 PM Andrew Lutomirski wrote:
>> >> There are, indeed, many ways for me to fix this on my machine. I'm
>> >> suggesting that Fedora change the default so that no one has
>> >> experiences this overhead by default.
>> >
>> > There are 3 levels of audit performance degradation.
>> > 1) audit is disabled. You get full speed
>> > 2) audit is enabled and no rules. This is the default for Fedora so that
>> > more information can be collected when AVC's occur.
>> > 3) audit is enabled and rules loaded. This does get a performance hit that
>> > can be measured. In this case, the person wanted auditing and is willing
>> > to take any performance hit it may incur.
>> >
>> > The audit system has been set for #2 for the last 8 or 9 years as a
>> > balance
>> > between getting information for avc's, not taking a big performance hit,
>> > and keeping setup easy for when people want to add auditing to their
>> > system.
>>
>> Right. I'm proposing changing the default from #2 to #1.
>
> I forgot to mention option 0) audit package not installed. I don't think the
> audit package is mandatory and that would be the default. But if you do
> install the audit package its assumed you want auditing in some capacity and
> are willing to take the minimal hit. You also get more audit events such as
> promiscuous socket use, user space events, and a couple other kernel events
> that are security related.
>
>> I think that #2-by-default is a terrible tradeoff. I suspect I've debugged
>> more selinux denials than the average user, and I have *never* *once*
>> looked at a 'syscall' entry in the log.
>
> The selinux people wanted the syscall event. Once upon a time, you used to
> have to add a rule to get the syscall information. But they decided they want
> more information by default. I would suggest reverting that patch as a test. I
> think the problem was that they needed a file path sometimes and would ask
> people to add an audit rule like "-w /etc/shadow -p w". But then the user
> wouldn't get a recurrence and they couldn't really fix the problem very fast.
> The exact details may be different, but I think this is the gist of it.
>
Here's an example from my logs:
type=AVC msg=audit(1383816002.656:3662): avc: denied { execute } for
pid=32707 comm="sh" name="ldconfig" dev="dm-2" ino=172883
scontext=system_u:system_r:smoltclient_t:s0-s0:c0.c1023
tcontext=system_u:object_r:ldconfig_exec_t:s0 tclass=file
type=SYSCALL msg=audit(1383816002.656:3662): arch=c03e syscall=59
success=no exit=-13 a0=9c0d30 a1=9c0e00 a2=9bfd70 a3=0 items=0
ppid=32706 pid=32707 auid=999 uid=999 gid=998 euid=999 suid=999
fsuid=999 egid=998 sgid=998 fsgid=998 ses=415 tty=(none) comm="sh"
exe="/usr/bin/bash"
subj=system_u:system_r:smoltclient_t:s0-s0:c0.c1023 key=(null)
The useful things (I think) are "dev" and "ino", both of which are in
the AVC line, not the syscall line.
>
>> I think that subjecting every Fedora user by default to 20-40 ns of extra
>> syscall latency for the sole benefit of getting those 'syscall' messages is a
>> bad tradeoff.
>
> I don't think all Fedora users have audit installed and would not see the hit.
From the F20 comps:
core
<_name>Core
<_description>Smallest possible installation
false
false
audit
audit is the very first mandatory package :-/
>
>> I'm willing to write kernel code to improve the situation. The
>> problem is that getting rid of TIF_SYSCALL_AUDIT when there are no
>> audit rules configured is messy. Better suggestions are welcome.
>
> The problem is getting TIF_SYSCALL_AUDIT back in all processes when auditing
> is enabled. We cannot stop the OS and stab that flag into all processes when
> audit gets re-enabled. Its best not to play with that flag.
Sure we can. I have patches to do that. They have other problems,
though, but that's fixable.
>
> The kernel logic is supposed to be something like
>
> if (tif & TIF_SYSCALL_AUDIT)
> if (current->audit_context)
> if (audit_ever_enabled)
> audit_syscall_entry()
>
> So, the overhead when disabled should only be an if statement or two.
On my laptop it's up to 1/3 of *total* syscall time. Linux fast-path
syscalls are fast, and audit disables the fast path.
--Andy
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Audit overhead and default rules
On Monday, February 10, 2014 12:41:08 PM Andrew Lutomirski wrote: > >> There are, indeed, many ways for me to fix this on my machine. I'm > >> suggesting that Fedora change the default so that no one has > >> experiences this overhead by default. > > > > There are 3 levels of audit performance degradation. > > 1) audit is disabled. You get full speed > > 2) audit is enabled and no rules. This is the default for Fedora so that > > more information can be collected when AVC's occur. > > 3) audit is enabled and rules loaded. This does get a performance hit that > > can be measured. In this case, the person wanted auditing and is willing > > to take any performance hit it may incur. > > > > The audit system has been set for #2 for the last 8 or 9 years as a > > balance > > between getting information for avc's, not taking a big performance hit, > > and keeping setup easy for when people want to add auditing to their > > system. > > Right. I'm proposing changing the default from #2 to #1. I forgot to mention option 0) audit package not installed. I don't think the audit package is mandatory and that would be the default. But if you do install the audit package its assumed you want auditing in some capacity and are willing to take the minimal hit. You also get more audit events such as promiscuous socket use, user space events, and a couple other kernel events that are security related. > I think that #2-by-default is a terrible tradeoff. I suspect I've debugged > more selinux denials than the average user, and I have *never* *once* > looked at a 'syscall' entry in the log. The selinux people wanted the syscall event. Once upon a time, you used to have to add a rule to get the syscall information. But they decided they want more information by default. I would suggest reverting that patch as a test. I think the problem was that they needed a file path sometimes and would ask people to add an audit rule like "-w /etc/shadow -p w". But then the user wouldn't get a recurrence and they couldn't really fix the problem very fast. The exact details may be different, but I think this is the gist of it. > I think that subjecting every Fedora user by default to 20-40 ns of extra > syscall latency for the sole benefit of getting those 'syscall' messages is a > bad tradeoff. I don't think all Fedora users have audit installed and would not see the hit. > I'm willing to write kernel code to improve the situation. The > problem is that getting rid of TIF_SYSCALL_AUDIT when there are no > audit rules configured is messy. Better suggestions are welcome. The problem is getting TIF_SYSCALL_AUDIT back in all processes when auditing is enabled. We cannot stop the OS and stab that flag into all processes when audit gets re-enabled. Its best not to play with that flag. The kernel logic is supposed to be something like if (tif & TIF_SYSCALL_AUDIT) if (current->audit_context) if (audit_ever_enabled) audit_syscall_entry() So, the overhead when disabled should only be an if statement or two. -Steve -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Audit overhead and default rules
On Mon, Feb 10, 2014 at 12:26 PM, Steve Grubb wrote: > On Monday, February 10, 2014 12:10:27 PM Andrew Lutomirski wrote: >> On Mon, Feb 10, 2014 at 12:06 PM, Steve Grubb wrote: >> > On Monday, February 10, 2014 11:05:38 AM Andrew Lutomirski wrote: >> >> On a default Fedora installation, every system call incurs a fair >> >> amount of overhead due to syscall auditing. This happens despite the >> >> fact that syscalls aren't actually audited, except as part of AVC >> >> denials. >> >> >> >> The overhead is something like 20-40ns per syscall, and the total time >> >> to do a simple syscall with auditing completely disabled is about 70ns >> >> on my laptop. So this is actually a large effect. >> > >> > Then pass -s=nochange on the auditd command prompt. This means that auditd >> > will not attempt to enable auditing. When auditing is not enabled, it will >> > not build an audit context and syscalls are slightly faster, but you will >> > loose a tiny bit of information that selinux would have liked to have. >> > >> >> What would people think about changing the default audit rules to add >> >> something like '-t task,never'? >> > >> > This filter is almost useless. Its never used in real life because it >> > creates inauditable processes which is exactly opposite of what people >> > normally want. >> >> It's also the only way to turn off TIF_SYSCALL_AUDIT in current >> kernels. I'm not attempting to justify the sanity of that; I'm just >> reading the code. > > Not enabling audit also causes TIF_SYSCALL_AUDIT to not be placed in the > process's flags. You have 2 choices: 1) performance 2) audit. They are > necessarily mutually exclusive. > > >> >> This would remove the overhead, but it would come at the cost of >> >> removing >> >> >> >> the syscall records from >> >> /var/log/audit/audit.log when an AVC denial occurs. >> >> >> >> This could make debugging selinux errors a bit harder, but it would be >> >> easy for users to re-enable full auditing. >> >> >> >> I've been playing with fixing this in the kernel, but it's a mess. >> > >> > Its also simple to fix in your config. >> >> There are, indeed, many ways for me to fix this on my machine. I'm >> suggesting that Fedora change the default so that no one has >> experiences this overhead by default. > > There are 3 levels of audit performance degradation. > 1) audit is disabled. You get full speed > 2) audit is enabled and no rules. This is the default for Fedora so that more > information can be collected when AVC's occur. > 3) audit is enabled and rules loaded. This does get a performance hit that can > be measured. In this case, the person wanted auditing and is willing to take > any performance hit it may incur. > > The audit system has been set for #2 for the last 8 or 9 years as a balance > between getting information for avc's, not taking a big performance hit, and > keeping setup easy for when people want to add auditing to their system. > Right. I'm proposing changing the default from #2 to #1. I think that #2-by-default is a terrible tradeoff. I suspect I've debugged more selinux denials than the average user, and I have *never* *once* looked at a 'syscall' entry in the log. I think that subjecting every Fedora user by default to 20-40 ns of extra syscall latency for the sole benefit of getting those 'syscall' messages is a bad tradeoff. I'm willing to write kernel code to improve the situation. The problem is that getting rid of TIF_SYSCALL_AUDIT when there are no audit rules configured is messy. Better suggestions are welcome. > >> If the default gets changed, I >> don't particularly care *which* change is made, so long as the effect >> is that TIF_SYSCALL_AUDIT doesn't get set (so there's no overhead) but >> that AVC denials still get logged (which I suspect is the overwhelming >> majority of the value added by audit support). > > AVC's should be logged with or without audit being enabled. Auditd will > collect any avc sent to it by selinux even if audit is disabled. Please try > adding -s=nochange to your config and see how that works for you. I will not be at all surprised if it works. > > -Steve -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Audit overhead and default rules
On Monday, February 10, 2014 12:10:27 PM Andrew Lutomirski wrote: > On Mon, Feb 10, 2014 at 12:06 PM, Steve Grubb wrote: > > On Monday, February 10, 2014 11:05:38 AM Andrew Lutomirski wrote: > >> On a default Fedora installation, every system call incurs a fair > >> amount of overhead due to syscall auditing. This happens despite the > >> fact that syscalls aren't actually audited, except as part of AVC > >> denials. > >> > >> The overhead is something like 20-40ns per syscall, and the total time > >> to do a simple syscall with auditing completely disabled is about 70ns > >> on my laptop. So this is actually a large effect. > > > > Then pass -s=nochange on the auditd command prompt. This means that auditd > > will not attempt to enable auditing. When auditing is not enabled, it will > > not build an audit context and syscalls are slightly faster, but you will > > loose a tiny bit of information that selinux would have liked to have. > > > >> What would people think about changing the default audit rules to add > >> something like '-t task,never'? > > > > This filter is almost useless. Its never used in real life because it > > creates inauditable processes which is exactly opposite of what people > > normally want. > > It's also the only way to turn off TIF_SYSCALL_AUDIT in current > kernels. I'm not attempting to justify the sanity of that; I'm just > reading the code. Not enabling audit also causes TIF_SYSCALL_AUDIT to not be placed in the process's flags. You have 2 choices: 1) performance 2) audit. They are necessarily mutually exclusive. > >> This would remove the overhead, but it would come at the cost of > >> removing > >> > >> the syscall records from > >> /var/log/audit/audit.log when an AVC denial occurs. > >> > >> This could make debugging selinux errors a bit harder, but it would be > >> easy for users to re-enable full auditing. > >> > >> I've been playing with fixing this in the kernel, but it's a mess. > > > > Its also simple to fix in your config. > > There are, indeed, many ways for me to fix this on my machine. I'm > suggesting that Fedora change the default so that no one has > experiences this overhead by default. There are 3 levels of audit performance degradation. 1) audit is disabled. You get full speed 2) audit is enabled and no rules. This is the default for Fedora so that more information can be collected when AVC's occur. 3) audit is enabled and rules loaded. This does get a performance hit that can be measured. In this case, the person wanted auditing and is willing to take any performance hit it may incur. The audit system has been set for #2 for the last 8 or 9 years as a balance between getting information for avc's, not taking a big performance hit, and keeping setup easy for when people want to add auditing to their system. > If the default gets changed, I > don't particularly care *which* change is made, so long as the effect > is that TIF_SYSCALL_AUDIT doesn't get set (so there's no overhead) but > that AVC denials still get logged (which I suspect is the overwhelming > majority of the value added by audit support). AVC's should be logged with or without audit being enabled. Auditd will collect any avc sent to it by selinux even if audit is disabled. Please try adding -s=nochange to your config and see how that works for you. -Steve -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Audit overhead and default rules
On Mon, Feb 10, 2014 at 12:06 PM, Steve Grubb wrote: > On Monday, February 10, 2014 11:05:38 AM Andrew Lutomirski wrote: >> On a default Fedora installation, every system call incurs a fair >> amount of overhead due to syscall auditing. This happens despite the >> fact that syscalls aren't actually audited, except as part of AVC >> denials. >> >> The overhead is something like 20-40ns per syscall, and the total time >> to do a simple syscall with auditing completely disabled is about 70ns >> on my laptop. So this is actually a large effect. > > Then pass -s=nochange on the auditd command prompt. This means that auditd > will not attempt to enable auditing. When auditing is not enabled, it will not > build an audit context and syscalls are slightly faster, but you will loose a > tiny bit of information that selinux would have liked to have. > > >> What would people think about changing the default audit rules to add >> something like '-t task,never'? > > This filter is almost useless. Its never used in real life because it creates > inauditable processes which is exactly opposite of what people normally want. It's also the only way to turn off TIF_SYSCALL_AUDIT in current kernels. I'm not attempting to justify the sanity of that; I'm just reading the code. > >> This would remove the overhead, but it would come at the cost of removing >> the syscall records from >> /var/log/audit/audit.log when an AVC denial occurs. >> >> This could make debugging selinux errors a bit harder, but it would be >> easy for users to re-enable full auditing. >> >> I've been playing with fixing this in the kernel, but it's a mess. > > Its also simple to fix in your config. There are, indeed, many ways for me to fix this on my machine. I'm suggesting that Fedora change the default so that no one has experiences this overhead by default. If the default gets changed, I don't particularly care *which* change is made, so long as the effect is that TIF_SYSCALL_AUDIT doesn't get set (so there's no overhead) but that AVC denials still get logged (which I suspect is the overwhelming majority of the value added by audit support). --Andy -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Audit overhead and default rules
On Monday, February 10, 2014 11:05:38 AM Andrew Lutomirski wrote: > On a default Fedora installation, every system call incurs a fair > amount of overhead due to syscall auditing. This happens despite the > fact that syscalls aren't actually audited, except as part of AVC > denials. > > The overhead is something like 20-40ns per syscall, and the total time > to do a simple syscall with auditing completely disabled is about 70ns > on my laptop. So this is actually a large effect. Then pass -s=nochange on the auditd command prompt. This means that auditd will not attempt to enable auditing. When auditing is not enabled, it will not build an audit context and syscalls are slightly faster, but you will loose a tiny bit of information that selinux would have liked to have. > What would people think about changing the default audit rules to add > something like '-t task,never'? This filter is almost useless. Its never used in real life because it creates inauditable processes which is exactly opposite of what people normally want. > This would remove the overhead, but it would come at the cost of removing > the syscall records from > /var/log/audit/audit.log when an AVC denial occurs. > > This could make debugging selinux errors a bit harder, but it would be > easy for users to re-enable full auditing. > > I've been playing with fixing this in the kernel, but it's a mess. Its also simple to fix in your config. -Steve -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Audit overhead and default rules
On a default Fedora installation, every system call incurs a fair amount of overhead due to syscall auditing. This happens despite the fact that syscalls aren't actually audited, except as part of AVC denials. The overhead is something like 20-40ns per syscall, and the total time to do a simple syscall with auditing completely disabled is about 70ns on my laptop. So this is actually a large effect. What would people think about changing the default audit rules to add something like '-t task,never'? This would remove the overhead, but it would come at the cost of removing the syscall records from /var/log/audit/audit.log when an AVC denial occurs. This could make debugging selinux errors a bit harder, but it would be easy for users to re-enable full auditing. I've been playing with fixing this in the kernel, but it's a mess. --Andy -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
