Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Sumit Semwal]

Hi Andy,

On 24 June 2017 at 10:13, Greg Kroah-Hartman  wrote:
> On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote:
>> Greg, for context, the issue here is that we made what was arguably a
>> design error in seccomp's interaction with ptrace.  After determining
>> that fixing it solved a bunch of problems and didn't break any user
>> programs, we fixed it.  There might be new code that relies on the fix
>> being present in the sense that it would be insecure without the fix.
>>
>> The problem is that the fix is moderately intrusive and doesn't seem
>> like a great candidate for backporting, although we could plausibly do
>> it.
>
> That's fine, not all bugfixes that tests are created to find, should be
> backported.  That's up to the stable maintainers, or someone who has a
> device/vendor tree based on that kernel if they want to do that or not.
>
> That has nothing to do with the fact that the test should fail or
> gracefully degrade.  Tests should fail if the action that they are
> testing fails.  They should degrade and not run if the _feature_ they
> are testing is not present.

So, any updates on this yet - getting the seccomp tests to degrade
gracefully? I realise you mentioned that the fix could be intrusive;
just wanted to know if it was on your radar still.
>
> Yes, sometimes this is hard, like with the seccomp stuff, and will not
> always work, but that's the rule for our userspace api independant of
> any testing framework or code.
>
> Look at xfstests, no one gets mad when it adds a new test that old
> kernels fail at.  It's up to someone else to either backport the kernel
> change, if they want it fixed in an old kernel, not to have xfstests
> just not run it at all!  There's nothing different here either.
>
> thanks,
>
> greg k-h

Thanks much,
Sumit.

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Sumit Semwal]

Hi Andy,

On 24 June 2017 at 10:13, Greg Kroah-Hartman  wrote:
> On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote:
>> Greg, for context, the issue here is that we made what was arguably a
>> design error in seccomp's interaction with ptrace.  After determining
>> that fixing it solved a bunch of problems and didn't break any user
>> programs, we fixed it.  There might be new code that relies on the fix
>> being present in the sense that it would be insecure without the fix.
>>
>> The problem is that the fix is moderately intrusive and doesn't seem
>> like a great candidate for backporting, although we could plausibly do
>> it.
>
> That's fine, not all bugfixes that tests are created to find, should be
> backported.  That's up to the stable maintainers, or someone who has a
> device/vendor tree based on that kernel if they want to do that or not.
>
> That has nothing to do with the fact that the test should fail or
> gracefully degrade.  Tests should fail if the action that they are
> testing fails.  They should degrade and not run if the _feature_ they
> are testing is not present.

So, any updates on this yet - getting the seccomp tests to degrade
gracefully? I realise you mentioned that the fix could be intrusive;
just wanted to know if it was on your radar still.
>
> Yes, sometimes this is hard, like with the seccomp stuff, and will not
> always work, but that's the rule for our userspace api independant of
> any testing framework or code.
>
> Look at xfstests, no one gets mad when it adds a new test that old
> kernels fail at.  It's up to someone else to either backport the kernel
> change, if they want it fixed in an old kernel, not to have xfstests
> just not run it at all!  There's nothing different here either.
>
> thanks,
>
> greg k-h

Thanks much,
Sumit.

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Luis R. Rodriguez]

On Sat, Jun 24, 2017 at 06:45:37AM +0200, Greg Kroah-Hartman wrote:
> On Sat, Jun 24, 2017 at 02:34:07AM +0200, Luis R. Rodriguez wrote:
> > So taking the position that any kselftest script on linux-next or a future
> > kernel should never break stable implicate that *any* fix going upstream for
> > which there is a respective ksefltest test *must* have a stable upstream 
> > fix.
> 
> What, no!  If it's a bug, that kselftest points out, great, it's up to
> stable or a vendor to backport that fix if they want it.

Makes sense!

> Again, it's just like any other test suite, look at xfstests, no one
> gets mad when it adds new tests that fail on old kernels, due to bugs
> there, right?

Correct but that means there is an information gap of which test cases fit into
this category. An option to run kselftest with the ability to only run tests
designed to also include stable would be good, otherwise Sumit Semwal or others
would be sending reports or issues for things folks already designed the fix to
*not* be a stable candidate. This actually would also prove useful to
distributions to ensure to then run kselftest as part of their distribution
test suite with all bells and whistles enabled so that they can make a
determination of what questionable "fixes" things to fold in somehow.

Unless of course we want these discussions to purposely bubble up as an
alternative to this kselftest option. It seems rather wasteful though.

> > Its not obvious to me that everyone is aware of this. What do we do about
> > those cases where we *don't* want a stable fix due to the complexity?
> 
> That's up to the stable maintainers or the vendors that maintain their
> own kernel trees.

But if the above is already decided having folks send emails about it
seems pointless.

  Luis

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Luis R. Rodriguez]

On Sat, Jun 24, 2017 at 06:45:37AM +0200, Greg Kroah-Hartman wrote:
> On Sat, Jun 24, 2017 at 02:34:07AM +0200, Luis R. Rodriguez wrote:
> > So taking the position that any kselftest script on linux-next or a future
> > kernel should never break stable implicate that *any* fix going upstream for
> > which there is a respective ksefltest test *must* have a stable upstream 
> > fix.
> 
> What, no!  If it's a bug, that kselftest points out, great, it's up to
> stable or a vendor to backport that fix if they want it.

Makes sense!

> Again, it's just like any other test suite, look at xfstests, no one
> gets mad when it adds new tests that fail on old kernels, due to bugs
> there, right?

Correct but that means there is an information gap of which test cases fit into
this category. An option to run kselftest with the ability to only run tests
designed to also include stable would be good, otherwise Sumit Semwal or others
would be sending reports or issues for things folks already designed the fix to
*not* be a stable candidate. This actually would also prove useful to
distributions to ensure to then run kselftest as part of their distribution
test suite with all bells and whistles enabled so that they can make a
determination of what questionable "fixes" things to fold in somehow.

Unless of course we want these discussions to purposely bubble up as an
alternative to this kselftest option. It seems rather wasteful though.

> > Its not obvious to me that everyone is aware of this. What do we do about
> > those cases where we *don't* want a stable fix due to the complexity?
> 
> That's up to the stable maintainers or the vendors that maintain their
> own kernel trees.

But if the above is already decided having folks send emails about it
seems pointless.

  Luis

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Greg Kroah-Hartman]

On Sat, Jun 24, 2017 at 02:34:07AM +0200, Luis R. Rodriguez wrote:
> So taking the position that any kselftest script on linux-next or a future
> kernel should never break stable implicate that *any* fix going upstream for
> which there is a respective ksefltest test *must* have a stable upstream fix.

What, no!  If it's a bug, that kselftest points out, great, it's up to
stable or a vendor to backport that fix if they want it.

Again, it's just like any other test suite, look at xfstests, no one
gets mad when it adds new tests that fail on old kernels, due to bugs
there, right?

> Its not obvious to me that everyone is aware of this. What do we do about
> those cases where we *don't* want a stable fix due to the complexity?

That's up to the stable maintainers or the vendors that maintain their
own kernel trees.

thanks,

greg k-h

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Greg Kroah-Hartman]

On Sat, Jun 24, 2017 at 02:34:07AM +0200, Luis R. Rodriguez wrote:
> So taking the position that any kselftest script on linux-next or a future
> kernel should never break stable implicate that *any* fix going upstream for
> which there is a respective ksefltest test *must* have a stable upstream fix.

What, no!  If it's a bug, that kselftest points out, great, it's up to
stable or a vendor to backport that fix if they want it.

Again, it's just like any other test suite, look at xfstests, no one
gets mad when it adds new tests that fail on old kernels, due to bugs
there, right?

> Its not obvious to me that everyone is aware of this. What do we do about
> those cases where we *don't* want a stable fix due to the complexity?

That's up to the stable maintainers or the vendors that maintain their
own kernel trees.

thanks,

greg k-h

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Greg Kroah-Hartman]

On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote:
> Greg, for context, the issue here is that we made what was arguably a
> design error in seccomp's interaction with ptrace.  After determining
> that fixing it solved a bunch of problems and didn't break any user
> programs, we fixed it.  There might be new code that relies on the fix
> being present in the sense that it would be insecure without the fix.
> 
> The problem is that the fix is moderately intrusive and doesn't seem
> like a great candidate for backporting, although we could plausibly do
> it.

That's fine, not all bugfixes that tests are created to find, should be
backported.  That's up to the stable maintainers, or someone who has a
device/vendor tree based on that kernel if they want to do that or not.

That has nothing to do with the fact that the test should fail or
gracefully degrade.  Tests should fail if the action that they are
testing fails.  They should degrade and not run if the _feature_ they
are testing is not present.

Yes, sometimes this is hard, like with the seccomp stuff, and will not
always work, but that's the rule for our userspace api independant of
any testing framework or code.

Look at xfstests, no one gets mad when it adds a new test that old
kernels fail at.  It's up to someone else to either backport the kernel
change, if they want it fixed in an old kernel, not to have xfstests
just not run it at all!  There's nothing different here either.

thanks,

greg k-h

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Greg Kroah-Hartman]

On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote:
> Greg, for context, the issue here is that we made what was arguably a
> design error in seccomp's interaction with ptrace.  After determining
> that fixing it solved a bunch of problems and didn't break any user
> programs, we fixed it.  There might be new code that relies on the fix
> being present in the sense that it would be insecure without the fix.
> 
> The problem is that the fix is moderately intrusive and doesn't seem
> like a great candidate for backporting, although we could plausibly do
> it.

That's fine, not all bugfixes that tests are created to find, should be
backported.  That's up to the stable maintainers, or someone who has a
device/vendor tree based on that kernel if they want to do that or not.

That has nothing to do with the fact that the test should fail or
gracefully degrade.  Tests should fail if the action that they are
testing fails.  They should degrade and not run if the _feature_ they
are testing is not present.

Yes, sometimes this is hard, like with the seccomp stuff, and will not
always work, but that's the rule for our userspace api independant of
any testing framework or code.

Look at xfstests, no one gets mad when it adds a new test that old
kernels fail at.  It's up to someone else to either backport the kernel
change, if they want it fixed in an old kernel, not to have xfstests
just not run it at all!  There's nothing different here either.

thanks,

greg k-h

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Luis R. Rodriguez]

On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote:
> On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman
>  wrote:
> > On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
> >> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> >> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> >> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
> >> Right. I really think stable kernels should be tested with their own
> >> selftests. If some test is needed in a stable kernel it should be
> >> backported to that stable kernel.
> >
> > Well, ideally all new features added to the kernel should be able to be
> > detected by userspace somehow if they are present or not.
> >
> > How do you expect a program to know if a feature has "failed" or is just
> > "not enabled/present in this kernel"?  Normally with syscalls this is
> > easy, same for sysfs changes.  Is seccomp in the bad state where there
> > is no way to detect the two different states here?  How is userspace
> > supposed to deal with that?
> >
> > We make fun of glibc having a zillion crazy tests to determine kernel
> > features, and recently, just not wrapping new syscalls at all because
> > they are just frustrated at the compatibility issues over time.  Let's
> > not make their life any harder than it has to be please.
> >
> > I don't see how any of the kselftest programs are any different than any
> > other userspace program that wants to use our kernel api, and as such,
> > any version of kselftest should be able to successfully run on any
> > kernel release.  If not, then we messed up in how we either wrote the
> > test, or how we added a new kernel api.  Neither is acceptable.
> 
> That's a fair point.

I agreed with it as well just a few threads ago due to similar issues, however,
thinking this over I'm afraid this has some interesting side consequences for
fixes and what code goes upstream into kselftest.

<-- snip -->

> The problem is that the fix is moderately intrusive and doesn't seem
> like a great candidate for backporting, although we could plausibly do
> it.

Such is the case often actually.

So taking the position that any kselftest script on linux-next or a future
kernel should never break stable implicate that *any* fix going upstream for
which there is a respective ksefltest test *must* have a stable upstream fix.

Its not obvious to me that everyone is aware of this. What do we do about
those cases where we *don't* want a stable fix due to the complexity?

  Luis

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Luis R. Rodriguez]

On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote:
> On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman
>  wrote:
> > On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
> >> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> >> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> >> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
> >> Right. I really think stable kernels should be tested with their own
> >> selftests. If some test is needed in a stable kernel it should be
> >> backported to that stable kernel.
> >
> > Well, ideally all new features added to the kernel should be able to be
> > detected by userspace somehow if they are present or not.
> >
> > How do you expect a program to know if a feature has "failed" or is just
> > "not enabled/present in this kernel"?  Normally with syscalls this is
> > easy, same for sysfs changes.  Is seccomp in the bad state where there
> > is no way to detect the two different states here?  How is userspace
> > supposed to deal with that?
> >
> > We make fun of glibc having a zillion crazy tests to determine kernel
> > features, and recently, just not wrapping new syscalls at all because
> > they are just frustrated at the compatibility issues over time.  Let's
> > not make their life any harder than it has to be please.
> >
> > I don't see how any of the kselftest programs are any different than any
> > other userspace program that wants to use our kernel api, and as such,
> > any version of kselftest should be able to successfully run on any
> > kernel release.  If not, then we messed up in how we either wrote the
> > test, or how we added a new kernel api.  Neither is acceptable.
> 
> That's a fair point.

I agreed with it as well just a few threads ago due to similar issues, however,
thinking this over I'm afraid this has some interesting side consequences for
fixes and what code goes upstream into kselftest.

<-- snip -->

> The problem is that the fix is moderately intrusive and doesn't seem
> like a great candidate for backporting, although we could plausibly do
> it.

Such is the case often actually.

So taking the position that any kselftest script on linux-next or a future
kernel should never break stable implicate that *any* fix going upstream for
which there is a respective ksefltest test *must* have a stable upstream fix.

Its not obvious to me that everyone is aware of this. What do we do about
those cases where we *don't* want a stable fix due to the complexity?

  Luis

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Tom Gall]

Hi Shuah,

On Fri, Jun 23, 2017 at 2:03 PM, Shuah Khan  wrote:
> On 06/22/2017 01:48 PM, Tom Gall wrote:
>> Hi
>>
>> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>>> On 06/22/2017 11:50 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> wrote:
 Hi Kees, Andy,

 On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

 Could you please help us sort this out? Our goal is to help Greg with
 testing stable kernels, and currently the seccomp tests fail due to
 missing feature (seccomp ptrace hole closure) getting tested via
 latest kselftest.

 If you feel the feature isn't a stable candidate, then could you
 please help make the test degrade gracefully in its absence?
>>>
>>> In some cases, it is not easy to degrade and/or check for a feature.
>>> Probably several security features could fall in this bucket.
>>>
>>>
>>> I don't really want to have that change be a backport -- it's quite
>>> invasive across multiple architectures.
>>>
>>> Agreed. The same test for kernel applies to tests as well. If a kernel
>>> feature can't be back-ported, the test for that feature will fall in the
>>> same bucket. It shouldn't be back-ported.
>>>
>>>
>>> I would say just add a kernel version check to the test. This is
>>> probably not the only selftest that will need such things. :)
>>
>> Adding release checks to selftests is going to problematic for 
>> maintenance.
>> Tests should fail gracefully if feature isn't supported in older kernels.
>>
>> Several tests do that now and please find a way to check for dependencies
>> and feature availability and fail the test gracefully. If there is a test
>> that can't do that for some reason, we can discuss it, but as a general
>> rule, I don't want to see kselftest patches that check release.
>
> If a future kernel inadvertently loses the new feature and degrades to
> the behavior of old kernels, that would be a serious bug and should be
> caught.
>>>
>>> Agreed. If I understand you correctly, by not testing stable kernels
>>> with their own selftests, some serious bugs could go undetected.
>>
>> Personally I'm a bit skeptical. I think the reasoning is more that the
>> latest selftests provide more coverage, and therefore should be better
>> tests, even on older kernels.
>>

 Right. I really think stable kernels should be tested with their own
 selftests. If some test is needed in a stable kernel it should be
 backported to that stable kernel.
>>>
>>> Correct. This is always a safe option. There might be cases that even
>>> prevent tests being built, especially if a new feature adds new fields
>>> to an existing structure.
>>>
>>> It appears in some cases, users want to run newer tests on older kernels.
>>> Some tests can clearly detect feature support using module presence and/or
>>> Kconfig enabled or disabled. These are conditions even on a kernel that
>>> supports a new module or new config option. The kernel the test is running
>>> on might not have the feature enabled or module might not be present. In
>>> these cases, it would be easier to detect and skip the test.
>>>
>>> However, some features aren't so easy. For example:
>>>
>>> - a new flag is added to a syscall, and new test is added. It might not
>>>   be easy to detect that.
>>> - We might have some tests that can't detect and skip.
>>>
>>> Based on this discussion, it is probably accurate to say:
>>>
>>> 1. It is recommended that selftests from the same release be run on the
>>>kernel.
>>> 2. Selftests from newer kernels will run on older kernels, user should
>>>understand the risks such as some tests might fail and might not
>>>detect feature degradation related bugs.
>>> 3. Selftests will fail gracefully on older releases if at all possible.
>>
>> How about gracefully be skipped instead of fail?
>>
>> The later suggests the test case in some situations can detect it's
>> pointless to run something and say as much instead of emitting a
>> failure that would be a waste of time to look into.
>>
>> As another example take tools/testing/selftests/net/psock_fanout.c
>> On 4.9 it'll fail to compile (using master's selftests) because

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Tom Gall]

Hi Shuah,

On Fri, Jun 23, 2017 at 2:03 PM, Shuah Khan  wrote:
> On 06/22/2017 01:48 PM, Tom Gall wrote:
>> Hi
>>
>> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>>> On 06/22/2017 11:50 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> wrote:
 Hi Kees, Andy,

 On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

 Could you please help us sort this out? Our goal is to help Greg with
 testing stable kernels, and currently the seccomp tests fail due to
 missing feature (seccomp ptrace hole closure) getting tested via
 latest kselftest.

 If you feel the feature isn't a stable candidate, then could you
 please help make the test degrade gracefully in its absence?
>>>
>>> In some cases, it is not easy to degrade and/or check for a feature.
>>> Probably several security features could fall in this bucket.
>>>
>>>
>>> I don't really want to have that change be a backport -- it's quite
>>> invasive across multiple architectures.
>>>
>>> Agreed. The same test for kernel applies to tests as well. If a kernel
>>> feature can't be back-ported, the test for that feature will fall in the
>>> same bucket. It shouldn't be back-ported.
>>>
>>>
>>> I would say just add a kernel version check to the test. This is
>>> probably not the only selftest that will need such things. :)
>>
>> Adding release checks to selftests is going to problematic for 
>> maintenance.
>> Tests should fail gracefully if feature isn't supported in older kernels.
>>
>> Several tests do that now and please find a way to check for dependencies
>> and feature availability and fail the test gracefully. If there is a test
>> that can't do that for some reason, we can discuss it, but as a general
>> rule, I don't want to see kselftest patches that check release.
>
> If a future kernel inadvertently loses the new feature and degrades to
> the behavior of old kernels, that would be a serious bug and should be
> caught.
>>>
>>> Agreed. If I understand you correctly, by not testing stable kernels
>>> with their own selftests, some serious bugs could go undetected.
>>
>> Personally I'm a bit skeptical. I think the reasoning is more that the
>> latest selftests provide more coverage, and therefore should be better
>> tests, even on older kernels.
>>

 Right. I really think stable kernels should be tested with their own
 selftests. If some test is needed in a stable kernel it should be
 backported to that stable kernel.
>>>
>>> Correct. This is always a safe option. There might be cases that even
>>> prevent tests being built, especially if a new feature adds new fields
>>> to an existing structure.
>>>
>>> It appears in some cases, users want to run newer tests on older kernels.
>>> Some tests can clearly detect feature support using module presence and/or
>>> Kconfig enabled or disabled. These are conditions even on a kernel that
>>> supports a new module or new config option. The kernel the test is running
>>> on might not have the feature enabled or module might not be present. In
>>> these cases, it would be easier to detect and skip the test.
>>>
>>> However, some features aren't so easy. For example:
>>>
>>> - a new flag is added to a syscall, and new test is added. It might not
>>>   be easy to detect that.
>>> - We might have some tests that can't detect and skip.
>>>
>>> Based on this discussion, it is probably accurate to say:
>>>
>>> 1. It is recommended that selftests from the same release be run on the
>>>kernel.
>>> 2. Selftests from newer kernels will run on older kernels, user should
>>>understand the risks such as some tests might fail and might not
>>>detect feature degradation related bugs.
>>> 3. Selftests will fail gracefully on older releases if at all possible.
>>
>> How about gracefully be skipped instead of fail?
>>
>> The later suggests the test case in some situations can detect it's
>> pointless to run something and say as much instead of emitting a
>> failure that would be a waste of time to look into.
>>
>> As another example take tools/testing/selftests/net/psock_fanout.c
>> On 4.9 it'll fail to compile (using master's selftests) because
>> PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for
>> that symbol and the psock_fanout test will 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 01:48 PM, Tom Gall wrote:
> Hi
> 
> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>> On 06/22/2017 11:50 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
 On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> wrote:
>>> Hi Kees, Andy,
>>>
>>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
 feature and test together.
 - This one also seems like a security hole being closed, and the
 'feature' could be a candidate for stable backports, but Arnd tried
 that, and it was quite non-trivial. So perhaps  we'll need some help
 from the subsystem developers here.
>>>
>>> Could you please help us sort this out? Our goal is to help Greg with
>>> testing stable kernels, and currently the seccomp tests fail due to
>>> missing feature (seccomp ptrace hole closure) getting tested via
>>> latest kselftest.
>>>
>>> If you feel the feature isn't a stable candidate, then could you
>>> please help make the test degrade gracefully in its absence?
>>
>> In some cases, it is not easy to degrade and/or check for a feature.
>> Probably several security features could fall in this bucket.
>>
>>
>> I don't really want to have that change be a backport -- it's quite
>> invasive across multiple architectures.
>>
>> Agreed. The same test for kernel applies to tests as well. If a kernel
>> feature can't be back-ported, the test for that feature will fall in the
>> same bucket. It shouldn't be back-ported.
>>
>>
>> I would say just add a kernel version check to the test. This is
>> probably not the only selftest that will need such things. :)
>
> Adding release checks to selftests is going to problematic for 
> maintenance.
> Tests should fail gracefully if feature isn't supported in older kernels.
>
> Several tests do that now and please find a way to check for dependencies
> and feature availability and fail the test gracefully. If there is a test
> that can't do that for some reason, we can discuss it, but as a general
> rule, I don't want to see kselftest patches that check release.

 If a future kernel inadvertently loses the new feature and degrades to
 the behavior of old kernels, that would be a serious bug and should be
 caught.
>>
>> Agreed. If I understand you correctly, by not testing stable kernels
>> with their own selftests, some serious bugs could go undetected.
> 
> Personally I'm a bit skeptical. I think the reasoning is more that the
> latest selftests provide more coverage, and therefore should be better
> tests, even on older kernels.
> 
>>>
>>> Right. I really think stable kernels should be tested with their own
>>> selftests. If some test is needed in a stable kernel it should be
>>> backported to that stable kernel.
>>
>> Correct. This is always a safe option. There might be cases that even
>> prevent tests being built, especially if a new feature adds new fields
>> to an existing structure.
>>
>> It appears in some cases, users want to run newer tests on older kernels.
>> Some tests can clearly detect feature support using module presence and/or
>> Kconfig enabled or disabled. These are conditions even on a kernel that
>> supports a new module or new config option. The kernel the test is running
>> on might not have the feature enabled or module might not be present. In
>> these cases, it would be easier to detect and skip the test.
>>
>> However, some features aren't so easy. For example:
>>
>> - a new flag is added to a syscall, and new test is added. It might not
>>   be easy to detect that.
>> - We might have some tests that can't detect and skip.
>>
>> Based on this discussion, it is probably accurate to say:
>>
>> 1. It is recommended that selftests from the same release be run on the
>>kernel.
>> 2. Selftests from newer kernels will run on older kernels, user should
>>understand the risks such as some tests might fail and might not
>>detect feature degradation related bugs.
>> 3. Selftests will fail gracefully on older releases if at all possible.
> 
> How about gracefully be skipped instead of fail?
> 
> The later suggests the test case in some situations can detect it's
> pointless to run something and say as much instead of emitting a
> failure that would be a waste of time to look into.
> 
> As another example take tools/testing/selftests/net/psock_fanout.c
> On 4.9 it'll fail to compile (using master's selftests) because
> PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for
> that symbol and the psock_fanout test will compile and run just fine.

Unfortunately adding 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 01:48 PM, Tom Gall wrote:
> Hi
> 
> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>> On 06/22/2017 11:50 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
 On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> wrote:
>>> Hi Kees, Andy,
>>>
>>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
 feature and test together.
 - This one also seems like a security hole being closed, and the
 'feature' could be a candidate for stable backports, but Arnd tried
 that, and it was quite non-trivial. So perhaps  we'll need some help
 from the subsystem developers here.
>>>
>>> Could you please help us sort this out? Our goal is to help Greg with
>>> testing stable kernels, and currently the seccomp tests fail due to
>>> missing feature (seccomp ptrace hole closure) getting tested via
>>> latest kselftest.
>>>
>>> If you feel the feature isn't a stable candidate, then could you
>>> please help make the test degrade gracefully in its absence?
>>
>> In some cases, it is not easy to degrade and/or check for a feature.
>> Probably several security features could fall in this bucket.
>>
>>
>> I don't really want to have that change be a backport -- it's quite
>> invasive across multiple architectures.
>>
>> Agreed. The same test for kernel applies to tests as well. If a kernel
>> feature can't be back-ported, the test for that feature will fall in the
>> same bucket. It shouldn't be back-ported.
>>
>>
>> I would say just add a kernel version check to the test. This is
>> probably not the only selftest that will need such things. :)
>
> Adding release checks to selftests is going to problematic for 
> maintenance.
> Tests should fail gracefully if feature isn't supported in older kernels.
>
> Several tests do that now and please find a way to check for dependencies
> and feature availability and fail the test gracefully. If there is a test
> that can't do that for some reason, we can discuss it, but as a general
> rule, I don't want to see kselftest patches that check release.

 If a future kernel inadvertently loses the new feature and degrades to
 the behavior of old kernels, that would be a serious bug and should be
 caught.
>>
>> Agreed. If I understand you correctly, by not testing stable kernels
>> with their own selftests, some serious bugs could go undetected.
> 
> Personally I'm a bit skeptical. I think the reasoning is more that the
> latest selftests provide more coverage, and therefore should be better
> tests, even on older kernels.
> 
>>>
>>> Right. I really think stable kernels should be tested with their own
>>> selftests. If some test is needed in a stable kernel it should be
>>> backported to that stable kernel.
>>
>> Correct. This is always a safe option. There might be cases that even
>> prevent tests being built, especially if a new feature adds new fields
>> to an existing structure.
>>
>> It appears in some cases, users want to run newer tests on older kernels.
>> Some tests can clearly detect feature support using module presence and/or
>> Kconfig enabled or disabled. These are conditions even on a kernel that
>> supports a new module or new config option. The kernel the test is running
>> on might not have the feature enabled or module might not be present. In
>> these cases, it would be easier to detect and skip the test.
>>
>> However, some features aren't so easy. For example:
>>
>> - a new flag is added to a syscall, and new test is added. It might not
>>   be easy to detect that.
>> - We might have some tests that can't detect and skip.
>>
>> Based on this discussion, it is probably accurate to say:
>>
>> 1. It is recommended that selftests from the same release be run on the
>>kernel.
>> 2. Selftests from newer kernels will run on older kernels, user should
>>understand the risks such as some tests might fail and might not
>>detect feature degradation related bugs.
>> 3. Selftests will fail gracefully on older releases if at all possible.
> 
> How about gracefully be skipped instead of fail?
> 
> The later suggests the test case in some situations can detect it's
> pointless to run something and say as much instead of emitting a
> failure that would be a waste of time to look into.
> 
> As another example take tools/testing/selftests/net/psock_fanout.c
> On 4.9 it'll fail to compile (using master's selftests) because
> PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for
> that symbol and the psock_fanout test will compile and run just fine.

Unfortunately adding PACKET_FANOUT_FLAG_UNIQUEID isn't correct. This is
a new feature that went into 4.12. You don't want to define this for 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 10:02 PM, Sumit Semwal wrote:
> Hi Shuah,
> 
> On 23 June 2017 at 01:53, Shuah Khan  wrote:
>> Hi Tom,
>>
>> On 06/22/2017 01:48 PM, Tom Gall wrote:
>>> Hi
>>>
>>> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
 On 06/22/2017 11:50 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>>> On 06/22/2017 10:53 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal 
  wrote:
> Hi Kees, Andy,
>
> On 15 June 2017 at 23:26, Sumit Semwal  
> wrote:
>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> feature and test together.
>> - This one also seems like a security hole being closed, and the
>> 'feature' could be a candidate for stable backports, but Arnd tried
>> that, and it was quite non-trivial. So perhaps  we'll need some help
>> from the subsystem developers here.
>
> Could you please help us sort this out? Our goal is to help Greg with
> testing stable kernels, and currently the seccomp tests fail due to
> missing feature (seccomp ptrace hole closure) getting tested via
> latest kselftest.
>
> If you feel the feature isn't a stable candidate, then could you
> please help make the test degrade gracefully in its absence?

 In some cases, it is not easy to degrade and/or check for a feature.
 Probably several security features could fall in this bucket.


 I don't really want to have that change be a backport -- it's quite
 invasive across multiple architectures.

 Agreed. The same test for kernel applies to tests as well. If a kernel
 feature can't be back-ported, the test for that feature will fall in the
 same bucket. It shouldn't be back-ported.


 I would say just add a kernel version check to the test. This is
 probably not the only selftest that will need such things. :)
>>>
>>> Adding release checks to selftests is going to problematic for 
>>> maintenance.
>>> Tests should fail gracefully if feature isn't supported in older 
>>> kernels.
>>>
>>> Several tests do that now and please find a way to check for 
>>> dependencies
>>> and feature availability and fail the test gracefully. If there is a 
>>> test
>>> that can't do that for some reason, we can discuss it, but as a general
>>> rule, I don't want to see kselftest patches that check release.
>>
>> If a future kernel inadvertently loses the new feature and degrades to
>> the behavior of old kernels, that would be a serious bug and should be
>> caught.

 Agreed. If I understand you correctly, by not testing stable kernels
 with their own selftests, some serious bugs could go undetected.
>>>
>>> Personally I'm a bit skeptical. I think the reasoning is more that the
>>> latest selftests provide more coverage, and therefore should be better
>>> tests, even on older kernels.
>>
>> The assumption that "the latest selftests provide more coverage, and
>> therefore should be better tests, even on older kernels." is incorrect.
>>
>> Selftests in general track the kernel features. In some cases, new
>> tests could be added that provide better coverage on older kernels,
>> however, it is more likely that new tests are added to test new kernel
>> features and enhancements to existing features. Based on the second
>> "enhancements to existing features" it is more important to test newer
>> kernels with older selftests. This does happen in kernel integration
>> cycles during development.
>>
>> As a general rule, testing stable kernels with their own selftests will
>> yield the best results.
>>
> I would have agreed totally, if the selftests and the kernel were in
> sync since forever. But since the kselftests are a comparatively
> recent addition, the number of tests available for features existing
> in LTS kernels is really quite small. Just as a comparison, 4.4-LTS
> misses tests for bpf, cpufreq, gpio, media_tests, networking, prctl,
> to name a few.

Yes. Several tests have been added since 4.4. Unfortunately these new
additions don't qualify as candidates for stable releases.

> 
> Also, while trying to run kselftests from later kernels with 4.4, we
> only had a few failures for existing features, while most other tests
> ran ok. Just another data point.

That is good to hear. Let's fix the problems as you find them and make
the selftests from newer releases work well (if not perfect) on older
kernels.

4.13 is going to see TAP13 changes as well, so if selftests from newer
kernels work for you, you will see TAB13 benefits.

thanks,
-- Shuah

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 10:02 PM, Sumit Semwal wrote:
> Hi Shuah,
> 
> On 23 June 2017 at 01:53, Shuah Khan  wrote:
>> Hi Tom,
>>
>> On 06/22/2017 01:48 PM, Tom Gall wrote:
>>> Hi
>>>
>>> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
 On 06/22/2017 11:50 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>>> On 06/22/2017 10:53 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal 
  wrote:
> Hi Kees, Andy,
>
> On 15 June 2017 at 23:26, Sumit Semwal  
> wrote:
>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> feature and test together.
>> - This one also seems like a security hole being closed, and the
>> 'feature' could be a candidate for stable backports, but Arnd tried
>> that, and it was quite non-trivial. So perhaps  we'll need some help
>> from the subsystem developers here.
>
> Could you please help us sort this out? Our goal is to help Greg with
> testing stable kernels, and currently the seccomp tests fail due to
> missing feature (seccomp ptrace hole closure) getting tested via
> latest kselftest.
>
> If you feel the feature isn't a stable candidate, then could you
> please help make the test degrade gracefully in its absence?

 In some cases, it is not easy to degrade and/or check for a feature.
 Probably several security features could fall in this bucket.


 I don't really want to have that change be a backport -- it's quite
 invasive across multiple architectures.

 Agreed. The same test for kernel applies to tests as well. If a kernel
 feature can't be back-ported, the test for that feature will fall in the
 same bucket. It shouldn't be back-ported.


 I would say just add a kernel version check to the test. This is
 probably not the only selftest that will need such things. :)
>>>
>>> Adding release checks to selftests is going to problematic for 
>>> maintenance.
>>> Tests should fail gracefully if feature isn't supported in older 
>>> kernels.
>>>
>>> Several tests do that now and please find a way to check for 
>>> dependencies
>>> and feature availability and fail the test gracefully. If there is a 
>>> test
>>> that can't do that for some reason, we can discuss it, but as a general
>>> rule, I don't want to see kselftest patches that check release.
>>
>> If a future kernel inadvertently loses the new feature and degrades to
>> the behavior of old kernels, that would be a serious bug and should be
>> caught.

 Agreed. If I understand you correctly, by not testing stable kernels
 with their own selftests, some serious bugs could go undetected.
>>>
>>> Personally I'm a bit skeptical. I think the reasoning is more that the
>>> latest selftests provide more coverage, and therefore should be better
>>> tests, even on older kernels.
>>
>> The assumption that "the latest selftests provide more coverage, and
>> therefore should be better tests, even on older kernels." is incorrect.
>>
>> Selftests in general track the kernel features. In some cases, new
>> tests could be added that provide better coverage on older kernels,
>> however, it is more likely that new tests are added to test new kernel
>> features and enhancements to existing features. Based on the second
>> "enhancements to existing features" it is more important to test newer
>> kernels with older selftests. This does happen in kernel integration
>> cycles during development.
>>
>> As a general rule, testing stable kernels with their own selftests will
>> yield the best results.
>>
> I would have agreed totally, if the selftests and the kernel were in
> sync since forever. But since the kselftests are a comparatively
> recent addition, the number of tests available for features existing
> in LTS kernels is really quite small. Just as a comparison, 4.4-LTS
> misses tests for bpf, cpufreq, gpio, media_tests, networking, prctl,
> to name a few.

Yes. Several tests have been added since 4.4. Unfortunately these new
additions don't qualify as candidates for stable releases.

> 
> Also, while trying to run kselftests from later kernels with 4.4, we
> only had a few failures for existing features, while most other tests
> ran ok. Just another data point.

That is good to hear. Let's fix the problems as you find them and make
the selftests from newer releases work well (if not perfect) on older
kernels.

4.13 is going to see TAP13 changes as well, so if selftests from newer
kernels work for you, you will see TAB13 benefits.

thanks,
-- Shuah

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Kees Cook]

On Thu, Jun 22, 2017 at 7:40 PM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman
>  wrote:
>> On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>>> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>>> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> >>> wrote:
>>>  Hi Kees, Andy,
>>> 
>>>  On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>>> > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>>> > feature and test together.
>>> > - This one also seems like a security hole being closed, and the
>>> > 'feature' could be a candidate for stable backports, but Arnd tried
>>> > that, and it was quite non-trivial. So perhaps  we'll need some help
>>> > from the subsystem developers here.
>>> 
>>>  Could you please help us sort this out? Our goal is to help Greg with
>>>  testing stable kernels, and currently the seccomp tests fail due to
>>>  missing feature (seccomp ptrace hole closure) getting tested via
>>>  latest kselftest.
>>> 
>>>  If you feel the feature isn't a stable candidate, then could you
>>>  please help make the test degrade gracefully in its absence?
>>> >>>
>>> >>> I don't really want to have that change be a backport -- it's quite
>>> >>> invasive across multiple architectures.
>>> >>>
>>> >>> I would say just add a kernel version check to the test. This is
>>> >>> probably not the only selftest that will need such things. :)
>>> >>
>>> >> Adding release checks to selftests is going to problematic for 
>>> >> maintenance.
>>> >> Tests should fail gracefully if feature isn't supported in older kernels.
>>> >>
>>> >> Several tests do that now and please find a way to check for dependencies
>>> >> and feature availability and fail the test gracefully. If there is a test
>>> >> that can't do that for some reason, we can discuss it, but as a general
>>> >> rule, I don't want to see kselftest patches that check release.
>>> >
>>> > If a future kernel inadvertently loses the new feature and degrades to
>>> > the behavior of old kernels, that would be a serious bug and should be
>>> > caught.
>>>
>>> Right. I really think stable kernels should be tested with their own
>>> selftests. If some test is needed in a stable kernel it should be
>>> backported to that stable kernel.
>>
>> Well, ideally all new features added to the kernel should be able to be
>> detected by userspace somehow if they are present or not.
>>
>> How do you expect a program to know if a feature has "failed" or is just
>> "not enabled/present in this kernel"?  Normally with syscalls this is
>> easy, same for sysfs changes.  Is seccomp in the bad state where there
>> is no way to detect the two different states here?  How is userspace
>> supposed to deal with that?
>>
>> We make fun of glibc having a zillion crazy tests to determine kernel
>> features, and recently, just not wrapping new syscalls at all because
>> they are just frustrated at the compatibility issues over time.  Let's
>> not make their life any harder than it has to be please.
>>
>> I don't see how any of the kselftest programs are any different than any
>> other userspace program that wants to use our kernel api, and as such,
>> any version of kselftest should be able to successfully run on any
>> kernel release.  If not, then we messed up in how we either wrote the
>> test, or how we added a new kernel api.  Neither is acceptable.
>
> That's a fair point.  We could add SECCOMP_GET_FEATURES that returns a
> mask of otherwise-difficult-to-probe features.

But that's silly. The self tests includes all kinds of bug fix tests,
and adding each of those to some features list seems like crazy
overkill. And every API in the kernel is going to do this?

> Greg, for context, the issue here is that we made what was arguably a
> design error in seccomp's interaction with ptrace.  After determining
> that fixing it solved a bunch of problems and didn't break any user
> programs, we fixed it.  There might be new code that relies on the fix
> being present in the sense that it would be insecure without the fix.
>
> The problem is that the fix is moderately intrusive and doesn't seem
> like a great candidate for backporting, although we could plausibly do
> it.

Even if we did all this for seccomp, we're left with the general case.
This is not a situation unique to seccomp. Behaviors and bug fix tests
are added to selftests over time, and not all of those things are
backport-worthy. If a certain test is desired for a stable kernel, we
should backport the test.

-Kees

-- 
Kees Cook
Pixel Security

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Kees Cook]

On Thu, Jun 22, 2017 at 7:40 PM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman
>  wrote:
>> On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>>> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>>> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> >>> wrote:
>>>  Hi Kees, Andy,
>>> 
>>>  On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>>> > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>>> > feature and test together.
>>> > - This one also seems like a security hole being closed, and the
>>> > 'feature' could be a candidate for stable backports, but Arnd tried
>>> > that, and it was quite non-trivial. So perhaps  we'll need some help
>>> > from the subsystem developers here.
>>> 
>>>  Could you please help us sort this out? Our goal is to help Greg with
>>>  testing stable kernels, and currently the seccomp tests fail due to
>>>  missing feature (seccomp ptrace hole closure) getting tested via
>>>  latest kselftest.
>>> 
>>>  If you feel the feature isn't a stable candidate, then could you
>>>  please help make the test degrade gracefully in its absence?
>>> >>>
>>> >>> I don't really want to have that change be a backport -- it's quite
>>> >>> invasive across multiple architectures.
>>> >>>
>>> >>> I would say just add a kernel version check to the test. This is
>>> >>> probably not the only selftest that will need such things. :)
>>> >>
>>> >> Adding release checks to selftests is going to problematic for 
>>> >> maintenance.
>>> >> Tests should fail gracefully if feature isn't supported in older kernels.
>>> >>
>>> >> Several tests do that now and please find a way to check for dependencies
>>> >> and feature availability and fail the test gracefully. If there is a test
>>> >> that can't do that for some reason, we can discuss it, but as a general
>>> >> rule, I don't want to see kselftest patches that check release.
>>> >
>>> > If a future kernel inadvertently loses the new feature and degrades to
>>> > the behavior of old kernels, that would be a serious bug and should be
>>> > caught.
>>>
>>> Right. I really think stable kernels should be tested with their own
>>> selftests. If some test is needed in a stable kernel it should be
>>> backported to that stable kernel.
>>
>> Well, ideally all new features added to the kernel should be able to be
>> detected by userspace somehow if they are present or not.
>>
>> How do you expect a program to know if a feature has "failed" or is just
>> "not enabled/present in this kernel"?  Normally with syscalls this is
>> easy, same for sysfs changes.  Is seccomp in the bad state where there
>> is no way to detect the two different states here?  How is userspace
>> supposed to deal with that?
>>
>> We make fun of glibc having a zillion crazy tests to determine kernel
>> features, and recently, just not wrapping new syscalls at all because
>> they are just frustrated at the compatibility issues over time.  Let's
>> not make their life any harder than it has to be please.
>>
>> I don't see how any of the kselftest programs are any different than any
>> other userspace program that wants to use our kernel api, and as such,
>> any version of kselftest should be able to successfully run on any
>> kernel release.  If not, then we messed up in how we either wrote the
>> test, or how we added a new kernel api.  Neither is acceptable.
>
> That's a fair point.  We could add SECCOMP_GET_FEATURES that returns a
> mask of otherwise-difficult-to-probe features.

But that's silly. The self tests includes all kinds of bug fix tests,
and adding each of those to some features list seems like crazy
overkill. And every API in the kernel is going to do this?

> Greg, for context, the issue here is that we made what was arguably a
> design error in seccomp's interaction with ptrace.  After determining
> that fixing it solved a bunch of problems and didn't break any user
> programs, we fixed it.  There might be new code that relies on the fix
> being present in the sense that it would be insecure without the fix.
>
> The problem is that the fix is moderately intrusive and doesn't seem
> like a great candidate for backporting, although we could plausibly do
> it.

Even if we did all this for seccomp, we're left with the general case.
This is not a situation unique to seccomp. Behaviors and bug fix tests
are added to selftests over time, and not all of those things are
backport-worthy. If a certain test is desired for a stable kernel, we
should backport the test.

-Kees

-- 
Kees Cook
Pixel Security

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Sumit Semwal]

Hi Shuah,

On 23 June 2017 at 01:53, Shuah Khan  wrote:
> Hi Tom,
>
> On 06/22/2017 01:48 PM, Tom Gall wrote:
>> Hi
>>
>> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>>> On 06/22/2017 11:50 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> wrote:
 Hi Kees, Andy,

 On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

 Could you please help us sort this out? Our goal is to help Greg with
 testing stable kernels, and currently the seccomp tests fail due to
 missing feature (seccomp ptrace hole closure) getting tested via
 latest kselftest.

 If you feel the feature isn't a stable candidate, then could you
 please help make the test degrade gracefully in its absence?
>>>
>>> In some cases, it is not easy to degrade and/or check for a feature.
>>> Probably several security features could fall in this bucket.
>>>
>>>
>>> I don't really want to have that change be a backport -- it's quite
>>> invasive across multiple architectures.
>>>
>>> Agreed. The same test for kernel applies to tests as well. If a kernel
>>> feature can't be back-ported, the test for that feature will fall in the
>>> same bucket. It shouldn't be back-ported.
>>>
>>>
>>> I would say just add a kernel version check to the test. This is
>>> probably not the only selftest that will need such things. :)
>>
>> Adding release checks to selftests is going to problematic for 
>> maintenance.
>> Tests should fail gracefully if feature isn't supported in older kernels.
>>
>> Several tests do that now and please find a way to check for dependencies
>> and feature availability and fail the test gracefully. If there is a test
>> that can't do that for some reason, we can discuss it, but as a general
>> rule, I don't want to see kselftest patches that check release.
>
> If a future kernel inadvertently loses the new feature and degrades to
> the behavior of old kernels, that would be a serious bug and should be
> caught.
>>>
>>> Agreed. If I understand you correctly, by not testing stable kernels
>>> with their own selftests, some serious bugs could go undetected.
>>
>> Personally I'm a bit skeptical. I think the reasoning is more that the
>> latest selftests provide more coverage, and therefore should be better
>> tests, even on older kernels.
>
> The assumption that "the latest selftests provide more coverage, and
> therefore should be better tests, even on older kernels." is incorrect.
>
> Selftests in general track the kernel features. In some cases, new
> tests could be added that provide better coverage on older kernels,
> however, it is more likely that new tests are added to test new kernel
> features and enhancements to existing features. Based on the second
> "enhancements to existing features" it is more important to test newer
> kernels with older selftests. This does happen in kernel integration
> cycles during development.
>
> As a general rule, testing stable kernels with their own selftests will
> yield the best results.
>
I would have agreed totally, if the selftests and the kernel were in
sync since forever. But since the kselftests are a comparatively
recent addition, the number of tests available for features existing
in LTS kernels is really quite small. Just as a comparison, 4.4-LTS
misses tests for bpf, cpufreq, gpio, media_tests, networking, prctl,
to name a few.

Also, while trying to run kselftests from later kernels with 4.4, we
only had a few failures for existing features, while most other tests
ran ok. Just another data point.

>>

 Right. I really think stable kernels should be tested with their own
 selftests. If some test is needed in a stable kernel it should be
 backported to that stable kernel.
>>>
>>> Correct. This is always a safe option. There might be cases that even
>>> prevent tests being built, especially if a new feature adds new fields
>>> to an existing structure.
>>>
>>> It appears in some cases, users want to run newer tests on older kernels.
>>> Some tests can clearly detect feature support using module presence and/or
>>> Kconfig enabled or disabled. These are conditions even on a kernel that
>>> supports a new module or new 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Sumit Semwal]

Hi Shuah,

On 23 June 2017 at 01:53, Shuah Khan  wrote:
> Hi Tom,
>
> On 06/22/2017 01:48 PM, Tom Gall wrote:
>> Hi
>>
>> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>>> On 06/22/2017 11:50 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> wrote:
 Hi Kees, Andy,

 On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

 Could you please help us sort this out? Our goal is to help Greg with
 testing stable kernels, and currently the seccomp tests fail due to
 missing feature (seccomp ptrace hole closure) getting tested via
 latest kselftest.

 If you feel the feature isn't a stable candidate, then could you
 please help make the test degrade gracefully in its absence?
>>>
>>> In some cases, it is not easy to degrade and/or check for a feature.
>>> Probably several security features could fall in this bucket.
>>>
>>>
>>> I don't really want to have that change be a backport -- it's quite
>>> invasive across multiple architectures.
>>>
>>> Agreed. The same test for kernel applies to tests as well. If a kernel
>>> feature can't be back-ported, the test for that feature will fall in the
>>> same bucket. It shouldn't be back-ported.
>>>
>>>
>>> I would say just add a kernel version check to the test. This is
>>> probably not the only selftest that will need such things. :)
>>
>> Adding release checks to selftests is going to problematic for 
>> maintenance.
>> Tests should fail gracefully if feature isn't supported in older kernels.
>>
>> Several tests do that now and please find a way to check for dependencies
>> and feature availability and fail the test gracefully. If there is a test
>> that can't do that for some reason, we can discuss it, but as a general
>> rule, I don't want to see kselftest patches that check release.
>
> If a future kernel inadvertently loses the new feature and degrades to
> the behavior of old kernels, that would be a serious bug and should be
> caught.
>>>
>>> Agreed. If I understand you correctly, by not testing stable kernels
>>> with their own selftests, some serious bugs could go undetected.
>>
>> Personally I'm a bit skeptical. I think the reasoning is more that the
>> latest selftests provide more coverage, and therefore should be better
>> tests, even on older kernels.
>
> The assumption that "the latest selftests provide more coverage, and
> therefore should be better tests, even on older kernels." is incorrect.
>
> Selftests in general track the kernel features. In some cases, new
> tests could be added that provide better coverage on older kernels,
> however, it is more likely that new tests are added to test new kernel
> features and enhancements to existing features. Based on the second
> "enhancements to existing features" it is more important to test newer
> kernels with older selftests. This does happen in kernel integration
> cycles during development.
>
> As a general rule, testing stable kernels with their own selftests will
> yield the best results.
>
I would have agreed totally, if the selftests and the kernel were in
sync since forever. But since the kselftests are a comparatively
recent addition, the number of tests available for features existing
in LTS kernels is really quite small. Just as a comparison, 4.4-LTS
misses tests for bpf, cpufreq, gpio, media_tests, networking, prctl,
to name a few.

Also, while trying to run kselftests from later kernels with 4.4, we
only had a few failures for existing features, while most other tests
ran ok. Just another data point.

>>

 Right. I really think stable kernels should be tested with their own
 selftests. If some test is needed in a stable kernel it should be
 backported to that stable kernel.
>>>
>>> Correct. This is always a safe option. There might be cases that even
>>> prevent tests being built, especially if a new feature adds new fields
>>> to an existing structure.
>>>
>>> It appears in some cases, users want to run newer tests on older kernels.
>>> Some tests can clearly detect feature support using module presence and/or
>>> Kconfig enabled or disabled. These are conditions even on a kernel that
>>> supports a new module or new config option. The kernel the test is running
>>> on might not have the feature enabled or module might not be present. 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Andy Lutomirski]

On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman
 wrote:
> On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> >>> wrote:
>>  Hi Kees, Andy,
>> 
>>  On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>> > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> > feature and test together.
>> > - This one also seems like a security hole being closed, and the
>> > 'feature' could be a candidate for stable backports, but Arnd tried
>> > that, and it was quite non-trivial. So perhaps  we'll need some help
>> > from the subsystem developers here.
>> 
>>  Could you please help us sort this out? Our goal is to help Greg with
>>  testing stable kernels, and currently the seccomp tests fail due to
>>  missing feature (seccomp ptrace hole closure) getting tested via
>>  latest kselftest.
>> 
>>  If you feel the feature isn't a stable candidate, then could you
>>  please help make the test degrade gracefully in its absence?
>> >>>
>> >>> I don't really want to have that change be a backport -- it's quite
>> >>> invasive across multiple architectures.
>> >>>
>> >>> I would say just add a kernel version check to the test. This is
>> >>> probably not the only selftest that will need such things. :)
>> >>
>> >> Adding release checks to selftests is going to problematic for 
>> >> maintenance.
>> >> Tests should fail gracefully if feature isn't supported in older kernels.
>> >>
>> >> Several tests do that now and please find a way to check for dependencies
>> >> and feature availability and fail the test gracefully. If there is a test
>> >> that can't do that for some reason, we can discuss it, but as a general
>> >> rule, I don't want to see kselftest patches that check release.
>> >
>> > If a future kernel inadvertently loses the new feature and degrades to
>> > the behavior of old kernels, that would be a serious bug and should be
>> > caught.
>>
>> Right. I really think stable kernels should be tested with their own
>> selftests. If some test is needed in a stable kernel it should be
>> backported to that stable kernel.
>
> Well, ideally all new features added to the kernel should be able to be
> detected by userspace somehow if they are present or not.
>
> How do you expect a program to know if a feature has "failed" or is just
> "not enabled/present in this kernel"?  Normally with syscalls this is
> easy, same for sysfs changes.  Is seccomp in the bad state where there
> is no way to detect the two different states here?  How is userspace
> supposed to deal with that?
>
> We make fun of glibc having a zillion crazy tests to determine kernel
> features, and recently, just not wrapping new syscalls at all because
> they are just frustrated at the compatibility issues over time.  Let's
> not make their life any harder than it has to be please.
>
> I don't see how any of the kselftest programs are any different than any
> other userspace program that wants to use our kernel api, and as such,
> any version of kselftest should be able to successfully run on any
> kernel release.  If not, then we messed up in how we either wrote the
> test, or how we added a new kernel api.  Neither is acceptable.

That's a fair point.  We could add SECCOMP_GET_FEATURES that returns a
mask of otherwise-difficult-to-probe features.

Greg, for context, the issue here is that we made what was arguably a
design error in seccomp's interaction with ptrace.  After determining
that fixing it solved a bunch of problems and didn't break any user
programs, we fixed it.  There might be new code that relies on the fix
being present in the sense that it would be insecure without the fix.

The problem is that the fix is moderately intrusive and doesn't seem
like a great candidate for backporting, although we could plausibly do
it.

--Andy

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Andy Lutomirski]

On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman
 wrote:
> On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> >>> wrote:
>>  Hi Kees, Andy,
>> 
>>  On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>> > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> > feature and test together.
>> > - This one also seems like a security hole being closed, and the
>> > 'feature' could be a candidate for stable backports, but Arnd tried
>> > that, and it was quite non-trivial. So perhaps  we'll need some help
>> > from the subsystem developers here.
>> 
>>  Could you please help us sort this out? Our goal is to help Greg with
>>  testing stable kernels, and currently the seccomp tests fail due to
>>  missing feature (seccomp ptrace hole closure) getting tested via
>>  latest kselftest.
>> 
>>  If you feel the feature isn't a stable candidate, then could you
>>  please help make the test degrade gracefully in its absence?
>> >>>
>> >>> I don't really want to have that change be a backport -- it's quite
>> >>> invasive across multiple architectures.
>> >>>
>> >>> I would say just add a kernel version check to the test. This is
>> >>> probably not the only selftest that will need such things. :)
>> >>
>> >> Adding release checks to selftests is going to problematic for 
>> >> maintenance.
>> >> Tests should fail gracefully if feature isn't supported in older kernels.
>> >>
>> >> Several tests do that now and please find a way to check for dependencies
>> >> and feature availability and fail the test gracefully. If there is a test
>> >> that can't do that for some reason, we can discuss it, but as a general
>> >> rule, I don't want to see kselftest patches that check release.
>> >
>> > If a future kernel inadvertently loses the new feature and degrades to
>> > the behavior of old kernels, that would be a serious bug and should be
>> > caught.
>>
>> Right. I really think stable kernels should be tested with their own
>> selftests. If some test is needed in a stable kernel it should be
>> backported to that stable kernel.
>
> Well, ideally all new features added to the kernel should be able to be
> detected by userspace somehow if they are present or not.
>
> How do you expect a program to know if a feature has "failed" or is just
> "not enabled/present in this kernel"?  Normally with syscalls this is
> easy, same for sysfs changes.  Is seccomp in the bad state where there
> is no way to detect the two different states here?  How is userspace
> supposed to deal with that?
>
> We make fun of glibc having a zillion crazy tests to determine kernel
> features, and recently, just not wrapping new syscalls at all because
> they are just frustrated at the compatibility issues over time.  Let's
> not make their life any harder than it has to be please.
>
> I don't see how any of the kselftest programs are any different than any
> other userspace program that wants to use our kernel api, and as such,
> any version of kselftest should be able to successfully run on any
> kernel release.  If not, then we messed up in how we either wrote the
> test, or how we added a new kernel api.  Neither is acceptable.

That's a fair point.  We could add SECCOMP_GET_FEATURES that returns a
mask of otherwise-difficult-to-probe features.

Greg, for context, the issue here is that we made what was arguably a
design error in seccomp's interaction with ptrace.  After determining
that fixing it solved a bunch of problems and didn't break any user
programs, we fixed it.  There might be new code that relies on the fix
being present in the sense that it would be insecure without the fix.

The problem is that the fix is moderately intrusive and doesn't seem
like a great candidate for backporting, although we could plausibly do
it.

--Andy

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Greg Kroah-Hartman]

On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
> >>> wrote:
>  Hi Kees, Andy,
> 
>  On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> > feature and test together.
> > - This one also seems like a security hole being closed, and the
> > 'feature' could be a candidate for stable backports, but Arnd tried
> > that, and it was quite non-trivial. So perhaps  we'll need some help
> > from the subsystem developers here.
> 
>  Could you please help us sort this out? Our goal is to help Greg with
>  testing stable kernels, and currently the seccomp tests fail due to
>  missing feature (seccomp ptrace hole closure) getting tested via
>  latest kselftest.
> 
>  If you feel the feature isn't a stable candidate, then could you
>  please help make the test degrade gracefully in its absence?
> >>>
> >>> I don't really want to have that change be a backport -- it's quite
> >>> invasive across multiple architectures.
> >>>
> >>> I would say just add a kernel version check to the test. This is
> >>> probably not the only selftest that will need such things. :)
> >>
> >> Adding release checks to selftests is going to problematic for maintenance.
> >> Tests should fail gracefully if feature isn't supported in older kernels.
> >>
> >> Several tests do that now and please find a way to check for dependencies
> >> and feature availability and fail the test gracefully. If there is a test
> >> that can't do that for some reason, we can discuss it, but as a general
> >> rule, I don't want to see kselftest patches that check release.
> >
> > If a future kernel inadvertently loses the new feature and degrades to
> > the behavior of old kernels, that would be a serious bug and should be
> > caught.
> 
> Right. I really think stable kernels should be tested with their own
> selftests. If some test is needed in a stable kernel it should be
> backported to that stable kernel.

Well, ideally all new features added to the kernel should be able to be
detected by userspace somehow if they are present or not.

How do you expect a program to know if a feature has "failed" or is just
"not enabled/present in this kernel"?  Normally with syscalls this is
easy, same for sysfs changes.  Is seccomp in the bad state where there
is no way to detect the two different states here?  How is userspace
supposed to deal with that?

We make fun of glibc having a zillion crazy tests to determine kernel
features, and recently, just not wrapping new syscalls at all because
they are just frustrated at the compatibility issues over time.  Let's
not make their life any harder than it has to be please.

I don't see how any of the kselftest programs are any different than any
other userspace program that wants to use our kernel api, and as such,
any version of kselftest should be able to successfully run on any
kernel release.  If not, then we messed up in how we either wrote the
test, or how we added a new kernel api.  Neither is acceptable.

thanks,

greg k-h

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Greg Kroah-Hartman]

On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
> >>> wrote:
>  Hi Kees, Andy,
> 
>  On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> > feature and test together.
> > - This one also seems like a security hole being closed, and the
> > 'feature' could be a candidate for stable backports, but Arnd tried
> > that, and it was quite non-trivial. So perhaps  we'll need some help
> > from the subsystem developers here.
> 
>  Could you please help us sort this out? Our goal is to help Greg with
>  testing stable kernels, and currently the seccomp tests fail due to
>  missing feature (seccomp ptrace hole closure) getting tested via
>  latest kselftest.
> 
>  If you feel the feature isn't a stable candidate, then could you
>  please help make the test degrade gracefully in its absence?
> >>>
> >>> I don't really want to have that change be a backport -- it's quite
> >>> invasive across multiple architectures.
> >>>
> >>> I would say just add a kernel version check to the test. This is
> >>> probably not the only selftest that will need such things. :)
> >>
> >> Adding release checks to selftests is going to problematic for maintenance.
> >> Tests should fail gracefully if feature isn't supported in older kernels.
> >>
> >> Several tests do that now and please find a way to check for dependencies
> >> and feature availability and fail the test gracefully. If there is a test
> >> that can't do that for some reason, we can discuss it, but as a general
> >> rule, I don't want to see kselftest patches that check release.
> >
> > If a future kernel inadvertently loses the new feature and degrades to
> > the behavior of old kernels, that would be a serious bug and should be
> > caught.
> 
> Right. I really think stable kernels should be tested with their own
> selftests. If some test is needed in a stable kernel it should be
> backported to that stable kernel.

Well, ideally all new features added to the kernel should be able to be
detected by userspace somehow if they are present or not.

How do you expect a program to know if a feature has "failed" or is just
"not enabled/present in this kernel"?  Normally with syscalls this is
easy, same for sysfs changes.  Is seccomp in the bad state where there
is no way to detect the two different states here?  How is userspace
supposed to deal with that?

We make fun of glibc having a zillion crazy tests to determine kernel
features, and recently, just not wrapping new syscalls at all because
they are just frustrated at the compatibility issues over time.  Let's
not make their life any harder than it has to be please.

I don't see how any of the kselftest programs are any different than any
other userspace program that wants to use our kernel api, and as such,
any version of kselftest should be able to successfully run on any
kernel release.  If not, then we messed up in how we either wrote the
test, or how we added a new kernel api.  Neither is acceptable.

thanks,

greg k-h

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

Hi Tom,

On 06/22/2017 01:48 PM, Tom Gall wrote:
> Hi
> 
> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>> On 06/22/2017 11:50 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
 On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> wrote:
>>> Hi Kees, Andy,
>>>
>>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
 feature and test together.
 - This one also seems like a security hole being closed, and the
 'feature' could be a candidate for stable backports, but Arnd tried
 that, and it was quite non-trivial. So perhaps  we'll need some help
 from the subsystem developers here.
>>>
>>> Could you please help us sort this out? Our goal is to help Greg with
>>> testing stable kernels, and currently the seccomp tests fail due to
>>> missing feature (seccomp ptrace hole closure) getting tested via
>>> latest kselftest.
>>>
>>> If you feel the feature isn't a stable candidate, then could you
>>> please help make the test degrade gracefully in its absence?
>>
>> In some cases, it is not easy to degrade and/or check for a feature.
>> Probably several security features could fall in this bucket.
>>
>>
>> I don't really want to have that change be a backport -- it's quite
>> invasive across multiple architectures.
>>
>> Agreed. The same test for kernel applies to tests as well. If a kernel
>> feature can't be back-ported, the test for that feature will fall in the
>> same bucket. It shouldn't be back-ported.
>>
>>
>> I would say just add a kernel version check to the test. This is
>> probably not the only selftest that will need such things. :)
>
> Adding release checks to selftests is going to problematic for 
> maintenance.
> Tests should fail gracefully if feature isn't supported in older kernels.
>
> Several tests do that now and please find a way to check for dependencies
> and feature availability and fail the test gracefully. If there is a test
> that can't do that for some reason, we can discuss it, but as a general
> rule, I don't want to see kselftest patches that check release.

 If a future kernel inadvertently loses the new feature and degrades to
 the behavior of old kernels, that would be a serious bug and should be
 caught.
>>
>> Agreed. If I understand you correctly, by not testing stable kernels
>> with their own selftests, some serious bugs could go undetected.
> 
> Personally I'm a bit skeptical. I think the reasoning is more that the
> latest selftests provide more coverage, and therefore should be better
> tests, even on older kernels.

The assumption that "the latest selftests provide more coverage, and
therefore should be better tests, even on older kernels." is incorrect.

Selftests in general track the kernel features. In some cases, new
tests could be added that provide better coverage on older kernels,
however, it is more likely that new tests are added to test new kernel
features and enhancements to existing features. Based on the second
"enhancements to existing features" it is more important to test newer
kernels with older selftests. This does happen in kernel integration
cycles during development.

As a general rule, testing stable kernels with their own selftests will
yield the best results.

> 
>>>
>>> Right. I really think stable kernels should be tested with their own
>>> selftests. If some test is needed in a stable kernel it should be
>>> backported to that stable kernel.
>>
>> Correct. This is always a safe option. There might be cases that even
>> prevent tests being built, especially if a new feature adds new fields
>> to an existing structure.
>>
>> It appears in some cases, users want to run newer tests on older kernels.
>> Some tests can clearly detect feature support using module presence and/or
>> Kconfig enabled or disabled. These are conditions even on a kernel that
>> supports a new module or new config option. The kernel the test is running
>> on might not have the feature enabled or module might not be present. In
>> these cases, it would be easier to detect and skip the test.
>>
>> However, some features aren't so easy. For example:
>>
>> - a new flag is added to a syscall, and new test is added. It might not
>>   be easy to detect that.
>> - We might have some tests that can't detect and skip.
>>
>> Based on this discussion, it is probably accurate to say:
>>
>> 1. It is recommended that selftests from the same release be run on the
>>kernel.
>> 2. Selftests from newer kernels will run on older kernels, user should
>>understand the risks such as some tests might fail and might 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

Hi Tom,

On 06/22/2017 01:48 PM, Tom Gall wrote:
> Hi
> 
> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>> On 06/22/2017 11:50 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
 On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> wrote:
>>> Hi Kees, Andy,
>>>
>>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
 feature and test together.
 - This one also seems like a security hole being closed, and the
 'feature' could be a candidate for stable backports, but Arnd tried
 that, and it was quite non-trivial. So perhaps  we'll need some help
 from the subsystem developers here.
>>>
>>> Could you please help us sort this out? Our goal is to help Greg with
>>> testing stable kernels, and currently the seccomp tests fail due to
>>> missing feature (seccomp ptrace hole closure) getting tested via
>>> latest kselftest.
>>>
>>> If you feel the feature isn't a stable candidate, then could you
>>> please help make the test degrade gracefully in its absence?
>>
>> In some cases, it is not easy to degrade and/or check for a feature.
>> Probably several security features could fall in this bucket.
>>
>>
>> I don't really want to have that change be a backport -- it's quite
>> invasive across multiple architectures.
>>
>> Agreed. The same test for kernel applies to tests as well. If a kernel
>> feature can't be back-ported, the test for that feature will fall in the
>> same bucket. It shouldn't be back-ported.
>>
>>
>> I would say just add a kernel version check to the test. This is
>> probably not the only selftest that will need such things. :)
>
> Adding release checks to selftests is going to problematic for 
> maintenance.
> Tests should fail gracefully if feature isn't supported in older kernels.
>
> Several tests do that now and please find a way to check for dependencies
> and feature availability and fail the test gracefully. If there is a test
> that can't do that for some reason, we can discuss it, but as a general
> rule, I don't want to see kselftest patches that check release.

 If a future kernel inadvertently loses the new feature and degrades to
 the behavior of old kernels, that would be a serious bug and should be
 caught.
>>
>> Agreed. If I understand you correctly, by not testing stable kernels
>> with their own selftests, some serious bugs could go undetected.
> 
> Personally I'm a bit skeptical. I think the reasoning is more that the
> latest selftests provide more coverage, and therefore should be better
> tests, even on older kernels.

The assumption that "the latest selftests provide more coverage, and
therefore should be better tests, even on older kernels." is incorrect.

Selftests in general track the kernel features. In some cases, new
tests could be added that provide better coverage on older kernels,
however, it is more likely that new tests are added to test new kernel
features and enhancements to existing features. Based on the second
"enhancements to existing features" it is more important to test newer
kernels with older selftests. This does happen in kernel integration
cycles during development.

As a general rule, testing stable kernels with their own selftests will
yield the best results.

> 
>>>
>>> Right. I really think stable kernels should be tested with their own
>>> selftests. If some test is needed in a stable kernel it should be
>>> backported to that stable kernel.
>>
>> Correct. This is always a safe option. There might be cases that even
>> prevent tests being built, especially if a new feature adds new fields
>> to an existing structure.
>>
>> It appears in some cases, users want to run newer tests on older kernels.
>> Some tests can clearly detect feature support using module presence and/or
>> Kconfig enabled or disabled. These are conditions even on a kernel that
>> supports a new module or new config option. The kernel the test is running
>> on might not have the feature enabled or module might not be present. In
>> these cases, it would be easier to detect and skip the test.
>>
>> However, some features aren't so easy. For example:
>>
>> - a new flag is added to a syscall, and new test is added. It might not
>>   be easy to detect that.
>> - We might have some tests that can't detect and skip.
>>
>> Based on this discussion, it is probably accurate to say:
>>
>> 1. It is recommended that selftests from the same release be run on the
>>kernel.
>> 2. Selftests from newer kernels will run on older kernels, user should
>>understand the risks such as some tests might fail and might not
>>detect feature degradation related bugs.
>> 3. Selftests will fail gracefully on older 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Tom Gall]

Hi

On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
> On 06/22/2017 11:50 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
 On 06/22/2017 10:53 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
> wrote:
>> Hi Kees, Andy,
>>
>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>>> feature and test together.
>>> - This one also seems like a security hole being closed, and the
>>> 'feature' could be a candidate for stable backports, but Arnd tried
>>> that, and it was quite non-trivial. So perhaps  we'll need some help
>>> from the subsystem developers here.
>>
>> Could you please help us sort this out? Our goal is to help Greg with
>> testing stable kernels, and currently the seccomp tests fail due to
>> missing feature (seccomp ptrace hole closure) getting tested via
>> latest kselftest.
>>
>> If you feel the feature isn't a stable candidate, then could you
>> please help make the test degrade gracefully in its absence?
>
> In some cases, it is not easy to degrade and/or check for a feature.
> Probably several security features could fall in this bucket.
>
>
> I don't really want to have that change be a backport -- it's quite
> invasive across multiple architectures.
>
> Agreed. The same test for kernel applies to tests as well. If a kernel
> feature can't be back-ported, the test for that feature will fall in the
> same bucket. It shouldn't be back-ported.
>
>
> I would say just add a kernel version check to the test. This is
> probably not the only selftest that will need such things. :)

 Adding release checks to selftests is going to problematic for maintenance.
 Tests should fail gracefully if feature isn't supported in older kernels.

 Several tests do that now and please find a way to check for dependencies
 and feature availability and fail the test gracefully. If there is a test
 that can't do that for some reason, we can discuss it, but as a general
 rule, I don't want to see kselftest patches that check release.
>>>
>>> If a future kernel inadvertently loses the new feature and degrades to
>>> the behavior of old kernels, that would be a serious bug and should be
>>> caught.
>
> Agreed. If I understand you correctly, by not testing stable kernels
> with their own selftests, some serious bugs could go undetected.

Personally I'm a bit skeptical. I think the reasoning is more that the
latest selftests provide more coverage, and therefore should be better
tests, even on older kernels.

>>
>> Right. I really think stable kernels should be tested with their own
>> selftests. If some test is needed in a stable kernel it should be
>> backported to that stable kernel.
>
> Correct. This is always a safe option. There might be cases that even
> prevent tests being built, especially if a new feature adds new fields
> to an existing structure.
>
> It appears in some cases, users want to run newer tests on older kernels.
> Some tests can clearly detect feature support using module presence and/or
> Kconfig enabled or disabled. These are conditions even on a kernel that
> supports a new module or new config option. The kernel the test is running
> on might not have the feature enabled or module might not be present. In
> these cases, it would be easier to detect and skip the test.
>
> However, some features aren't so easy. For example:
>
> - a new flag is added to a syscall, and new test is added. It might not
>   be easy to detect that.
> - We might have some tests that can't detect and skip.
>
> Based on this discussion, it is probably accurate to say:
>
> 1. It is recommended that selftests from the same release be run on the
>kernel.
> 2. Selftests from newer kernels will run on older kernels, user should
>understand the risks such as some tests might fail and might not
>detect feature degradation related bugs.
> 3. Selftests will fail gracefully on older releases if at all possible.

How about gracefully be skipped instead of fail?

The later suggests the test case in some situations can detect it's
pointless to run something and say as much instead of emitting a
failure that would be a waste of time to look into.

As another example take tools/testing/selftests/net/psock_fanout.c
On 4.9 it'll fail to compile (using master's selftests) because
PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for
that symbol and the psock_fanout test will compile and run just fine.

> Sumit!
>
> 1. What are the reasons for testing older kernel with selftests from
>newer kernels? What are the benefits you see for doing so?

I think the presumption is the latest greatest collection 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Tom Gall]

Hi

On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
> On 06/22/2017 11:50 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
 On 06/22/2017 10:53 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
> wrote:
>> Hi Kees, Andy,
>>
>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>>> feature and test together.
>>> - This one also seems like a security hole being closed, and the
>>> 'feature' could be a candidate for stable backports, but Arnd tried
>>> that, and it was quite non-trivial. So perhaps  we'll need some help
>>> from the subsystem developers here.
>>
>> Could you please help us sort this out? Our goal is to help Greg with
>> testing stable kernels, and currently the seccomp tests fail due to
>> missing feature (seccomp ptrace hole closure) getting tested via
>> latest kselftest.
>>
>> If you feel the feature isn't a stable candidate, then could you
>> please help make the test degrade gracefully in its absence?
>
> In some cases, it is not easy to degrade and/or check for a feature.
> Probably several security features could fall in this bucket.
>
>
> I don't really want to have that change be a backport -- it's quite
> invasive across multiple architectures.
>
> Agreed. The same test for kernel applies to tests as well. If a kernel
> feature can't be back-ported, the test for that feature will fall in the
> same bucket. It shouldn't be back-ported.
>
>
> I would say just add a kernel version check to the test. This is
> probably not the only selftest that will need such things. :)

 Adding release checks to selftests is going to problematic for maintenance.
 Tests should fail gracefully if feature isn't supported in older kernels.

 Several tests do that now and please find a way to check for dependencies
 and feature availability and fail the test gracefully. If there is a test
 that can't do that for some reason, we can discuss it, but as a general
 rule, I don't want to see kselftest patches that check release.
>>>
>>> If a future kernel inadvertently loses the new feature and degrades to
>>> the behavior of old kernels, that would be a serious bug and should be
>>> caught.
>
> Agreed. If I understand you correctly, by not testing stable kernels
> with their own selftests, some serious bugs could go undetected.

Personally I'm a bit skeptical. I think the reasoning is more that the
latest selftests provide more coverage, and therefore should be better
tests, even on older kernels.

>>
>> Right. I really think stable kernels should be tested with their own
>> selftests. If some test is needed in a stable kernel it should be
>> backported to that stable kernel.
>
> Correct. This is always a safe option. There might be cases that even
> prevent tests being built, especially if a new feature adds new fields
> to an existing structure.
>
> It appears in some cases, users want to run newer tests on older kernels.
> Some tests can clearly detect feature support using module presence and/or
> Kconfig enabled or disabled. These are conditions even on a kernel that
> supports a new module or new config option. The kernel the test is running
> on might not have the feature enabled or module might not be present. In
> these cases, it would be easier to detect and skip the test.
>
> However, some features aren't so easy. For example:
>
> - a new flag is added to a syscall, and new test is added. It might not
>   be easy to detect that.
> - We might have some tests that can't detect and skip.
>
> Based on this discussion, it is probably accurate to say:
>
> 1. It is recommended that selftests from the same release be run on the
>kernel.
> 2. Selftests from newer kernels will run on older kernels, user should
>understand the risks such as some tests might fail and might not
>detect feature degradation related bugs.
> 3. Selftests will fail gracefully on older releases if at all possible.

How about gracefully be skipped instead of fail?

The later suggests the test case in some situations can detect it's
pointless to run something and say as much instead of emitting a
failure that would be a waste of time to look into.

As another example take tools/testing/selftests/net/psock_fanout.c
On 4.9 it'll fail to compile (using master's selftests) because
PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for
that symbol and the psock_fanout test will compile and run just fine.

> Sumit!
>
> 1. What are the reasons for testing older kernel with selftests from
>newer kernels? What are the benefits you see for doing so?

I think the presumption is the latest greatest collection of selftests
are the best, most complete.

>I am looking to understand the need/reasons for this 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 11:50 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>>> On 06/22/2017 10:53 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
 wrote:
> Hi Kees, Andy,
>
> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> feature and test together.
>> - This one also seems like a security hole being closed, and the
>> 'feature' could be a candidate for stable backports, but Arnd tried
>> that, and it was quite non-trivial. So perhaps  we'll need some help
>> from the subsystem developers here.
>
> Could you please help us sort this out? Our goal is to help Greg with
> testing stable kernels, and currently the seccomp tests fail due to
> missing feature (seccomp ptrace hole closure) getting tested via
> latest kselftest.
>
> If you feel the feature isn't a stable candidate, then could you
> please help make the test degrade gracefully in its absence?

In some cases, it is not easy to degrade and/or check for a feature.
Probably several security features could fall in this bucket.


 I don't really want to have that change be a backport -- it's quite
 invasive across multiple architectures.

Agreed. The same test for kernel applies to tests as well. If a kernel
feature can't be back-ported, the test for that feature will fall in the
same bucket. It shouldn't be back-ported.


 I would say just add a kernel version check to the test. This is
 probably not the only selftest that will need such things. :)
>>>
>>> Adding release checks to selftests is going to problematic for maintenance.
>>> Tests should fail gracefully if feature isn't supported in older kernels.
>>>
>>> Several tests do that now and please find a way to check for dependencies
>>> and feature availability and fail the test gracefully. If there is a test
>>> that can't do that for some reason, we can discuss it, but as a general
>>> rule, I don't want to see kselftest patches that check release.
>>
>> If a future kernel inadvertently loses the new feature and degrades to
>> the behavior of old kernels, that would be a serious bug and should be
>> caught.

Agreed. If I understand you correctly, by not testing stable kernels
with their own selftests, some serious bugs could go undetected.

> 
> Right. I really think stable kernels should be tested with their own
> selftests. If some test is needed in a stable kernel it should be
> backported to that stable kernel.

Correct. This is always a safe option. There might be cases that even
prevent tests being built, especially if a new feature adds new fields
to an existing structure.

It appears in some cases, users want to run newer tests on older kernels.
Some tests can clearly detect feature support using module presence and/or
Kconfig enabled or disabled. These are conditions even on a kernel that
supports a new module or new config option. The kernel the test is running
on might not have the feature enabled or module might not be present. In
these cases, it would be easier to detect and skip the test.

However, some features aren't so easy. For example:

- a new flag is added to a syscall, and new test is added. It might not
  be easy to detect that.
- We might have some tests that can't detect and skip.

Based on this discussion, it is probably accurate to say:

1. It is recommended that selftests from the same release be run on the
   kernel.
2. Selftests from newer kernels will run on older kernels, user should
   understand the risks such as some tests might fail and might not
   detect feature degradation related bugs.
3. Selftests will fail gracefully on older releases if at all possible.

Sumit!

1. What are the reasons for testing older kernel with selftests from
   newer kernels? What are the benefits you see for doing so?
   
   I am looking to understand the need/reasons for this use-case. In our
   previous discussion on this subject, I did say, you should be able to
   do so with some exceptions.

2. Do you test kernels with the selftests from the same release?

3. Do you find testing with newer selftests to be useful?

thanks,
-- Shuah

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 11:50 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>>> On 06/22/2017 10:53 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
 wrote:
> Hi Kees, Andy,
>
> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> feature and test together.
>> - This one also seems like a security hole being closed, and the
>> 'feature' could be a candidate for stable backports, but Arnd tried
>> that, and it was quite non-trivial. So perhaps  we'll need some help
>> from the subsystem developers here.
>
> Could you please help us sort this out? Our goal is to help Greg with
> testing stable kernels, and currently the seccomp tests fail due to
> missing feature (seccomp ptrace hole closure) getting tested via
> latest kselftest.
>
> If you feel the feature isn't a stable candidate, then could you
> please help make the test degrade gracefully in its absence?

In some cases, it is not easy to degrade and/or check for a feature.
Probably several security features could fall in this bucket.


 I don't really want to have that change be a backport -- it's quite
 invasive across multiple architectures.

Agreed. The same test for kernel applies to tests as well. If a kernel
feature can't be back-ported, the test for that feature will fall in the
same bucket. It shouldn't be back-ported.


 I would say just add a kernel version check to the test. This is
 probably not the only selftest that will need such things. :)
>>>
>>> Adding release checks to selftests is going to problematic for maintenance.
>>> Tests should fail gracefully if feature isn't supported in older kernels.
>>>
>>> Several tests do that now and please find a way to check for dependencies
>>> and feature availability and fail the test gracefully. If there is a test
>>> that can't do that for some reason, we can discuss it, but as a general
>>> rule, I don't want to see kselftest patches that check release.
>>
>> If a future kernel inadvertently loses the new feature and degrades to
>> the behavior of old kernels, that would be a serious bug and should be
>> caught.

Agreed. If I understand you correctly, by not testing stable kernels
with their own selftests, some serious bugs could go undetected.

> 
> Right. I really think stable kernels should be tested with their own
> selftests. If some test is needed in a stable kernel it should be
> backported to that stable kernel.

Correct. This is always a safe option. There might be cases that even
prevent tests being built, especially if a new feature adds new fields
to an existing structure.

It appears in some cases, users want to run newer tests on older kernels.
Some tests can clearly detect feature support using module presence and/or
Kconfig enabled or disabled. These are conditions even on a kernel that
supports a new module or new config option. The kernel the test is running
on might not have the feature enabled or module might not be present. In
these cases, it would be easier to detect and skip the test.

However, some features aren't so easy. For example:

- a new flag is added to a syscall, and new test is added. It might not
  be easy to detect that.
- We might have some tests that can't detect and skip.

Based on this discussion, it is probably accurate to say:

1. It is recommended that selftests from the same release be run on the
   kernel.
2. Selftests from newer kernels will run on older kernels, user should
   understand the risks such as some tests might fail and might not
   detect feature degradation related bugs.
3. Selftests will fail gracefully on older releases if at all possible.

Sumit!

1. What are the reasons for testing older kernel with selftests from
   newer kernels? What are the benefits you see for doing so?
   
   I am looking to understand the need/reasons for this use-case. In our
   previous discussion on this subject, I did say, you should be able to
   do so with some exceptions.

2. Do you test kernels with the selftests from the same release?

3. Do you find testing with newer selftests to be useful?

thanks,
-- Shuah

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Kees Cook]

On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> wrote:
 Hi Kees, Andy,

 On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

 Could you please help us sort this out? Our goal is to help Greg with
 testing stable kernels, and currently the seccomp tests fail due to
 missing feature (seccomp ptrace hole closure) getting tested via
 latest kselftest.

 If you feel the feature isn't a stable candidate, then could you
 please help make the test degrade gracefully in its absence?
>>>
>>> I don't really want to have that change be a backport -- it's quite
>>> invasive across multiple architectures.
>>>
>>> I would say just add a kernel version check to the test. This is
>>> probably not the only selftest that will need such things. :)
>>
>> Adding release checks to selftests is going to problematic for maintenance.
>> Tests should fail gracefully if feature isn't supported in older kernels.
>>
>> Several tests do that now and please find a way to check for dependencies
>> and feature availability and fail the test gracefully. If there is a test
>> that can't do that for some reason, we can discuss it, but as a general
>> rule, I don't want to see kselftest patches that check release.
>
> If a future kernel inadvertently loses the new feature and degrades to
> the behavior of old kernels, that would be a serious bug and should be
> caught.

Right. I really think stable kernels should be tested with their own
selftests. If some test is needed in a stable kernel it should be
backported to that stable kernel.

-Kees

-- 
Kees Cook
Pixel Security

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Kees Cook]

On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> wrote:
 Hi Kees, Andy,

 On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

 Could you please help us sort this out? Our goal is to help Greg with
 testing stable kernels, and currently the seccomp tests fail due to
 missing feature (seccomp ptrace hole closure) getting tested via
 latest kselftest.

 If you feel the feature isn't a stable candidate, then could you
 please help make the test degrade gracefully in its absence?
>>>
>>> I don't really want to have that change be a backport -- it's quite
>>> invasive across multiple architectures.
>>>
>>> I would say just add a kernel version check to the test. This is
>>> probably not the only selftest that will need such things. :)
>>
>> Adding release checks to selftests is going to problematic for maintenance.
>> Tests should fail gracefully if feature isn't supported in older kernels.
>>
>> Several tests do that now and please find a way to check for dependencies
>> and feature availability and fail the test gracefully. If there is a test
>> that can't do that for some reason, we can discuss it, but as a general
>> rule, I don't want to see kselftest patches that check release.
>
> If a future kernel inadvertently loses the new feature and degrades to
> the behavior of old kernels, that would be a serious bug and should be
> caught.

Right. I really think stable kernels should be tested with their own
selftests. If some test is needed in a stable kernel it should be
backported to that stable kernel.

-Kees

-- 
Kees Cook
Pixel Security

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Andy Lutomirski]

On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> wrote:
>>> Hi Kees, Andy,
>>>
>>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
 feature and test together.
 - This one also seems like a security hole being closed, and the
 'feature' could be a candidate for stable backports, but Arnd tried
 that, and it was quite non-trivial. So perhaps  we'll need some help
 from the subsystem developers here.
>>>
>>> Could you please help us sort this out? Our goal is to help Greg with
>>> testing stable kernels, and currently the seccomp tests fail due to
>>> missing feature (seccomp ptrace hole closure) getting tested via
>>> latest kselftest.
>>>
>>> If you feel the feature isn't a stable candidate, then could you
>>> please help make the test degrade gracefully in its absence?
>>
>> I don't really want to have that change be a backport -- it's quite
>> invasive across multiple architectures.
>>
>> I would say just add a kernel version check to the test. This is
>> probably not the only selftest that will need such things. :)
>
> Adding release checks to selftests is going to problematic for maintenance.
> Tests should fail gracefully if feature isn't supported in older kernels.
>
> Several tests do that now and please find a way to check for dependencies
> and feature availability and fail the test gracefully. If there is a test
> that can't do that for some reason, we can discuss it, but as a general
> rule, I don't want to see kselftest patches that check release.

If a future kernel inadvertently loses the new feature and degrades to
the behavior of old kernels, that would be a serious bug and should be
caught.

--Andy

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Andy Lutomirski]

On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> wrote:
>>> Hi Kees, Andy,
>>>
>>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
 feature and test together.
 - This one also seems like a security hole being closed, and the
 'feature' could be a candidate for stable backports, but Arnd tried
 that, and it was quite non-trivial. So perhaps  we'll need some help
 from the subsystem developers here.
>>>
>>> Could you please help us sort this out? Our goal is to help Greg with
>>> testing stable kernels, and currently the seccomp tests fail due to
>>> missing feature (seccomp ptrace hole closure) getting tested via
>>> latest kselftest.
>>>
>>> If you feel the feature isn't a stable candidate, then could you
>>> please help make the test degrade gracefully in its absence?
>>
>> I don't really want to have that change be a backport -- it's quite
>> invasive across multiple architectures.
>>
>> I would say just add a kernel version check to the test. This is
>> probably not the only selftest that will need such things. :)
>
> Adding release checks to selftests is going to problematic for maintenance.
> Tests should fail gracefully if feature isn't supported in older kernels.
>
> Several tests do that now and please find a way to check for dependencies
> and feature availability and fail the test gracefully. If there is a test
> that can't do that for some reason, we can discuss it, but as a general
> rule, I don't want to see kselftest patches that check release.

If a future kernel inadvertently loses the new feature and degrades to
the behavior of old kernels, that would be a serious bug and should be
caught.

--Andy

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 10:53 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  wrote:
>> Hi Kees, Andy,
>>
>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>>> feature and test together.
>>> - This one also seems like a security hole being closed, and the
>>> 'feature' could be a candidate for stable backports, but Arnd tried
>>> that, and it was quite non-trivial. So perhaps  we'll need some help
>>> from the subsystem developers here.
>>
>> Could you please help us sort this out? Our goal is to help Greg with
>> testing stable kernels, and currently the seccomp tests fail due to
>> missing feature (seccomp ptrace hole closure) getting tested via
>> latest kselftest.
>>
>> If you feel the feature isn't a stable candidate, then could you
>> please help make the test degrade gracefully in its absence?
> 
> I don't really want to have that change be a backport -- it's quite
> invasive across multiple architectures.
> 
> I would say just add a kernel version check to the test. This is
> probably not the only selftest that will need such things. :)

Adding release checks to selftests is going to problematic for maintenance.
Tests should fail gracefully if feature isn't supported in older kernels.

Several tests do that now and please find a way to check for dependencies
and feature availability and fail the test gracefully. If there is a test
that can't do that for some reason, we can discuss it, but as a general
rule, I don't want to see kselftest patches that check release.

thanks,
-- Shuah

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 10:53 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  wrote:
>> Hi Kees, Andy,
>>
>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>>> feature and test together.
>>> - This one also seems like a security hole being closed, and the
>>> 'feature' could be a candidate for stable backports, but Arnd tried
>>> that, and it was quite non-trivial. So perhaps  we'll need some help
>>> from the subsystem developers here.
>>
>> Could you please help us sort this out? Our goal is to help Greg with
>> testing stable kernels, and currently the seccomp tests fail due to
>> missing feature (seccomp ptrace hole closure) getting tested via
>> latest kselftest.
>>
>> If you feel the feature isn't a stable candidate, then could you
>> please help make the test degrade gracefully in its absence?
> 
> I don't really want to have that change be a backport -- it's quite
> invasive across multiple architectures.
> 
> I would say just add a kernel version check to the test. This is
> probably not the only selftest that will need such things. :)

Adding release checks to selftests is going to problematic for maintenance.
Tests should fail gracefully if feature isn't supported in older kernels.

Several tests do that now and please find a way to check for dependencies
and feature availability and fail the test gracefully. If there is a test
that can't do that for some reason, we can discuss it, but as a general
rule, I don't want to see kselftest patches that check release.

thanks,
-- Shuah

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Kees Cook]

On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  wrote:
> Hi Kees, Andy,
>
> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> feature and test together.
>> - This one also seems like a security hole being closed, and the
>> 'feature' could be a candidate for stable backports, but Arnd tried
>> that, and it was quite non-trivial. So perhaps  we'll need some help
>> from the subsystem developers here.
>
> Could you please help us sort this out? Our goal is to help Greg with
> testing stable kernels, and currently the seccomp tests fail due to
> missing feature (seccomp ptrace hole closure) getting tested via
> latest kselftest.
>
> If you feel the feature isn't a stable candidate, then could you
> please help make the test degrade gracefully in its absence?

I don't really want to have that change be a backport -- it's quite
invasive across multiple architectures.

I would say just add a kernel version check to the test. This is
probably not the only selftest that will need such things. :)

I'd be happy to review such changes!

-Kees

-- 
Kees Cook
Pixel Security

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Kees Cook]

On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  wrote:
> Hi Kees, Andy,
>
> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> feature and test together.
>> - This one also seems like a security hole being closed, and the
>> 'feature' could be a candidate for stable backports, but Arnd tried
>> that, and it was quite non-trivial. So perhaps  we'll need some help
>> from the subsystem developers here.
>
> Could you please help us sort this out? Our goal is to help Greg with
> testing stable kernels, and currently the seccomp tests fail due to
> missing feature (seccomp ptrace hole closure) getting tested via
> latest kselftest.
>
> If you feel the feature isn't a stable candidate, then could you
> please help make the test degrade gracefully in its absence?

I don't really want to have that change be a backport -- it's quite
invasive across multiple architectures.

I would say just add a kernel version check to the test. This is
probably not the only selftest that will need such things. :)

I'd be happy to review such changes!

-Kees

-- 
Kees Cook
Pixel Security