Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Sumit Semwal]

Hi Andy,

On 24 June 2017 at 10:13, Greg Kroah-Hartman  wrote:
> On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote:
>> Greg, for context, the issue here is that we made what was arguably a
>> design error in seccomp's interaction with ptrace.  After determining
>> that fixing it solved a bunch of problems and didn't break any user
>> programs, we fixed it.  There might be new code that relies on the fix
>> being present in the sense that it would be insecure without the fix.
>>
>> The problem is that the fix is moderately intrusive and doesn't seem
>> like a great candidate for backporting, although we could plausibly do
>> it.
>
> That's fine, not all bugfixes that tests are created to find, should be
> backported.  That's up to the stable maintainers, or someone who has a
> device/vendor tree based on that kernel if they want to do that or not.
>
> That has nothing to do with the fact that the test should fail or
> gracefully degrade.  Tests should fail if the action that they are
> testing fails.  They should degrade and not run if the _feature_ they
> are testing is not present.

So, any updates on this yet - getting the seccomp tests to degrade
gracefully? I realise you mentioned that the fix could be intrusive;
just wanted to know if it was on your radar still.
>
> Yes, sometimes this is hard, like with the seccomp stuff, and will not
> always work, but that's the rule for our userspace api independant of
> any testing framework or code.
>
> Look at xfstests, no one gets mad when it adds a new test that old
> kernels fail at.  It's up to someone else to either backport the kernel
> change, if they want it fixed in an old kernel, not to have xfstests
> just not run it at all!  There's nothing different here either.
>
> thanks,
>
> greg k-h

Thanks much,
Sumit.

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Sumit Semwal]

Hi Andy,

On 24 June 2017 at 10:13, Greg Kroah-Hartman  wrote:
> On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote:
>> Greg, for context, the issue here is that we made what was arguably a
>> design error in seccomp's interaction with ptrace.  After determining
>> that fixing it solved a bunch of problems and didn't break any user
>> programs, we fixed it.  There might be new code that relies on the fix
>> being present in the sense that it would be insecure without the fix.
>>
>> The problem is that the fix is moderately intrusive and doesn't seem
>> like a great candidate for backporting, although we could plausibly do
>> it.
>
> That's fine, not all bugfixes that tests are created to find, should be
> backported.  That's up to the stable maintainers, or someone who has a
> device/vendor tree based on that kernel if they want to do that or not.
>
> That has nothing to do with the fact that the test should fail or
> gracefully degrade.  Tests should fail if the action that they are
> testing fails.  They should degrade and not run if the _feature_ they
> are testing is not present.

So, any updates on this yet - getting the seccomp tests to degrade
gracefully? I realise you mentioned that the fix could be intrusive;
just wanted to know if it was on your radar still.
>
> Yes, sometimes this is hard, like with the seccomp stuff, and will not
> always work, but that's the rule for our userspace api independant of
> any testing framework or code.
>
> Look at xfstests, no one gets mad when it adds a new test that old
> kernels fail at.  It's up to someone else to either backport the kernel
> change, if they want it fixed in an old kernel, not to have xfstests
> just not run it at all!  There's nothing different here either.
>
> thanks,
>
> greg k-h

Thanks much,
Sumit.

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Luis R. Rodriguez]

On Sat, Jun 24, 2017 at 06:45:37AM +0200, Greg Kroah-Hartman wrote:
> On Sat, Jun 24, 2017 at 02:34:07AM +0200, Luis R. Rodriguez wrote:
> > So taking the position that any kselftest script on linux-next or a future
> > kernel should never break stable implicate that *any* fix going upstream for
> > which there is a respective ksefltest test *must* have a stable upstream 
> > fix.
> 
> What, no!  If it's a bug, that kselftest points out, great, it's up to
> stable or a vendor to backport that fix if they want it.

Makes sense!

> Again, it's just like any other test suite, look at xfstests, no one
> gets mad when it adds new tests that fail on old kernels, due to bugs
> there, right?

Correct but that means there is an information gap of which test cases fit into
this category. An option to run kselftest with the ability to only run tests
designed to also include stable would be good, otherwise Sumit Semwal or others
would be sending reports or issues for things folks already designed the fix to
*not* be a stable candidate. This actually would also prove useful to
distributions to ensure to then run kselftest as part of their distribution
test suite with all bells and whistles enabled so that they can make a
determination of what questionable "fixes" things to fold in somehow.

Unless of course we want these discussions to purposely bubble up as an
alternative to this kselftest option. It seems rather wasteful though.

> > Its not obvious to me that everyone is aware of this. What do we do about
> > those cases where we *don't* want a stable fix due to the complexity?
> 
> That's up to the stable maintainers or the vendors that maintain their
> own kernel trees.

But if the above is already decided having folks send emails about it
seems pointless.

  Luis

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Luis R. Rodriguez]

On Sat, Jun 24, 2017 at 06:45:37AM +0200, Greg Kroah-Hartman wrote:
> On Sat, Jun 24, 2017 at 02:34:07AM +0200, Luis R. Rodriguez wrote:
> > So taking the position that any kselftest script on linux-next or a future
> > kernel should never break stable implicate that *any* fix going upstream for
> > which there is a respective ksefltest test *must* have a stable upstream 
> > fix.
> 
> What, no!  If it's a bug, that kselftest points out, great, it's up to
> stable or a vendor to backport that fix if they want it.

Makes sense!

> Again, it's just like any other test suite, look at xfstests, no one
> gets mad when it adds new tests that fail on old kernels, due to bugs
> there, right?

Correct but that means there is an information gap of which test cases fit into
this category. An option to run kselftest with the ability to only run tests
designed to also include stable would be good, otherwise Sumit Semwal or others
would be sending reports or issues for things folks already designed the fix to
*not* be a stable candidate. This actually would also prove useful to
distributions to ensure to then run kselftest as part of their distribution
test suite with all bells and whistles enabled so that they can make a
determination of what questionable "fixes" things to fold in somehow.

Unless of course we want these discussions to purposely bubble up as an
alternative to this kselftest option. It seems rather wasteful though.

> > Its not obvious to me that everyone is aware of this. What do we do about
> > those cases where we *don't* want a stable fix due to the complexity?
> 
> That's up to the stable maintainers or the vendors that maintain their
> own kernel trees.

But if the above is already decided having folks send emails about it
seems pointless.

  Luis

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Greg Kroah-Hartman]

On Sat, Jun 24, 2017 at 02:34:07AM +0200, Luis R. Rodriguez wrote:
> So taking the position that any kselftest script on linux-next or a future
> kernel should never break stable implicate that *any* fix going upstream for
> which there is a respective ksefltest test *must* have a stable upstream fix.

What, no!  If it's a bug, that kselftest points out, great, it's up to
stable or a vendor to backport that fix if they want it.

Again, it's just like any other test suite, look at xfstests, no one
gets mad when it adds new tests that fail on old kernels, due to bugs
there, right?

> Its not obvious to me that everyone is aware of this. What do we do about
> those cases where we *don't* want a stable fix due to the complexity?

That's up to the stable maintainers or the vendors that maintain their
own kernel trees.

thanks,

greg k-h

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Greg Kroah-Hartman]

On Sat, Jun 24, 2017 at 02:34:07AM +0200, Luis R. Rodriguez wrote:
> So taking the position that any kselftest script on linux-next or a future
> kernel should never break stable implicate that *any* fix going upstream for
> which there is a respective ksefltest test *must* have a stable upstream fix.

What, no!  If it's a bug, that kselftest points out, great, it's up to
stable or a vendor to backport that fix if they want it.

Again, it's just like any other test suite, look at xfstests, no one
gets mad when it adds new tests that fail on old kernels, due to bugs
there, right?

> Its not obvious to me that everyone is aware of this. What do we do about
> those cases where we *don't* want a stable fix due to the complexity?

That's up to the stable maintainers or the vendors that maintain their
own kernel trees.

thanks,

greg k-h

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Greg Kroah-Hartman]

On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote:
> Greg, for context, the issue here is that we made what was arguably a
> design error in seccomp's interaction with ptrace.  After determining
> that fixing it solved a bunch of problems and didn't break any user
> programs, we fixed it.  There might be new code that relies on the fix
> being present in the sense that it would be insecure without the fix.
> 
> The problem is that the fix is moderately intrusive and doesn't seem
> like a great candidate for backporting, although we could plausibly do
> it.

That's fine, not all bugfixes that tests are created to find, should be
backported.  That's up to the stable maintainers, or someone who has a
device/vendor tree based on that kernel if they want to do that or not.

That has nothing to do with the fact that the test should fail or
gracefully degrade.  Tests should fail if the action that they are
testing fails.  They should degrade and not run if the _feature_ they
are testing is not present.

Yes, sometimes this is hard, like with the seccomp stuff, and will not
always work, but that's the rule for our userspace api independant of
any testing framework or code.

Look at xfstests, no one gets mad when it adds a new test that old
kernels fail at.  It's up to someone else to either backport the kernel
change, if they want it fixed in an old kernel, not to have xfstests
just not run it at all!  There's nothing different here either.

thanks,

greg k-h

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Greg Kroah-Hartman]

On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote:
> Greg, for context, the issue here is that we made what was arguably a
> design error in seccomp's interaction with ptrace.  After determining
> that fixing it solved a bunch of problems and didn't break any user
> programs, we fixed it.  There might be new code that relies on the fix
> being present in the sense that it would be insecure without the fix.
> 
> The problem is that the fix is moderately intrusive and doesn't seem
> like a great candidate for backporting, although we could plausibly do
> it.

That's fine, not all bugfixes that tests are created to find, should be
backported.  That's up to the stable maintainers, or someone who has a
device/vendor tree based on that kernel if they want to do that or not.

That has nothing to do with the fact that the test should fail or
gracefully degrade.  Tests should fail if the action that they are
testing fails.  They should degrade and not run if the _feature_ they
are testing is not present.

Yes, sometimes this is hard, like with the seccomp stuff, and will not
always work, but that's the rule for our userspace api independant of
any testing framework or code.

Look at xfstests, no one gets mad when it adds a new test that old
kernels fail at.  It's up to someone else to either backport the kernel
change, if they want it fixed in an old kernel, not to have xfstests
just not run it at all!  There's nothing different here either.

thanks,

greg k-h

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Luis R. Rodriguez]

On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote:
> On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman
>  wrote:
> > On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
> >> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> >> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> >> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
> >> Right. I really think stable kernels should be tested with their own
> >> selftests. If some test is needed in a stable kernel it should be
> >> backported to that stable kernel.
> >
> > Well, ideally all new features added to the kernel should be able to be
> > detected by userspace somehow if they are present or not.
> >
> > How do you expect a program to know if a feature has "failed" or is just
> > "not enabled/present in this kernel"?  Normally with syscalls this is
> > easy, same for sysfs changes.  Is seccomp in the bad state where there
> > is no way to detect the two different states here?  How is userspace
> > supposed to deal with that?
> >
> > We make fun of glibc having a zillion crazy tests to determine kernel
> > features, and recently, just not wrapping new syscalls at all because
> > they are just frustrated at the compatibility issues over time.  Let's
> > not make their life any harder than it has to be please.
> >
> > I don't see how any of the kselftest programs are any different than any
> > other userspace program that wants to use our kernel api, and as such,
> > any version of kselftest should be able to successfully run on any
> > kernel release.  If not, then we messed up in how we either wrote the
> > test, or how we added a new kernel api.  Neither is acceptable.
> 
> That's a fair point.

I agreed with it as well just a few threads ago due to similar issues, however,
thinking this over I'm afraid this has some interesting side consequences for
fixes and what code goes upstream into kselftest.

<-- snip -->

> The problem is that the fix is moderately intrusive and doesn't seem
> like a great candidate for backporting, although we could plausibly do
> it.

Such is the case often actually.

So taking the position that any kselftest script on linux-next or a future
kernel should never break stable implicate that *any* fix going upstream for
which there is a respective ksefltest test *must* have a stable upstream fix.

Its not obvious to me that everyone is aware of this. What do we do about
those cases where we *don't* want a stable fix due to the complexity?

  Luis

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Luis R. Rodriguez]

On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote:
> On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman
>  wrote:
> > On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
> >> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> >> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> >> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
> >> Right. I really think stable kernels should be tested with their own
> >> selftests. If some test is needed in a stable kernel it should be
> >> backported to that stable kernel.
> >
> > Well, ideally all new features added to the kernel should be able to be
> > detected by userspace somehow if they are present or not.
> >
> > How do you expect a program to know if a feature has "failed" or is just
> > "not enabled/present in this kernel"?  Normally with syscalls this is
> > easy, same for sysfs changes.  Is seccomp in the bad state where there
> > is no way to detect the two different states here?  How is userspace
> > supposed to deal with that?
> >
> > We make fun of glibc having a zillion crazy tests to determine kernel
> > features, and recently, just not wrapping new syscalls at all because
> > they are just frustrated at the compatibility issues over time.  Let's
> > not make their life any harder than it has to be please.
> >
> > I don't see how any of the kselftest programs are any different than any
> > other userspace program that wants to use our kernel api, and as such,
> > any version of kselftest should be able to successfully run on any
> > kernel release.  If not, then we messed up in how we either wrote the
> > test, or how we added a new kernel api.  Neither is acceptable.
> 
> That's a fair point.

I agreed with it as well just a few threads ago due to similar issues, however,
thinking this over I'm afraid this has some interesting side consequences for
fixes and what code goes upstream into kselftest.

<-- snip -->

> The problem is that the fix is moderately intrusive and doesn't seem
> like a great candidate for backporting, although we could plausibly do
> it.

Such is the case often actually.

So taking the position that any kselftest script on linux-next or a future
kernel should never break stable implicate that *any* fix going upstream for
which there is a respective ksefltest test *must* have a stable upstream fix.

Its not obvious to me that everyone is aware of this. What do we do about
those cases where we *don't* want a stable fix due to the complexity?

  Luis

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Tom Gall]

Hi Shuah,

On Fri, Jun 23, 2017 at 2:03 PM, Shuah Khan  wrote:
> On 06/22/2017 01:48 PM, Tom Gall wrote:
>> Hi
>>
>> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>>> On 06/22/2017 11:50 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> wrote:
 Hi Kees, Andy,

 On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

 Could you please help us sort this out? Our goal is to help Greg with
 testing stable kernels, and currently the seccomp tests fail due to
 missing feature (seccomp ptrace hole closure) getting tested via
 latest kselftest.

 If you feel the feature isn't a stable candidate, then could you
 please help make the test degrade gracefully in its absence?
>>>
>>> In some cases, it is not easy to degrade and/or check for a feature.
>>> Probably several security features could fall in this bucket.
>>>
>>>
>>> I don't really want to have that change be a backport -- it's quite
>>> invasive across multiple architectures.
>>>
>>> Agreed. The same test for kernel applies to tests as well. If a kernel
>>> feature can't be back-ported, the test for that feature will fall in the
>>> same bucket. It shouldn't be back-ported.
>>>
>>>
>>> I would say just add a kernel version check to the test. This is
>>> probably not the only selftest that will need such things. :)
>>
>> Adding release checks to selftests is going to problematic for 
>> maintenance.
>> Tests should fail gracefully if feature isn't supported in older kernels.
>>
>> Several tests do that now and please find a way to check for dependencies
>> and feature availability and fail the test gracefully. If there is a test
>> that can't do that for some reason, we can discuss it, but as a general
>> rule, I don't want to see kselftest patches that check release.
>
> If a future kernel inadvertently loses the new feature and degrades to
> the behavior of old kernels, that would be a serious bug and should be
> caught.
>>>
>>> Agreed. If I understand you correctly, by not testing stable kernels
>>> with their own selftests, some serious bugs could go undetected.
>>
>> Personally I'm a bit skeptical. I think the reasoning is more that the
>> latest selftests provide more coverage, and therefore should be better
>> tests, even on older kernels.
>>

 Right. I really think stable kernels should be tested with their own
 selftests. If some test is needed in a stable kernel it should be
 backported to that stable kernel.
>>>
>>> Correct. This is always a safe option. There might be cases that even
>>> prevent tests being built, especially if a new feature adds new fields
>>> to an existing structure.
>>>
>>> It appears in some cases, users want to run newer tests on older kernels.
>>> Some tests can clearly detect feature support using module presence and/or
>>> Kconfig enabled or disabled. These are conditions even on a kernel that
>>> supports a new module or new config option. The kernel the test is running
>>> on might not have the feature enabled or module might not be present. In
>>> these cases, it would be easier to detect and skip the test.
>>>
>>> However, some features aren't so easy. For example:
>>>
>>> - a new flag is added to a syscall, and new test is added. It might not
>>>   be easy to detect that.
>>> - We might have some tests that can't detect and skip.
>>>
>>> Based on this discussion, it is probably accurate to say:
>>>
>>> 1. It is recommended that selftests from the same release be run on the
>>>kernel.
>>> 2. Selftests from newer kernels will run on older kernels, user should
>>>understand the risks such as some tests might fail and might not
>>>detect feature degradation related bugs.
>>> 3. Selftests will fail gracefully on older releases if at all possible.
>>
>> How about gracefully be skipped instead of fail?
>>
>> The later suggests the test case in some situations can detect it's
>> pointless to run something and say as much instead of emitting a
>> failure that would be a waste of time to look into.
>>
>> As another example take tools/testing/selftests/net/psock_fanout.c
>> On 4.9 it'll fail to compile (using master's selftests) because

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Tom Gall]

Hi Shuah,

On Fri, Jun 23, 2017 at 2:03 PM, Shuah Khan  wrote:
> On 06/22/2017 01:48 PM, Tom Gall wrote:
>> Hi
>>
>> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>>> On 06/22/2017 11:50 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> wrote:
 Hi Kees, Andy,

 On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

 Could you please help us sort this out? Our goal is to help Greg with
 testing stable kernels, and currently the seccomp tests fail due to
 missing feature (seccomp ptrace hole closure) getting tested via
 latest kselftest.

 If you feel the feature isn't a stable candidate, then could you
 please help make the test degrade gracefully in its absence?
>>>
>>> In some cases, it is not easy to degrade and/or check for a feature.
>>> Probably several security features could fall in this bucket.
>>>
>>>
>>> I don't really want to have that change be a backport -- it's quite
>>> invasive across multiple architectures.
>>>
>>> Agreed. The same test for kernel applies to tests as well. If a kernel
>>> feature can't be back-ported, the test for that feature will fall in the
>>> same bucket. It shouldn't be back-ported.
>>>
>>>
>>> I would say just add a kernel version check to the test. This is
>>> probably not the only selftest that will need such things. :)
>>
>> Adding release checks to selftests is going to problematic for 
>> maintenance.
>> Tests should fail gracefully if feature isn't supported in older kernels.
>>
>> Several tests do that now and please find a way to check for dependencies
>> and feature availability and fail the test gracefully. If there is a test
>> that can't do that for some reason, we can discuss it, but as a general
>> rule, I don't want to see kselftest patches that check release.
>
> If a future kernel inadvertently loses the new feature and degrades to
> the behavior of old kernels, that would be a serious bug and should be
> caught.
>>>
>>> Agreed. If I understand you correctly, by not testing stable kernels
>>> with their own selftests, some serious bugs could go undetected.
>>
>> Personally I'm a bit skeptical. I think the reasoning is more that the
>> latest selftests provide more coverage, and therefore should be better
>> tests, even on older kernels.
>>

 Right. I really think stable kernels should be tested with their own
 selftests. If some test is needed in a stable kernel it should be
 backported to that stable kernel.
>>>
>>> Correct. This is always a safe option. There might be cases that even
>>> prevent tests being built, especially if a new feature adds new fields
>>> to an existing structure.
>>>
>>> It appears in some cases, users want to run newer tests on older kernels.
>>> Some tests can clearly detect feature support using module presence and/or
>>> Kconfig enabled or disabled. These are conditions even on a kernel that
>>> supports a new module or new config option. The kernel the test is running
>>> on might not have the feature enabled or module might not be present. In
>>> these cases, it would be easier to detect and skip the test.
>>>
>>> However, some features aren't so easy. For example:
>>>
>>> - a new flag is added to a syscall, and new test is added. It might not
>>>   be easy to detect that.
>>> - We might have some tests that can't detect and skip.
>>>
>>> Based on this discussion, it is probably accurate to say:
>>>
>>> 1. It is recommended that selftests from the same release be run on the
>>>kernel.
>>> 2. Selftests from newer kernels will run on older kernels, user should
>>>understand the risks such as some tests might fail and might not
>>>detect feature degradation related bugs.
>>> 3. Selftests will fail gracefully on older releases if at all possible.
>>
>> How about gracefully be skipped instead of fail?
>>
>> The later suggests the test case in some situations can detect it's
>> pointless to run something and say as much instead of emitting a
>> failure that would be a waste of time to look into.
>>
>> As another example take tools/testing/selftests/net/psock_fanout.c
>> On 4.9 it'll fail to compile (using master's selftests) because
>> PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for
>> that symbol and the psock_fanout test will 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 01:48 PM, Tom Gall wrote:
> Hi
> 
> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>> On 06/22/2017 11:50 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
 On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> wrote:
>>> Hi Kees, Andy,
>>>
>>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
 feature and test together.
 - This one also seems like a security hole being closed, and the
 'feature' could be a candidate for stable backports, but Arnd tried
 that, and it was quite non-trivial. So perhaps  we'll need some help
 from the subsystem developers here.
>>>
>>> Could you please help us sort this out? Our goal is to help Greg with
>>> testing stable kernels, and currently the seccomp tests fail due to
>>> missing feature (seccomp ptrace hole closure) getting tested via
>>> latest kselftest.
>>>
>>> If you feel the feature isn't a stable candidate, then could you
>>> please help make the test degrade gracefully in its absence?
>>
>> In some cases, it is not easy to degrade and/or check for a feature.
>> Probably several security features could fall in this bucket.
>>
>>
>> I don't really want to have that change be a backport -- it's quite
>> invasive across multiple architectures.
>>
>> Agreed. The same test for kernel applies to tests as well. If a kernel
>> feature can't be back-ported, the test for that feature will fall in the
>> same bucket. It shouldn't be back-ported.
>>
>>
>> I would say just add a kernel version check to the test. This is
>> probably not the only selftest that will need such things. :)
>
> Adding release checks to selftests is going to problematic for 
> maintenance.
> Tests should fail gracefully if feature isn't supported in older kernels.
>
> Several tests do that now and please find a way to check for dependencies
> and feature availability and fail the test gracefully. If there is a test
> that can't do that for some reason, we can discuss it, but as a general
> rule, I don't want to see kselftest patches that check release.

 If a future kernel inadvertently loses the new feature and degrades to
 the behavior of old kernels, that would be a serious bug and should be
 caught.
>>
>> Agreed. If I understand you correctly, by not testing stable kernels
>> with their own selftests, some serious bugs could go undetected.
> 
> Personally I'm a bit skeptical. I think the reasoning is more that the
> latest selftests provide more coverage, and therefore should be better
> tests, even on older kernels.
> 
>>>
>>> Right. I really think stable kernels should be tested with their own
>>> selftests. If some test is needed in a stable kernel it should be
>>> backported to that stable kernel.
>>
>> Correct. This is always a safe option. There might be cases that even
>> prevent tests being built, especially if a new feature adds new fields
>> to an existing structure.
>>
>> It appears in some cases, users want to run newer tests on older kernels.
>> Some tests can clearly detect feature support using module presence and/or
>> Kconfig enabled or disabled. These are conditions even on a kernel that
>> supports a new module or new config option. The kernel the test is running
>> on might not have the feature enabled or module might not be present. In
>> these cases, it would be easier to detect and skip the test.
>>
>> However, some features aren't so easy. For example:
>>
>> - a new flag is added to a syscall, and new test is added. It might not
>>   be easy to detect that.
>> - We might have some tests that can't detect and skip.
>>
>> Based on this discussion, it is probably accurate to say:
>>
>> 1. It is recommended that selftests from the same release be run on the
>>kernel.
>> 2. Selftests from newer kernels will run on older kernels, user should
>>understand the risks such as some tests might fail and might not
>>detect feature degradation related bugs.
>> 3. Selftests will fail gracefully on older releases if at all possible.
> 
> How about gracefully be skipped instead of fail?
> 
> The later suggests the test case in some situations can detect it's
> pointless to run something and say as much instead of emitting a
> failure that would be a waste of time to look into.
> 
> As another example take tools/testing/selftests/net/psock_fanout.c
> On 4.9 it'll fail to compile (using master's selftests) because
> PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for
> that symbol and the psock_fanout test will compile and run just fine.

Unfortunately adding 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 01:48 PM, Tom Gall wrote:
> Hi
> 
> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>> On 06/22/2017 11:50 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
 On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> wrote:
>>> Hi Kees, Andy,
>>>
>>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
 feature and test together.
 - This one also seems like a security hole being closed, and the
 'feature' could be a candidate for stable backports, but Arnd tried
 that, and it was quite non-trivial. So perhaps  we'll need some help
 from the subsystem developers here.
>>>
>>> Could you please help us sort this out? Our goal is to help Greg with
>>> testing stable kernels, and currently the seccomp tests fail due to
>>> missing feature (seccomp ptrace hole closure) getting tested via
>>> latest kselftest.
>>>
>>> If you feel the feature isn't a stable candidate, then could you
>>> please help make the test degrade gracefully in its absence?
>>
>> In some cases, it is not easy to degrade and/or check for a feature.
>> Probably several security features could fall in this bucket.
>>
>>
>> I don't really want to have that change be a backport -- it's quite
>> invasive across multiple architectures.
>>
>> Agreed. The same test for kernel applies to tests as well. If a kernel
>> feature can't be back-ported, the test for that feature will fall in the
>> same bucket. It shouldn't be back-ported.
>>
>>
>> I would say just add a kernel version check to the test. This is
>> probably not the only selftest that will need such things. :)
>
> Adding release checks to selftests is going to problematic for 
> maintenance.
> Tests should fail gracefully if feature isn't supported in older kernels.
>
> Several tests do that now and please find a way to check for dependencies
> and feature availability and fail the test gracefully. If there is a test
> that can't do that for some reason, we can discuss it, but as a general
> rule, I don't want to see kselftest patches that check release.

 If a future kernel inadvertently loses the new feature and degrades to
 the behavior of old kernels, that would be a serious bug and should be
 caught.
>>
>> Agreed. If I understand you correctly, by not testing stable kernels
>> with their own selftests, some serious bugs could go undetected.
> 
> Personally I'm a bit skeptical. I think the reasoning is more that the
> latest selftests provide more coverage, and therefore should be better
> tests, even on older kernels.
> 
>>>
>>> Right. I really think stable kernels should be tested with their own
>>> selftests. If some test is needed in a stable kernel it should be
>>> backported to that stable kernel.
>>
>> Correct. This is always a safe option. There might be cases that even
>> prevent tests being built, especially if a new feature adds new fields
>> to an existing structure.
>>
>> It appears in some cases, users want to run newer tests on older kernels.
>> Some tests can clearly detect feature support using module presence and/or
>> Kconfig enabled or disabled. These are conditions even on a kernel that
>> supports a new module or new config option. The kernel the test is running
>> on might not have the feature enabled or module might not be present. In
>> these cases, it would be easier to detect and skip the test.
>>
>> However, some features aren't so easy. For example:
>>
>> - a new flag is added to a syscall, and new test is added. It might not
>>   be easy to detect that.
>> - We might have some tests that can't detect and skip.
>>
>> Based on this discussion, it is probably accurate to say:
>>
>> 1. It is recommended that selftests from the same release be run on the
>>kernel.
>> 2. Selftests from newer kernels will run on older kernels, user should
>>understand the risks such as some tests might fail and might not
>>detect feature degradation related bugs.
>> 3. Selftests will fail gracefully on older releases if at all possible.
> 
> How about gracefully be skipped instead of fail?
> 
> The later suggests the test case in some situations can detect it's
> pointless to run something and say as much instead of emitting a
> failure that would be a waste of time to look into.
> 
> As another example take tools/testing/selftests/net/psock_fanout.c
> On 4.9 it'll fail to compile (using master's selftests) because
> PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for
> that symbol and the psock_fanout test will compile and run just fine.

Unfortunately adding PACKET_FANOUT_FLAG_UNIQUEID isn't correct. This is
a new feature that went into 4.12. You don't want to define this for 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 10:02 PM, Sumit Semwal wrote:
> Hi Shuah,
> 
> On 23 June 2017 at 01:53, Shuah Khan  wrote:
>> Hi Tom,
>>
>> On 06/22/2017 01:48 PM, Tom Gall wrote:
>>> Hi
>>>
>>> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
 On 06/22/2017 11:50 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>>> On 06/22/2017 10:53 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal 
  wrote:
> Hi Kees, Andy,
>
> On 15 June 2017 at 23:26, Sumit Semwal  
> wrote:
>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> feature and test together.
>> - This one also seems like a security hole being closed, and the
>> 'feature' could be a candidate for stable backports, but Arnd tried
>> that, and it was quite non-trivial. So perhaps  we'll need some help
>> from the subsystem developers here.
>
> Could you please help us sort this out? Our goal is to help Greg with
> testing stable kernels, and currently the seccomp tests fail due to
> missing feature (seccomp ptrace hole closure) getting tested via
> latest kselftest.
>
> If you feel the feature isn't a stable candidate, then could you
> please help make the test degrade gracefully in its absence?

 In some cases, it is not easy to degrade and/or check for a feature.
 Probably several security features could fall in this bucket.


 I don't really want to have that change be a backport -- it's quite
 invasive across multiple architectures.

 Agreed. The same test for kernel applies to tests as well. If a kernel
 feature can't be back-ported, the test for that feature will fall in the
 same bucket. It shouldn't be back-ported.


 I would say just add a kernel version check to the test. This is
 probably not the only selftest that will need such things. :)
>>>
>>> Adding release checks to selftests is going to problematic for 
>>> maintenance.
>>> Tests should fail gracefully if feature isn't supported in older 
>>> kernels.
>>>
>>> Several tests do that now and please find a way to check for 
>>> dependencies
>>> and feature availability and fail the test gracefully. If there is a 
>>> test
>>> that can't do that for some reason, we can discuss it, but as a general
>>> rule, I don't want to see kselftest patches that check release.
>>
>> If a future kernel inadvertently loses the new feature and degrades to
>> the behavior of old kernels, that would be a serious bug and should be
>> caught.

 Agreed. If I understand you correctly, by not testing stable kernels
 with their own selftests, some serious bugs could go undetected.
>>>
>>> Personally I'm a bit skeptical. I think the reasoning is more that the
>>> latest selftests provide more coverage, and therefore should be better
>>> tests, even on older kernels.
>>
>> The assumption that "the latest selftests provide more coverage, and
>> therefore should be better tests, even on older kernels." is incorrect.
>>
>> Selftests in general track the kernel features. In some cases, new
>> tests could be added that provide better coverage on older kernels,
>> however, it is more likely that new tests are added to test new kernel
>> features and enhancements to existing features. Based on the second
>> "enhancements to existing features" it is more important to test newer
>> kernels with older selftests. This does happen in kernel integration
>> cycles during development.
>>
>> As a general rule, testing stable kernels with their own selftests will
>> yield the best results.
>>
> I would have agreed totally, if the selftests and the kernel were in
> sync since forever. But since the kselftests are a comparatively
> recent addition, the number of tests available for features existing
> in LTS kernels is really quite small. Just as a comparison, 4.4-LTS
> misses tests for bpf, cpufreq, gpio, media_tests, networking, prctl,
> to name a few.

Yes. Several tests have been added since 4.4. Unfortunately these new
additions don't qualify as candidates for stable releases.

> 
> Also, while trying to run kselftests from later kernels with 4.4, we
> only had a few failures for existing features, while most other tests
> ran ok. Just another data point.

That is good to hear. Let's fix the problems as you find them and make
the selftests from newer releases work well (if not perfect) on older
kernels.

4.13 is going to see TAP13 changes as well, so if selftests from newer
kernels work for you, you will see TAB13 benefits.

thanks,
-- Shuah

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 10:02 PM, Sumit Semwal wrote:
> Hi Shuah,
> 
> On 23 June 2017 at 01:53, Shuah Khan  wrote:
>> Hi Tom,
>>
>> On 06/22/2017 01:48 PM, Tom Gall wrote:
>>> Hi
>>>
>>> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
 On 06/22/2017 11:50 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>>> On 06/22/2017 10:53 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal 
  wrote:
> Hi Kees, Andy,
>
> On 15 June 2017 at 23:26, Sumit Semwal  
> wrote:
>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> feature and test together.
>> - This one also seems like a security hole being closed, and the
>> 'feature' could be a candidate for stable backports, but Arnd tried
>> that, and it was quite non-trivial. So perhaps  we'll need some help
>> from the subsystem developers here.
>
> Could you please help us sort this out? Our goal is to help Greg with
> testing stable kernels, and currently the seccomp tests fail due to
> missing feature (seccomp ptrace hole closure) getting tested via
> latest kselftest.
>
> If you feel the feature isn't a stable candidate, then could you
> please help make the test degrade gracefully in its absence?

 In some cases, it is not easy to degrade and/or check for a feature.
 Probably several security features could fall in this bucket.


 I don't really want to have that change be a backport -- it's quite
 invasive across multiple architectures.

 Agreed. The same test for kernel applies to tests as well. If a kernel
 feature can't be back-ported, the test for that feature will fall in the
 same bucket. It shouldn't be back-ported.


 I would say just add a kernel version check to the test. This is
 probably not the only selftest that will need such things. :)
>>>
>>> Adding release checks to selftests is going to problematic for 
>>> maintenance.
>>> Tests should fail gracefully if feature isn't supported in older 
>>> kernels.
>>>
>>> Several tests do that now and please find a way to check for 
>>> dependencies
>>> and feature availability and fail the test gracefully. If there is a 
>>> test
>>> that can't do that for some reason, we can discuss it, but as a general
>>> rule, I don't want to see kselftest patches that check release.
>>
>> If a future kernel inadvertently loses the new feature and degrades to
>> the behavior of old kernels, that would be a serious bug and should be
>> caught.

 Agreed. If I understand you correctly, by not testing stable kernels
 with their own selftests, some serious bugs could go undetected.
>>>
>>> Personally I'm a bit skeptical. I think the reasoning is more that the
>>> latest selftests provide more coverage, and therefore should be better
>>> tests, even on older kernels.
>>
>> The assumption that "the latest selftests provide more coverage, and
>> therefore should be better tests, even on older kernels." is incorrect.
>>
>> Selftests in general track the kernel features. In some cases, new
>> tests could be added that provide better coverage on older kernels,
>> however, it is more likely that new tests are added to test new kernel
>> features and enhancements to existing features. Based on the second
>> "enhancements to existing features" it is more important to test newer
>> kernels with older selftests. This does happen in kernel integration
>> cycles during development.
>>
>> As a general rule, testing stable kernels with their own selftests will
>> yield the best results.
>>
> I would have agreed totally, if the selftests and the kernel were in
> sync since forever. But since the kselftests are a comparatively
> recent addition, the number of tests available for features existing
> in LTS kernels is really quite small. Just as a comparison, 4.4-LTS
> misses tests for bpf, cpufreq, gpio, media_tests, networking, prctl,
> to name a few.

Yes. Several tests have been added since 4.4. Unfortunately these new
additions don't qualify as candidates for stable releases.

> 
> Also, while trying to run kselftests from later kernels with 4.4, we
> only had a few failures for existing features, while most other tests
> ran ok. Just another data point.

That is good to hear. Let's fix the problems as you find them and make
the selftests from newer releases work well (if not perfect) on older
kernels.

4.13 is going to see TAP13 changes as well, so if selftests from newer
kernels work for you, you will see TAB13 benefits.

thanks,
-- Shuah

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Kees Cook]

On Thu, Jun 22, 2017 at 7:40 PM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman
>  wrote:
>> On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>>> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>>> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> >>> wrote:
>>>  Hi Kees, Andy,
>>> 
>>>  On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>>> > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>>> > feature and test together.
>>> > - This one also seems like a security hole being closed, and the
>>> > 'feature' could be a candidate for stable backports, but Arnd tried
>>> > that, and it was quite non-trivial. So perhaps  we'll need some help
>>> > from the subsystem developers here.
>>> 
>>>  Could you please help us sort this out? Our goal is to help Greg with
>>>  testing stable kernels, and currently the seccomp tests fail due to
>>>  missing feature (seccomp ptrace hole closure) getting tested via
>>>  latest kselftest.
>>> 
>>>  If you feel the feature isn't a stable candidate, then could you
>>>  please help make the test degrade gracefully in its absence?
>>> >>>
>>> >>> I don't really want to have that change be a backport -- it's quite
>>> >>> invasive across multiple architectures.
>>> >>>
>>> >>> I would say just add a kernel version check to the test. This is
>>> >>> probably not the only selftest that will need such things. :)
>>> >>
>>> >> Adding release checks to selftests is going to problematic for 
>>> >> maintenance.
>>> >> Tests should fail gracefully if feature isn't supported in older kernels.
>>> >>
>>> >> Several tests do that now and please find a way to check for dependencies
>>> >> and feature availability and fail the test gracefully. If there is a test
>>> >> that can't do that for some reason, we can discuss it, but as a general
>>> >> rule, I don't want to see kselftest patches that check release.
>>> >
>>> > If a future kernel inadvertently loses the new feature and degrades to
>>> > the behavior of old kernels, that would be a serious bug and should be
>>> > caught.
>>>
>>> Right. I really think stable kernels should be tested with their own
>>> selftests. If some test is needed in a stable kernel it should be
>>> backported to that stable kernel.
>>
>> Well, ideally all new features added to the kernel should be able to be
>> detected by userspace somehow if they are present or not.
>>
>> How do you expect a program to know if a feature has "failed" or is just
>> "not enabled/present in this kernel"?  Normally with syscalls this is
>> easy, same for sysfs changes.  Is seccomp in the bad state where there
>> is no way to detect the two different states here?  How is userspace
>> supposed to deal with that?
>>
>> We make fun of glibc having a zillion crazy tests to determine kernel
>> features, and recently, just not wrapping new syscalls at all because
>> they are just frustrated at the compatibility issues over time.  Let's
>> not make their life any harder than it has to be please.
>>
>> I don't see how any of the kselftest programs are any different than any
>> other userspace program that wants to use our kernel api, and as such,
>> any version of kselftest should be able to successfully run on any
>> kernel release.  If not, then we messed up in how we either wrote the
>> test, or how we added a new kernel api.  Neither is acceptable.
>
> That's a fair point.  We could add SECCOMP_GET_FEATURES that returns a
> mask of otherwise-difficult-to-probe features.

But that's silly. The self tests includes all kinds of bug fix tests,
and adding each of those to some features list seems like crazy
overkill. And every API in the kernel is going to do this?

> Greg, for context, the issue here is that we made what was arguably a
> design error in seccomp's interaction with ptrace.  After determining
> that fixing it solved a bunch of problems and didn't break any user
> programs, we fixed it.  There might be new code that relies on the fix
> being present in the sense that it would be insecure without the fix.
>
> The problem is that the fix is moderately intrusive and doesn't seem
> like a great candidate for backporting, although we could plausibly do
> it.

Even if we did all this for seccomp, we're left with the general case.
This is not a situation unique to seccomp. Behaviors and bug fix tests
are added to selftests over time, and not all of those things are
backport-worthy. If a certain test is desired for a stable kernel, we
should backport the test.

-Kees

-- 
Kees Cook
Pixel Security

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Kees Cook]

On Thu, Jun 22, 2017 at 7:40 PM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman
>  wrote:
>> On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>>> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>>> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> >>> wrote:
>>>  Hi Kees, Andy,
>>> 
>>>  On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>>> > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>>> > feature and test together.
>>> > - This one also seems like a security hole being closed, and the
>>> > 'feature' could be a candidate for stable backports, but Arnd tried
>>> > that, and it was quite non-trivial. So perhaps  we'll need some help
>>> > from the subsystem developers here.
>>> 
>>>  Could you please help us sort this out? Our goal is to help Greg with
>>>  testing stable kernels, and currently the seccomp tests fail due to
>>>  missing feature (seccomp ptrace hole closure) getting tested via
>>>  latest kselftest.
>>> 
>>>  If you feel the feature isn't a stable candidate, then could you
>>>  please help make the test degrade gracefully in its absence?
>>> >>>
>>> >>> I don't really want to have that change be a backport -- it's quite
>>> >>> invasive across multiple architectures.
>>> >>>
>>> >>> I would say just add a kernel version check to the test. This is
>>> >>> probably not the only selftest that will need such things. :)
>>> >>
>>> >> Adding release checks to selftests is going to problematic for 
>>> >> maintenance.
>>> >> Tests should fail gracefully if feature isn't supported in older kernels.
>>> >>
>>> >> Several tests do that now and please find a way to check for dependencies
>>> >> and feature availability and fail the test gracefully. If there is a test
>>> >> that can't do that for some reason, we can discuss it, but as a general
>>> >> rule, I don't want to see kselftest patches that check release.
>>> >
>>> > If a future kernel inadvertently loses the new feature and degrades to
>>> > the behavior of old kernels, that would be a serious bug and should be
>>> > caught.
>>>
>>> Right. I really think stable kernels should be tested with their own
>>> selftests. If some test is needed in a stable kernel it should be
>>> backported to that stable kernel.
>>
>> Well, ideally all new features added to the kernel should be able to be
>> detected by userspace somehow if they are present or not.
>>
>> How do you expect a program to know if a feature has "failed" or is just
>> "not enabled/present in this kernel"?  Normally with syscalls this is
>> easy, same for sysfs changes.  Is seccomp in the bad state where there
>> is no way to detect the two different states here?  How is userspace
>> supposed to deal with that?
>>
>> We make fun of glibc having a zillion crazy tests to determine kernel
>> features, and recently, just not wrapping new syscalls at all because
>> they are just frustrated at the compatibility issues over time.  Let's
>> not make their life any harder than it has to be please.
>>
>> I don't see how any of the kselftest programs are any different than any
>> other userspace program that wants to use our kernel api, and as such,
>> any version of kselftest should be able to successfully run on any
>> kernel release.  If not, then we messed up in how we either wrote the
>> test, or how we added a new kernel api.  Neither is acceptable.
>
> That's a fair point.  We could add SECCOMP_GET_FEATURES that returns a
> mask of otherwise-difficult-to-probe features.

But that's silly. The self tests includes all kinds of bug fix tests,
and adding each of those to some features list seems like crazy
overkill. And every API in the kernel is going to do this?

> Greg, for context, the issue here is that we made what was arguably a
> design error in seccomp's interaction with ptrace.  After determining
> that fixing it solved a bunch of problems and didn't break any user
> programs, we fixed it.  There might be new code that relies on the fix
> being present in the sense that it would be insecure without the fix.
>
> The problem is that the fix is moderately intrusive and doesn't seem
> like a great candidate for backporting, although we could plausibly do
> it.

Even if we did all this for seccomp, we're left with the general case.
This is not a situation unique to seccomp. Behaviors and bug fix tests
are added to selftests over time, and not all of those things are
backport-worthy. If a certain test is desired for a stable kernel, we
should backport the test.

-Kees

-- 
Kees Cook
Pixel Security

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Sumit Semwal]

Hi Shuah,

On 23 June 2017 at 01:53, Shuah Khan  wrote:
> Hi Tom,
>
> On 06/22/2017 01:48 PM, Tom Gall wrote:
>> Hi
>>
>> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>>> On 06/22/2017 11:50 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> wrote:
 Hi Kees, Andy,

 On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

 Could you please help us sort this out? Our goal is to help Greg with
 testing stable kernels, and currently the seccomp tests fail due to
 missing feature (seccomp ptrace hole closure) getting tested via
 latest kselftest.

 If you feel the feature isn't a stable candidate, then could you
 please help make the test degrade gracefully in its absence?
>>>
>>> In some cases, it is not easy to degrade and/or check for a feature.
>>> Probably several security features could fall in this bucket.
>>>
>>>
>>> I don't really want to have that change be a backport -- it's quite
>>> invasive across multiple architectures.
>>>
>>> Agreed. The same test for kernel applies to tests as well. If a kernel
>>> feature can't be back-ported, the test for that feature will fall in the
>>> same bucket. It shouldn't be back-ported.
>>>
>>>
>>> I would say just add a kernel version check to the test. This is
>>> probably not the only selftest that will need such things. :)
>>
>> Adding release checks to selftests is going to problematic for 
>> maintenance.
>> Tests should fail gracefully if feature isn't supported in older kernels.
>>
>> Several tests do that now and please find a way to check for dependencies
>> and feature availability and fail the test gracefully. If there is a test
>> that can't do that for some reason, we can discuss it, but as a general
>> rule, I don't want to see kselftest patches that check release.
>
> If a future kernel inadvertently loses the new feature and degrades to
> the behavior of old kernels, that would be a serious bug and should be
> caught.
>>>
>>> Agreed. If I understand you correctly, by not testing stable kernels
>>> with their own selftests, some serious bugs could go undetected.
>>
>> Personally I'm a bit skeptical. I think the reasoning is more that the
>> latest selftests provide more coverage, and therefore should be better
>> tests, even on older kernels.
>
> The assumption that "the latest selftests provide more coverage, and
> therefore should be better tests, even on older kernels." is incorrect.
>
> Selftests in general track the kernel features. In some cases, new
> tests could be added that provide better coverage on older kernels,
> however, it is more likely that new tests are added to test new kernel
> features and enhancements to existing features. Based on the second
> "enhancements to existing features" it is more important to test newer
> kernels with older selftests. This does happen in kernel integration
> cycles during development.
>
> As a general rule, testing stable kernels with their own selftests will
> yield the best results.
>
I would have agreed totally, if the selftests and the kernel were in
sync since forever. But since the kselftests are a comparatively
recent addition, the number of tests available for features existing
in LTS kernels is really quite small. Just as a comparison, 4.4-LTS
misses tests for bpf, cpufreq, gpio, media_tests, networking, prctl,
to name a few.

Also, while trying to run kselftests from later kernels with 4.4, we
only had a few failures for existing features, while most other tests
ran ok. Just another data point.

>>

 Right. I really think stable kernels should be tested with their own
 selftests. If some test is needed in a stable kernel it should be
 backported to that stable kernel.
>>>
>>> Correct. This is always a safe option. There might be cases that even
>>> prevent tests being built, especially if a new feature adds new fields
>>> to an existing structure.
>>>
>>> It appears in some cases, users want to run newer tests on older kernels.
>>> Some tests can clearly detect feature support using module presence and/or
>>> Kconfig enabled or disabled. These are conditions even on a kernel that
>>> supports a new module or new 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Sumit Semwal]

Hi Shuah,

On 23 June 2017 at 01:53, Shuah Khan  wrote:
> Hi Tom,
>
> On 06/22/2017 01:48 PM, Tom Gall wrote:
>> Hi
>>
>> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>>> On 06/22/2017 11:50 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> wrote:
 Hi Kees, Andy,

 On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

 Could you please help us sort this out? Our goal is to help Greg with
 testing stable kernels, and currently the seccomp tests fail due to
 missing feature (seccomp ptrace hole closure) getting tested via
 latest kselftest.

 If you feel the feature isn't a stable candidate, then could you
 please help make the test degrade gracefully in its absence?
>>>
>>> In some cases, it is not easy to degrade and/or check for a feature.
>>> Probably several security features could fall in this bucket.
>>>
>>>
>>> I don't really want to have that change be a backport -- it's quite
>>> invasive across multiple architectures.
>>>
>>> Agreed. The same test for kernel applies to tests as well. If a kernel
>>> feature can't be back-ported, the test for that feature will fall in the
>>> same bucket. It shouldn't be back-ported.
>>>
>>>
>>> I would say just add a kernel version check to the test. This is
>>> probably not the only selftest that will need such things. :)
>>
>> Adding release checks to selftests is going to problematic for 
>> maintenance.
>> Tests should fail gracefully if feature isn't supported in older kernels.
>>
>> Several tests do that now and please find a way to check for dependencies
>> and feature availability and fail the test gracefully. If there is a test
>> that can't do that for some reason, we can discuss it, but as a general
>> rule, I don't want to see kselftest patches that check release.
>
> If a future kernel inadvertently loses the new feature and degrades to
> the behavior of old kernels, that would be a serious bug and should be
> caught.
>>>
>>> Agreed. If I understand you correctly, by not testing stable kernels
>>> with their own selftests, some serious bugs could go undetected.
>>
>> Personally I'm a bit skeptical. I think the reasoning is more that the
>> latest selftests provide more coverage, and therefore should be better
>> tests, even on older kernels.
>
> The assumption that "the latest selftests provide more coverage, and
> therefore should be better tests, even on older kernels." is incorrect.
>
> Selftests in general track the kernel features. In some cases, new
> tests could be added that provide better coverage on older kernels,
> however, it is more likely that new tests are added to test new kernel
> features and enhancements to existing features. Based on the second
> "enhancements to existing features" it is more important to test newer
> kernels with older selftests. This does happen in kernel integration
> cycles during development.
>
> As a general rule, testing stable kernels with their own selftests will
> yield the best results.
>
I would have agreed totally, if the selftests and the kernel were in
sync since forever. But since the kselftests are a comparatively
recent addition, the number of tests available for features existing
in LTS kernels is really quite small. Just as a comparison, 4.4-LTS
misses tests for bpf, cpufreq, gpio, media_tests, networking, prctl,
to name a few.

Also, while trying to run kselftests from later kernels with 4.4, we
only had a few failures for existing features, while most other tests
ran ok. Just another data point.

>>

 Right. I really think stable kernels should be tested with their own
 selftests. If some test is needed in a stable kernel it should be
 backported to that stable kernel.
>>>
>>> Correct. This is always a safe option. There might be cases that even
>>> prevent tests being built, especially if a new feature adds new fields
>>> to an existing structure.
>>>
>>> It appears in some cases, users want to run newer tests on older kernels.
>>> Some tests can clearly detect feature support using module presence and/or
>>> Kconfig enabled or disabled. These are conditions even on a kernel that
>>> supports a new module or new config option. The kernel the test is running
>>> on might not have the feature enabled or module might not be present. 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Andy Lutomirski]

On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman
 wrote:
> On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> >>> wrote:
>>  Hi Kees, Andy,
>> 
>>  On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>> > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> > feature and test together.
>> > - This one also seems like a security hole being closed, and the
>> > 'feature' could be a candidate for stable backports, but Arnd tried
>> > that, and it was quite non-trivial. So perhaps  we'll need some help
>> > from the subsystem developers here.
>> 
>>  Could you please help us sort this out? Our goal is to help Greg with
>>  testing stable kernels, and currently the seccomp tests fail due to
>>  missing feature (seccomp ptrace hole closure) getting tested via
>>  latest kselftest.
>> 
>>  If you feel the feature isn't a stable candidate, then could you
>>  please help make the test degrade gracefully in its absence?
>> >>>
>> >>> I don't really want to have that change be a backport -- it's quite
>> >>> invasive across multiple architectures.
>> >>>
>> >>> I would say just add a kernel version check to the test. This is
>> >>> probably not the only selftest that will need such things. :)
>> >>
>> >> Adding release checks to selftests is going to problematic for 
>> >> maintenance.
>> >> Tests should fail gracefully if feature isn't supported in older kernels.
>> >>
>> >> Several tests do that now and please find a way to check for dependencies
>> >> and feature availability and fail the test gracefully. If there is a test
>> >> that can't do that for some reason, we can discuss it, but as a general
>> >> rule, I don't want to see kselftest patches that check release.
>> >
>> > If a future kernel inadvertently loses the new feature and degrades to
>> > the behavior of old kernels, that would be a serious bug and should be
>> > caught.
>>
>> Right. I really think stable kernels should be tested with their own
>> selftests. If some test is needed in a stable kernel it should be
>> backported to that stable kernel.
>
> Well, ideally all new features added to the kernel should be able to be
> detected by userspace somehow if they are present or not.
>
> How do you expect a program to know if a feature has "failed" or is just
> "not enabled/present in this kernel"?  Normally with syscalls this is
> easy, same for sysfs changes.  Is seccomp in the bad state where there
> is no way to detect the two different states here?  How is userspace
> supposed to deal with that?
>
> We make fun of glibc having a zillion crazy tests to determine kernel
> features, and recently, just not wrapping new syscalls at all because
> they are just frustrated at the compatibility issues over time.  Let's
> not make their life any harder than it has to be please.
>
> I don't see how any of the kselftest programs are any different than any
> other userspace program that wants to use our kernel api, and as such,
> any version of kselftest should be able to successfully run on any
> kernel release.  If not, then we messed up in how we either wrote the
> test, or how we added a new kernel api.  Neither is acceptable.

That's a fair point.  We could add SECCOMP_GET_FEATURES that returns a
mask of otherwise-difficult-to-probe features.

Greg, for context, the issue here is that we made what was arguably a
design error in seccomp's interaction with ptrace.  After determining
that fixing it solved a bunch of problems and didn't break any user
programs, we fixed it.  There might be new code that relies on the fix
being present in the sense that it would be insecure without the fix.

The problem is that the fix is moderately intrusive and doesn't seem
like a great candidate for backporting, although we could plausibly do
it.

--Andy

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Andy Lutomirski]

On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman
 wrote:
> On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> >>> wrote:
>>  Hi Kees, Andy,
>> 
>>  On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>> > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> > feature and test together.
>> > - This one also seems like a security hole being closed, and the
>> > 'feature' could be a candidate for stable backports, but Arnd tried
>> > that, and it was quite non-trivial. So perhaps  we'll need some help
>> > from the subsystem developers here.
>> 
>>  Could you please help us sort this out? Our goal is to help Greg with
>>  testing stable kernels, and currently the seccomp tests fail due to
>>  missing feature (seccomp ptrace hole closure) getting tested via
>>  latest kselftest.
>> 
>>  If you feel the feature isn't a stable candidate, then could you
>>  please help make the test degrade gracefully in its absence?
>> >>>
>> >>> I don't really want to have that change be a backport -- it's quite
>> >>> invasive across multiple architectures.
>> >>>
>> >>> I would say just add a kernel version check to the test. This is
>> >>> probably not the only selftest that will need such things. :)
>> >>
>> >> Adding release checks to selftests is going to problematic for 
>> >> maintenance.
>> >> Tests should fail gracefully if feature isn't supported in older kernels.
>> >>
>> >> Several tests do that now and please find a way to check for dependencies
>> >> and feature availability and fail the test gracefully. If there is a test
>> >> that can't do that for some reason, we can discuss it, but as a general
>> >> rule, I don't want to see kselftest patches that check release.
>> >
>> > If a future kernel inadvertently loses the new feature and degrades to
>> > the behavior of old kernels, that would be a serious bug and should be
>> > caught.
>>
>> Right. I really think stable kernels should be tested with their own
>> selftests. If some test is needed in a stable kernel it should be
>> backported to that stable kernel.
>
> Well, ideally all new features added to the kernel should be able to be
> detected by userspace somehow if they are present or not.
>
> How do you expect a program to know if a feature has "failed" or is just
> "not enabled/present in this kernel"?  Normally with syscalls this is
> easy, same for sysfs changes.  Is seccomp in the bad state where there
> is no way to detect the two different states here?  How is userspace
> supposed to deal with that?
>
> We make fun of glibc having a zillion crazy tests to determine kernel
> features, and recently, just not wrapping new syscalls at all because
> they are just frustrated at the compatibility issues over time.  Let's
> not make their life any harder than it has to be please.
>
> I don't see how any of the kselftest programs are any different than any
> other userspace program that wants to use our kernel api, and as such,
> any version of kselftest should be able to successfully run on any
> kernel release.  If not, then we messed up in how we either wrote the
> test, or how we added a new kernel api.  Neither is acceptable.

That's a fair point.  We could add SECCOMP_GET_FEATURES that returns a
mask of otherwise-difficult-to-probe features.

Greg, for context, the issue here is that we made what was arguably a
design error in seccomp's interaction with ptrace.  After determining
that fixing it solved a bunch of problems and didn't break any user
programs, we fixed it.  There might be new code that relies on the fix
being present in the sense that it would be insecure without the fix.

The problem is that the fix is moderately intrusive and doesn't seem
like a great candidate for backporting, although we could plausibly do
it.

--Andy

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Greg Kroah-Hartman]

On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
> >>> wrote:
>  Hi Kees, Andy,
> 
>  On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> > feature and test together.
> > - This one also seems like a security hole being closed, and the
> > 'feature' could be a candidate for stable backports, but Arnd tried
> > that, and it was quite non-trivial. So perhaps  we'll need some help
> > from the subsystem developers here.
> 
>  Could you please help us sort this out? Our goal is to help Greg with
>  testing stable kernels, and currently the seccomp tests fail due to
>  missing feature (seccomp ptrace hole closure) getting tested via
>  latest kselftest.
> 
>  If you feel the feature isn't a stable candidate, then could you
>  please help make the test degrade gracefully in its absence?
> >>>
> >>> I don't really want to have that change be a backport -- it's quite
> >>> invasive across multiple architectures.
> >>>
> >>> I would say just add a kernel version check to the test. This is
> >>> probably not the only selftest that will need such things. :)
> >>
> >> Adding release checks to selftests is going to problematic for maintenance.
> >> Tests should fail gracefully if feature isn't supported in older kernels.
> >>
> >> Several tests do that now and please find a way to check for dependencies
> >> and feature availability and fail the test gracefully. If there is a test
> >> that can't do that for some reason, we can discuss it, but as a general
> >> rule, I don't want to see kselftest patches that check release.
> >
> > If a future kernel inadvertently loses the new feature and degrades to
> > the behavior of old kernels, that would be a serious bug and should be
> > caught.
> 
> Right. I really think stable kernels should be tested with their own
> selftests. If some test is needed in a stable kernel it should be
> backported to that stable kernel.

Well, ideally all new features added to the kernel should be able to be
detected by userspace somehow if they are present or not.

How do you expect a program to know if a feature has "failed" or is just
"not enabled/present in this kernel"?  Normally with syscalls this is
easy, same for sysfs changes.  Is seccomp in the bad state where there
is no way to detect the two different states here?  How is userspace
supposed to deal with that?

We make fun of glibc having a zillion crazy tests to determine kernel
features, and recently, just not wrapping new syscalls at all because
they are just frustrated at the compatibility issues over time.  Let's
not make their life any harder than it has to be please.

I don't see how any of the kselftest programs are any different than any
other userspace program that wants to use our kernel api, and as such,
any version of kselftest should be able to successfully run on any
kernel release.  If not, then we messed up in how we either wrote the
test, or how we added a new kernel api.  Neither is acceptable.

thanks,

greg k-h

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Greg Kroah-Hartman]

On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> >> On 06/22/2017 10:53 AM, Kees Cook wrote:
> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
> >>> wrote:
>  Hi Kees, Andy,
> 
>  On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> > feature and test together.
> > - This one also seems like a security hole being closed, and the
> > 'feature' could be a candidate for stable backports, but Arnd tried
> > that, and it was quite non-trivial. So perhaps  we'll need some help
> > from the subsystem developers here.
> 
>  Could you please help us sort this out? Our goal is to help Greg with
>  testing stable kernels, and currently the seccomp tests fail due to
>  missing feature (seccomp ptrace hole closure) getting tested via
>  latest kselftest.
> 
>  If you feel the feature isn't a stable candidate, then could you
>  please help make the test degrade gracefully in its absence?
> >>>
> >>> I don't really want to have that change be a backport -- it's quite
> >>> invasive across multiple architectures.
> >>>
> >>> I would say just add a kernel version check to the test. This is
> >>> probably not the only selftest that will need such things. :)
> >>
> >> Adding release checks to selftests is going to problematic for maintenance.
> >> Tests should fail gracefully if feature isn't supported in older kernels.
> >>
> >> Several tests do that now and please find a way to check for dependencies
> >> and feature availability and fail the test gracefully. If there is a test
> >> that can't do that for some reason, we can discuss it, but as a general
> >> rule, I don't want to see kselftest patches that check release.
> >
> > If a future kernel inadvertently loses the new feature and degrades to
> > the behavior of old kernels, that would be a serious bug and should be
> > caught.
> 
> Right. I really think stable kernels should be tested with their own
> selftests. If some test is needed in a stable kernel it should be
> backported to that stable kernel.

Well, ideally all new features added to the kernel should be able to be
detected by userspace somehow if they are present or not.

How do you expect a program to know if a feature has "failed" or is just
"not enabled/present in this kernel"?  Normally with syscalls this is
easy, same for sysfs changes.  Is seccomp in the bad state where there
is no way to detect the two different states here?  How is userspace
supposed to deal with that?

We make fun of glibc having a zillion crazy tests to determine kernel
features, and recently, just not wrapping new syscalls at all because
they are just frustrated at the compatibility issues over time.  Let's
not make their life any harder than it has to be please.

I don't see how any of the kselftest programs are any different than any
other userspace program that wants to use our kernel api, and as such,
any version of kselftest should be able to successfully run on any
kernel release.  If not, then we messed up in how we either wrote the
test, or how we added a new kernel api.  Neither is acceptable.

thanks,

greg k-h

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

Hi Tom,

On 06/22/2017 01:48 PM, Tom Gall wrote:
> Hi
> 
> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>> On 06/22/2017 11:50 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
 On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> wrote:
>>> Hi Kees, Andy,
>>>
>>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
 feature and test together.
 - This one also seems like a security hole being closed, and the
 'feature' could be a candidate for stable backports, but Arnd tried
 that, and it was quite non-trivial. So perhaps  we'll need some help
 from the subsystem developers here.
>>>
>>> Could you please help us sort this out? Our goal is to help Greg with
>>> testing stable kernels, and currently the seccomp tests fail due to
>>> missing feature (seccomp ptrace hole closure) getting tested via
>>> latest kselftest.
>>>
>>> If you feel the feature isn't a stable candidate, then could you
>>> please help make the test degrade gracefully in its absence?
>>
>> In some cases, it is not easy to degrade and/or check for a feature.
>> Probably several security features could fall in this bucket.
>>
>>
>> I don't really want to have that change be a backport -- it's quite
>> invasive across multiple architectures.
>>
>> Agreed. The same test for kernel applies to tests as well. If a kernel
>> feature can't be back-ported, the test for that feature will fall in the
>> same bucket. It shouldn't be back-ported.
>>
>>
>> I would say just add a kernel version check to the test. This is
>> probably not the only selftest that will need such things. :)
>
> Adding release checks to selftests is going to problematic for 
> maintenance.
> Tests should fail gracefully if feature isn't supported in older kernels.
>
> Several tests do that now and please find a way to check for dependencies
> and feature availability and fail the test gracefully. If there is a test
> that can't do that for some reason, we can discuss it, but as a general
> rule, I don't want to see kselftest patches that check release.

 If a future kernel inadvertently loses the new feature and degrades to
 the behavior of old kernels, that would be a serious bug and should be
 caught.
>>
>> Agreed. If I understand you correctly, by not testing stable kernels
>> with their own selftests, some serious bugs could go undetected.
> 
> Personally I'm a bit skeptical. I think the reasoning is more that the
> latest selftests provide more coverage, and therefore should be better
> tests, even on older kernels.

The assumption that "the latest selftests provide more coverage, and
therefore should be better tests, even on older kernels." is incorrect.

Selftests in general track the kernel features. In some cases, new
tests could be added that provide better coverage on older kernels,
however, it is more likely that new tests are added to test new kernel
features and enhancements to existing features. Based on the second
"enhancements to existing features" it is more important to test newer
kernels with older selftests. This does happen in kernel integration
cycles during development.

As a general rule, testing stable kernels with their own selftests will
yield the best results.

> 
>>>
>>> Right. I really think stable kernels should be tested with their own
>>> selftests. If some test is needed in a stable kernel it should be
>>> backported to that stable kernel.
>>
>> Correct. This is always a safe option. There might be cases that even
>> prevent tests being built, especially if a new feature adds new fields
>> to an existing structure.
>>
>> It appears in some cases, users want to run newer tests on older kernels.
>> Some tests can clearly detect feature support using module presence and/or
>> Kconfig enabled or disabled. These are conditions even on a kernel that
>> supports a new module or new config option. The kernel the test is running
>> on might not have the feature enabled or module might not be present. In
>> these cases, it would be easier to detect and skip the test.
>>
>> However, some features aren't so easy. For example:
>>
>> - a new flag is added to a syscall, and new test is added. It might not
>>   be easy to detect that.
>> - We might have some tests that can't detect and skip.
>>
>> Based on this discussion, it is probably accurate to say:
>>
>> 1. It is recommended that selftests from the same release be run on the
>>kernel.
>> 2. Selftests from newer kernels will run on older kernels, user should
>>understand the risks such as some tests might fail and might 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

Hi Tom,

On 06/22/2017 01:48 PM, Tom Gall wrote:
> Hi
> 
> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
>> On 06/22/2017 11:50 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
 On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> wrote:
>>> Hi Kees, Andy,
>>>
>>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
 feature and test together.
 - This one also seems like a security hole being closed, and the
 'feature' could be a candidate for stable backports, but Arnd tried
 that, and it was quite non-trivial. So perhaps  we'll need some help
 from the subsystem developers here.
>>>
>>> Could you please help us sort this out? Our goal is to help Greg with
>>> testing stable kernels, and currently the seccomp tests fail due to
>>> missing feature (seccomp ptrace hole closure) getting tested via
>>> latest kselftest.
>>>
>>> If you feel the feature isn't a stable candidate, then could you
>>> please help make the test degrade gracefully in its absence?
>>
>> In some cases, it is not easy to degrade and/or check for a feature.
>> Probably several security features could fall in this bucket.
>>
>>
>> I don't really want to have that change be a backport -- it's quite
>> invasive across multiple architectures.
>>
>> Agreed. The same test for kernel applies to tests as well. If a kernel
>> feature can't be back-ported, the test for that feature will fall in the
>> same bucket. It shouldn't be back-ported.
>>
>>
>> I would say just add a kernel version check to the test. This is
>> probably not the only selftest that will need such things. :)
>
> Adding release checks to selftests is going to problematic for 
> maintenance.
> Tests should fail gracefully if feature isn't supported in older kernels.
>
> Several tests do that now and please find a way to check for dependencies
> and feature availability and fail the test gracefully. If there is a test
> that can't do that for some reason, we can discuss it, but as a general
> rule, I don't want to see kselftest patches that check release.

 If a future kernel inadvertently loses the new feature and degrades to
 the behavior of old kernels, that would be a serious bug and should be
 caught.
>>
>> Agreed. If I understand you correctly, by not testing stable kernels
>> with their own selftests, some serious bugs could go undetected.
> 
> Personally I'm a bit skeptical. I think the reasoning is more that the
> latest selftests provide more coverage, and therefore should be better
> tests, even on older kernels.

The assumption that "the latest selftests provide more coverage, and
therefore should be better tests, even on older kernels." is incorrect.

Selftests in general track the kernel features. In some cases, new
tests could be added that provide better coverage on older kernels,
however, it is more likely that new tests are added to test new kernel
features and enhancements to existing features. Based on the second
"enhancements to existing features" it is more important to test newer
kernels with older selftests. This does happen in kernel integration
cycles during development.

As a general rule, testing stable kernels with their own selftests will
yield the best results.

> 
>>>
>>> Right. I really think stable kernels should be tested with their own
>>> selftests. If some test is needed in a stable kernel it should be
>>> backported to that stable kernel.
>>
>> Correct. This is always a safe option. There might be cases that even
>> prevent tests being built, especially if a new feature adds new fields
>> to an existing structure.
>>
>> It appears in some cases, users want to run newer tests on older kernels.
>> Some tests can clearly detect feature support using module presence and/or
>> Kconfig enabled or disabled. These are conditions even on a kernel that
>> supports a new module or new config option. The kernel the test is running
>> on might not have the feature enabled or module might not be present. In
>> these cases, it would be easier to detect and skip the test.
>>
>> However, some features aren't so easy. For example:
>>
>> - a new flag is added to a syscall, and new test is added. It might not
>>   be easy to detect that.
>> - We might have some tests that can't detect and skip.
>>
>> Based on this discussion, it is probably accurate to say:
>>
>> 1. It is recommended that selftests from the same release be run on the
>>kernel.
>> 2. Selftests from newer kernels will run on older kernels, user should
>>understand the risks such as some tests might fail and might not
>>detect feature degradation related bugs.
>> 3. Selftests will fail gracefully on older 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Tom Gall]

Hi

On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
> On 06/22/2017 11:50 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
 On 06/22/2017 10:53 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
> wrote:
>> Hi Kees, Andy,
>>
>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>>> feature and test together.
>>> - This one also seems like a security hole being closed, and the
>>> 'feature' could be a candidate for stable backports, but Arnd tried
>>> that, and it was quite non-trivial. So perhaps  we'll need some help
>>> from the subsystem developers here.
>>
>> Could you please help us sort this out? Our goal is to help Greg with
>> testing stable kernels, and currently the seccomp tests fail due to
>> missing feature (seccomp ptrace hole closure) getting tested via
>> latest kselftest.
>>
>> If you feel the feature isn't a stable candidate, then could you
>> please help make the test degrade gracefully in its absence?
>
> In some cases, it is not easy to degrade and/or check for a feature.
> Probably several security features could fall in this bucket.
>
>
> I don't really want to have that change be a backport -- it's quite
> invasive across multiple architectures.
>
> Agreed. The same test for kernel applies to tests as well. If a kernel
> feature can't be back-ported, the test for that feature will fall in the
> same bucket. It shouldn't be back-ported.
>
>
> I would say just add a kernel version check to the test. This is
> probably not the only selftest that will need such things. :)

 Adding release checks to selftests is going to problematic for maintenance.
 Tests should fail gracefully if feature isn't supported in older kernels.

 Several tests do that now and please find a way to check for dependencies
 and feature availability and fail the test gracefully. If there is a test
 that can't do that for some reason, we can discuss it, but as a general
 rule, I don't want to see kselftest patches that check release.
>>>
>>> If a future kernel inadvertently loses the new feature and degrades to
>>> the behavior of old kernels, that would be a serious bug and should be
>>> caught.
>
> Agreed. If I understand you correctly, by not testing stable kernels
> with their own selftests, some serious bugs could go undetected.

Personally I'm a bit skeptical. I think the reasoning is more that the
latest selftests provide more coverage, and therefore should be better
tests, even on older kernels.

>>
>> Right. I really think stable kernels should be tested with their own
>> selftests. If some test is needed in a stable kernel it should be
>> backported to that stable kernel.
>
> Correct. This is always a safe option. There might be cases that even
> prevent tests being built, especially if a new feature adds new fields
> to an existing structure.
>
> It appears in some cases, users want to run newer tests on older kernels.
> Some tests can clearly detect feature support using module presence and/or
> Kconfig enabled or disabled. These are conditions even on a kernel that
> supports a new module or new config option. The kernel the test is running
> on might not have the feature enabled or module might not be present. In
> these cases, it would be easier to detect and skip the test.
>
> However, some features aren't so easy. For example:
>
> - a new flag is added to a syscall, and new test is added. It might not
>   be easy to detect that.
> - We might have some tests that can't detect and skip.
>
> Based on this discussion, it is probably accurate to say:
>
> 1. It is recommended that selftests from the same release be run on the
>kernel.
> 2. Selftests from newer kernels will run on older kernels, user should
>understand the risks such as some tests might fail and might not
>detect feature degradation related bugs.
> 3. Selftests will fail gracefully on older releases if at all possible.

How about gracefully be skipped instead of fail?

The later suggests the test case in some situations can detect it's
pointless to run something and say as much instead of emitting a
failure that would be a waste of time to look into.

As another example take tools/testing/selftests/net/psock_fanout.c
On 4.9 it'll fail to compile (using master's selftests) because
PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for
that symbol and the psock_fanout test will compile and run just fine.

> Sumit!
>
> 1. What are the reasons for testing older kernel with selftests from
>newer kernels? What are the benefits you see for doing so?

I think the presumption is the latest greatest collection 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Tom Gall]

Hi

On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan  wrote:
> On 06/22/2017 11:50 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
 On 06/22/2017 10:53 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
> wrote:
>> Hi Kees, Andy,
>>
>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>>> feature and test together.
>>> - This one also seems like a security hole being closed, and the
>>> 'feature' could be a candidate for stable backports, but Arnd tried
>>> that, and it was quite non-trivial. So perhaps  we'll need some help
>>> from the subsystem developers here.
>>
>> Could you please help us sort this out? Our goal is to help Greg with
>> testing stable kernels, and currently the seccomp tests fail due to
>> missing feature (seccomp ptrace hole closure) getting tested via
>> latest kselftest.
>>
>> If you feel the feature isn't a stable candidate, then could you
>> please help make the test degrade gracefully in its absence?
>
> In some cases, it is not easy to degrade and/or check for a feature.
> Probably several security features could fall in this bucket.
>
>
> I don't really want to have that change be a backport -- it's quite
> invasive across multiple architectures.
>
> Agreed. The same test for kernel applies to tests as well. If a kernel
> feature can't be back-ported, the test for that feature will fall in the
> same bucket. It shouldn't be back-ported.
>
>
> I would say just add a kernel version check to the test. This is
> probably not the only selftest that will need such things. :)

 Adding release checks to selftests is going to problematic for maintenance.
 Tests should fail gracefully if feature isn't supported in older kernels.

 Several tests do that now and please find a way to check for dependencies
 and feature availability and fail the test gracefully. If there is a test
 that can't do that for some reason, we can discuss it, but as a general
 rule, I don't want to see kselftest patches that check release.
>>>
>>> If a future kernel inadvertently loses the new feature and degrades to
>>> the behavior of old kernels, that would be a serious bug and should be
>>> caught.
>
> Agreed. If I understand you correctly, by not testing stable kernels
> with their own selftests, some serious bugs could go undetected.

Personally I'm a bit skeptical. I think the reasoning is more that the
latest selftests provide more coverage, and therefore should be better
tests, even on older kernels.

>>
>> Right. I really think stable kernels should be tested with their own
>> selftests. If some test is needed in a stable kernel it should be
>> backported to that stable kernel.
>
> Correct. This is always a safe option. There might be cases that even
> prevent tests being built, especially if a new feature adds new fields
> to an existing structure.
>
> It appears in some cases, users want to run newer tests on older kernels.
> Some tests can clearly detect feature support using module presence and/or
> Kconfig enabled or disabled. These are conditions even on a kernel that
> supports a new module or new config option. The kernel the test is running
> on might not have the feature enabled or module might not be present. In
> these cases, it would be easier to detect and skip the test.
>
> However, some features aren't so easy. For example:
>
> - a new flag is added to a syscall, and new test is added. It might not
>   be easy to detect that.
> - We might have some tests that can't detect and skip.
>
> Based on this discussion, it is probably accurate to say:
>
> 1. It is recommended that selftests from the same release be run on the
>kernel.
> 2. Selftests from newer kernels will run on older kernels, user should
>understand the risks such as some tests might fail and might not
>detect feature degradation related bugs.
> 3. Selftests will fail gracefully on older releases if at all possible.

How about gracefully be skipped instead of fail?

The later suggests the test case in some situations can detect it's
pointless to run something and say as much instead of emitting a
failure that would be a waste of time to look into.

As another example take tools/testing/selftests/net/psock_fanout.c
On 4.9 it'll fail to compile (using master's selftests) because
PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for
that symbol and the psock_fanout test will compile and run just fine.

> Sumit!
>
> 1. What are the reasons for testing older kernel with selftests from
>newer kernels? What are the benefits you see for doing so?

I think the presumption is the latest greatest collection of selftests
are the best, most complete.

>I am looking to understand the need/reasons for this 

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 11:50 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>>> On 06/22/2017 10:53 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
 wrote:
> Hi Kees, Andy,
>
> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> feature and test together.
>> - This one also seems like a security hole being closed, and the
>> 'feature' could be a candidate for stable backports, but Arnd tried
>> that, and it was quite non-trivial. So perhaps  we'll need some help
>> from the subsystem developers here.
>
> Could you please help us sort this out? Our goal is to help Greg with
> testing stable kernels, and currently the seccomp tests fail due to
> missing feature (seccomp ptrace hole closure) getting tested via
> latest kselftest.
>
> If you feel the feature isn't a stable candidate, then could you
> please help make the test degrade gracefully in its absence?

In some cases, it is not easy to degrade and/or check for a feature.
Probably several security features could fall in this bucket.


 I don't really want to have that change be a backport -- it's quite
 invasive across multiple architectures.

Agreed. The same test for kernel applies to tests as well. If a kernel
feature can't be back-ported, the test for that feature will fall in the
same bucket. It shouldn't be back-ported.


 I would say just add a kernel version check to the test. This is
 probably not the only selftest that will need such things. :)
>>>
>>> Adding release checks to selftests is going to problematic for maintenance.
>>> Tests should fail gracefully if feature isn't supported in older kernels.
>>>
>>> Several tests do that now and please find a way to check for dependencies
>>> and feature availability and fail the test gracefully. If there is a test
>>> that can't do that for some reason, we can discuss it, but as a general
>>> rule, I don't want to see kselftest patches that check release.
>>
>> If a future kernel inadvertently loses the new feature and degrades to
>> the behavior of old kernels, that would be a serious bug and should be
>> caught.

Agreed. If I understand you correctly, by not testing stable kernels
with their own selftests, some serious bugs could go undetected.

> 
> Right. I really think stable kernels should be tested with their own
> selftests. If some test is needed in a stable kernel it should be
> backported to that stable kernel.

Correct. This is always a safe option. There might be cases that even
prevent tests being built, especially if a new feature adds new fields
to an existing structure.

It appears in some cases, users want to run newer tests on older kernels.
Some tests can clearly detect feature support using module presence and/or
Kconfig enabled or disabled. These are conditions even on a kernel that
supports a new module or new config option. The kernel the test is running
on might not have the feature enabled or module might not be present. In
these cases, it would be easier to detect and skip the test.

However, some features aren't so easy. For example:

- a new flag is added to a syscall, and new test is added. It might not
  be easy to detect that.
- We might have some tests that can't detect and skip.

Based on this discussion, it is probably accurate to say:

1. It is recommended that selftests from the same release be run on the
   kernel.
2. Selftests from newer kernels will run on older kernels, user should
   understand the risks such as some tests might fail and might not
   detect feature degradation related bugs.
3. Selftests will fail gracefully on older releases if at all possible.

Sumit!

1. What are the reasons for testing older kernel with selftests from
   newer kernels? What are the benefits you see for doing so?
   
   I am looking to understand the need/reasons for this use-case. In our
   previous discussion on this subject, I did say, you should be able to
   do so with some exceptions.

2. Do you test kernels with the selftests from the same release?

3. Do you find testing with newer selftests to be useful?

thanks,
-- Shuah

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 11:50 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>>> On 06/22/2017 10:53 AM, Kees Cook wrote:
 On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
 wrote:
> Hi Kees, Andy,
>
> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> feature and test together.
>> - This one also seems like a security hole being closed, and the
>> 'feature' could be a candidate for stable backports, but Arnd tried
>> that, and it was quite non-trivial. So perhaps  we'll need some help
>> from the subsystem developers here.
>
> Could you please help us sort this out? Our goal is to help Greg with
> testing stable kernels, and currently the seccomp tests fail due to
> missing feature (seccomp ptrace hole closure) getting tested via
> latest kselftest.
>
> If you feel the feature isn't a stable candidate, then could you
> please help make the test degrade gracefully in its absence?

In some cases, it is not easy to degrade and/or check for a feature.
Probably several security features could fall in this bucket.


 I don't really want to have that change be a backport -- it's quite
 invasive across multiple architectures.

Agreed. The same test for kernel applies to tests as well. If a kernel
feature can't be back-ported, the test for that feature will fall in the
same bucket. It shouldn't be back-ported.


 I would say just add a kernel version check to the test. This is
 probably not the only selftest that will need such things. :)
>>>
>>> Adding release checks to selftests is going to problematic for maintenance.
>>> Tests should fail gracefully if feature isn't supported in older kernels.
>>>
>>> Several tests do that now and please find a way to check for dependencies
>>> and feature availability and fail the test gracefully. If there is a test
>>> that can't do that for some reason, we can discuss it, but as a general
>>> rule, I don't want to see kselftest patches that check release.
>>
>> If a future kernel inadvertently loses the new feature and degrades to
>> the behavior of old kernels, that would be a serious bug and should be
>> caught.

Agreed. If I understand you correctly, by not testing stable kernels
with their own selftests, some serious bugs could go undetected.

> 
> Right. I really think stable kernels should be tested with their own
> selftests. If some test is needed in a stable kernel it should be
> backported to that stable kernel.

Correct. This is always a safe option. There might be cases that even
prevent tests being built, especially if a new feature adds new fields
to an existing structure.

It appears in some cases, users want to run newer tests on older kernels.
Some tests can clearly detect feature support using module presence and/or
Kconfig enabled or disabled. These are conditions even on a kernel that
supports a new module or new config option. The kernel the test is running
on might not have the feature enabled or module might not be present. In
these cases, it would be easier to detect and skip the test.

However, some features aren't so easy. For example:

- a new flag is added to a syscall, and new test is added. It might not
  be easy to detect that.
- We might have some tests that can't detect and skip.

Based on this discussion, it is probably accurate to say:

1. It is recommended that selftests from the same release be run on the
   kernel.
2. Selftests from newer kernels will run on older kernels, user should
   understand the risks such as some tests might fail and might not
   detect feature degradation related bugs.
3. Selftests will fail gracefully on older releases if at all possible.

Sumit!

1. What are the reasons for testing older kernel with selftests from
   newer kernels? What are the benefits you see for doing so?
   
   I am looking to understand the need/reasons for this use-case. In our
   previous discussion on this subject, I did say, you should be able to
   do so with some exceptions.

2. Do you test kernels with the selftests from the same release?

3. Do you find testing with newer selftests to be useful?

thanks,
-- Shuah

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Kees Cook]

On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> wrote:
 Hi Kees, Andy,

 On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

 Could you please help us sort this out? Our goal is to help Greg with
 testing stable kernels, and currently the seccomp tests fail due to
 missing feature (seccomp ptrace hole closure) getting tested via
 latest kselftest.

 If you feel the feature isn't a stable candidate, then could you
 please help make the test degrade gracefully in its absence?
>>>
>>> I don't really want to have that change be a backport -- it's quite
>>> invasive across multiple architectures.
>>>
>>> I would say just add a kernel version check to the test. This is
>>> probably not the only selftest that will need such things. :)
>>
>> Adding release checks to selftests is going to problematic for maintenance.
>> Tests should fail gracefully if feature isn't supported in older kernels.
>>
>> Several tests do that now and please find a way to check for dependencies
>> and feature availability and fail the test gracefully. If there is a test
>> that can't do that for some reason, we can discuss it, but as a general
>> rule, I don't want to see kselftest patches that check release.
>
> If a future kernel inadvertently loses the new feature and degrades to
> the behavior of old kernels, that would be a serious bug and should be
> caught.

Right. I really think stable kernels should be tested with their own
selftests. If some test is needed in a stable kernel it should be
backported to that stable kernel.

-Kees

-- 
Kees Cook
Pixel Security

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Kees Cook]

On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski  wrote:
> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
>> On 06/22/2017 10:53 AM, Kees Cook wrote:
>>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>>> wrote:
 Hi Kees, Andy,

 On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

 Could you please help us sort this out? Our goal is to help Greg with
 testing stable kernels, and currently the seccomp tests fail due to
 missing feature (seccomp ptrace hole closure) getting tested via
 latest kselftest.

 If you feel the feature isn't a stable candidate, then could you
 please help make the test degrade gracefully in its absence?
>>>
>>> I don't really want to have that change be a backport -- it's quite
>>> invasive across multiple architectures.
>>>
>>> I would say just add a kernel version check to the test. This is
>>> probably not the only selftest that will need such things. :)
>>
>> Adding release checks to selftests is going to problematic for maintenance.
>> Tests should fail gracefully if feature isn't supported in older kernels.
>>
>> Several tests do that now and please find a way to check for dependencies
>> and feature availability and fail the test gracefully. If there is a test
>> that can't do that for some reason, we can discuss it, but as a general
>> rule, I don't want to see kselftest patches that check release.
>
> If a future kernel inadvertently loses the new feature and degrades to
> the behavior of old kernels, that would be a serious bug and should be
> caught.

Right. I really think stable kernels should be tested with their own
selftests. If some test is needed in a stable kernel it should be
backported to that stable kernel.

-Kees

-- 
Kees Cook
Pixel Security

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Andy Lutomirski]

On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> wrote:
>>> Hi Kees, Andy,
>>>
>>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
 feature and test together.
 - This one also seems like a security hole being closed, and the
 'feature' could be a candidate for stable backports, but Arnd tried
 that, and it was quite non-trivial. So perhaps  we'll need some help
 from the subsystem developers here.
>>>
>>> Could you please help us sort this out? Our goal is to help Greg with
>>> testing stable kernels, and currently the seccomp tests fail due to
>>> missing feature (seccomp ptrace hole closure) getting tested via
>>> latest kselftest.
>>>
>>> If you feel the feature isn't a stable candidate, then could you
>>> please help make the test degrade gracefully in its absence?
>>
>> I don't really want to have that change be a backport -- it's quite
>> invasive across multiple architectures.
>>
>> I would say just add a kernel version check to the test. This is
>> probably not the only selftest that will need such things. :)
>
> Adding release checks to selftests is going to problematic for maintenance.
> Tests should fail gracefully if feature isn't supported in older kernels.
>
> Several tests do that now and please find a way to check for dependencies
> and feature availability and fail the test gracefully. If there is a test
> that can't do that for some reason, we can discuss it, but as a general
> rule, I don't want to see kselftest patches that check release.

If a future kernel inadvertently loses the new feature and degrades to
the behavior of old kernels, that would be a serious bug and should be
caught.

--Andy

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Andy Lutomirski]

On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan  wrote:
> On 06/22/2017 10:53 AM, Kees Cook wrote:
>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  
>> wrote:
>>> Hi Kees, Andy,
>>>
>>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
 feature and test together.
 - This one also seems like a security hole being closed, and the
 'feature' could be a candidate for stable backports, but Arnd tried
 that, and it was quite non-trivial. So perhaps  we'll need some help
 from the subsystem developers here.
>>>
>>> Could you please help us sort this out? Our goal is to help Greg with
>>> testing stable kernels, and currently the seccomp tests fail due to
>>> missing feature (seccomp ptrace hole closure) getting tested via
>>> latest kselftest.
>>>
>>> If you feel the feature isn't a stable candidate, then could you
>>> please help make the test degrade gracefully in its absence?
>>
>> I don't really want to have that change be a backport -- it's quite
>> invasive across multiple architectures.
>>
>> I would say just add a kernel version check to the test. This is
>> probably not the only selftest that will need such things. :)
>
> Adding release checks to selftests is going to problematic for maintenance.
> Tests should fail gracefully if feature isn't supported in older kernels.
>
> Several tests do that now and please find a way to check for dependencies
> and feature availability and fail the test gracefully. If there is a test
> that can't do that for some reason, we can discuss it, but as a general
> rule, I don't want to see kselftest patches that check release.

If a future kernel inadvertently loses the new feature and degrades to
the behavior of old kernels, that would be a serious bug and should be
caught.

--Andy

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 10:53 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  wrote:
>> Hi Kees, Andy,
>>
>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>>> feature and test together.
>>> - This one also seems like a security hole being closed, and the
>>> 'feature' could be a candidate for stable backports, but Arnd tried
>>> that, and it was quite non-trivial. So perhaps  we'll need some help
>>> from the subsystem developers here.
>>
>> Could you please help us sort this out? Our goal is to help Greg with
>> testing stable kernels, and currently the seccomp tests fail due to
>> missing feature (seccomp ptrace hole closure) getting tested via
>> latest kselftest.
>>
>> If you feel the feature isn't a stable candidate, then could you
>> please help make the test degrade gracefully in its absence?
> 
> I don't really want to have that change be a backport -- it's quite
> invasive across multiple architectures.
> 
> I would say just add a kernel version check to the test. This is
> probably not the only selftest that will need such things. :)

Adding release checks to selftests is going to problematic for maintenance.
Tests should fail gracefully if feature isn't supported in older kernels.

Several tests do that now and please find a way to check for dependencies
and feature availability and fail the test gracefully. If there is a test
that can't do that for some reason, we can discuss it, but as a general
rule, I don't want to see kselftest patches that check release.

thanks,
-- Shuah

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Shuah Khan]

On 06/22/2017 10:53 AM, Kees Cook wrote:
> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  wrote:
>> Hi Kees, Andy,
>>
>> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>>> feature and test together.
>>> - This one also seems like a security hole being closed, and the
>>> 'feature' could be a candidate for stable backports, but Arnd tried
>>> that, and it was quite non-trivial. So perhaps  we'll need some help
>>> from the subsystem developers here.
>>
>> Could you please help us sort this out? Our goal is to help Greg with
>> testing stable kernels, and currently the seccomp tests fail due to
>> missing feature (seccomp ptrace hole closure) getting tested via
>> latest kselftest.
>>
>> If you feel the feature isn't a stable candidate, then could you
>> please help make the test degrade gracefully in its absence?
> 
> I don't really want to have that change be a backport -- it's quite
> invasive across multiple architectures.
> 
> I would say just add a kernel version check to the test. This is
> probably not the only selftest that will need such things. :)

Adding release checks to selftests is going to problematic for maintenance.
Tests should fail gracefully if feature isn't supported in older kernels.

Several tests do that now and please find a way to check for dependencies
and feature availability and fail the test gracefully. If there is a test
that can't do that for some reason, we can discuss it, but as a general
rule, I don't want to see kselftest patches that check release.

thanks,
-- Shuah

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Kees Cook]

On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  wrote:
> Hi Kees, Andy,
>
> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> feature and test together.
>> - This one also seems like a security hole being closed, and the
>> 'feature' could be a candidate for stable backports, but Arnd tried
>> that, and it was quite non-trivial. So perhaps  we'll need some help
>> from the subsystem developers here.
>
> Could you please help us sort this out? Our goal is to help Greg with
> testing stable kernels, and currently the seccomp tests fail due to
> missing feature (seccomp ptrace hole closure) getting tested via
> latest kselftest.
>
> If you feel the feature isn't a stable candidate, then could you
> please help make the test degrade gracefully in its absence?

I don't really want to have that change be a backport -- it's quite
invasive across multiple architectures.

I would say just add a kernel version check to the test. This is
probably not the only selftest that will need such things. :)

I'd be happy to review such changes!

-Kees

-- 
Kees Cook
Pixel Security

Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Kees Cook]

On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal  wrote:
> Hi Kees, Andy,
>
> On 15 June 2017 at 23:26, Sumit Semwal  wrote:
>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
>> feature and test together.
>> - This one also seems like a security hole being closed, and the
>> 'feature' could be a candidate for stable backports, but Arnd tried
>> that, and it was quite non-trivial. So perhaps  we'll need some help
>> from the subsystem developers here.
>
> Could you please help us sort this out? Our goal is to help Greg with
> testing stable kernels, and currently the seccomp tests fail due to
> missing feature (seccomp ptrace hole closure) getting tested via
> latest kselftest.
>
> If you feel the feature isn't a stable candidate, then could you
> please help make the test degrade gracefully in its absence?

I don't really want to have that change be a backport -- it's quite
invasive across multiple architectures.

I would say just add a kernel version check to the test. This is
probably not the only selftest that will need such things. :)

I'd be happy to review such changes!

-Kees

-- 
Kees Cook
Pixel Security

seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Sumit Semwal]

Hi Kees, Andy,

On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

Could you please help us sort this out? Our goal is to help Greg with
testing stable kernels, and currently the seccomp tests fail due to
missing feature (seccomp ptrace hole closure) getting tested via
latest kselftest.

If you feel the feature isn't a stable candidate, then could you
please help make the test degrade gracefully in its absence?

Greatly appreciated!

Thanks and best regards,
Sumit

seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]

[Sumit Semwal]

Hi Kees, Andy,

On 15 June 2017 at 23:26, Sumit Semwal  wrote:
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

Could you please help us sort this out? Our goal is to help Greg with
testing stable kernels, and currently the seccomp tests fail due to
missing feature (seccomp ptrace hole closure) getting tested via
latest kselftest.

If you feel the feature isn't a stable candidate, then could you
please help make the test degrade gracefully in its absence?

Greatly appreciated!

Thanks and best regards,
Sumit

Re: LTS testing with latest kselftests - some failures

[Kees Cook]

On Fri, Jun 16, 2017 at 9:46 AM, Luis R. Rodriguez  wrote:
>> 47e0bbb7fa98 (test: firmware_class: report errors properly on failure)
>
> Hrm, come to think of it, this *might* have been a stable fix, however the
> fix did not mention any specific about real issue with this. Kees?

This was mostly a cosmetic fix, though it does fix the return code. It
can certainly go to stable, but I try to only push more critical
things to -stable.

-Kees

-- 
Kees Cook
Pixel Security

Re: LTS testing with latest kselftests - some failures

[Kees Cook]

On Fri, Jun 16, 2017 at 9:46 AM, Luis R. Rodriguez  wrote:
>> 47e0bbb7fa98 (test: firmware_class: report errors properly on failure)
>
> Hrm, come to think of it, this *might* have been a stable fix, however the
> fix did not mention any specific about real issue with this. Kees?

This was mostly a cosmetic fix, though it does fix the return code. It
can certainly go to stable, but I try to only push more critical
things to -stable.

-Kees

-- 
Kees Cook
Pixel Security

Re: LTS testing with latest kselftests - some failures

[Luis R. Rodriguez]

On Mon, Jun 19, 2017 at 10:55:01PM +0800, Greg Kroah-Hartman wrote:
> On Mon, Jun 19, 2017 at 04:48:05PM +0200, Luis R. Rodriguez wrote:
> > On Sat, Jun 17, 2017 at 06:16:35AM +0200, Greg Kroah-Hartman wrote:
> > > On Fri, Jun 16, 2017 at 09:47:21PM +0200, Luis R. Rodriguez wrote:
> > > > Some of the knobs however are for extending tests for
> > > > existing APIs in older kernels, the async and custom fallback one are an
> > > > example.  There are a series of test cases later added which could help
> > > > test LTS kernels. Would Linaro pick these test driver enhancements to 
> > > > help
> > > > increase coverage of tests? Or is it not worth it? If its worth it then
> > > > what I was curious was how to help make this easier for this process to
> > > > bloom.
> > > 
> > > I don't understand, what do you mean by "pick these test driver
> > > enhancements"?  What kind of "knobs" are there in tests?  Shouldn't the
> > > tests "just work" with no kind of special configuration of the tests be
> > > needed?  No user is going to know to enable something special.
> > 
> > Test driver knobs, so for instance the async and custom patches referenced
> > enable the shell script to use the async api and the custom API.
> 
> Ah, testing kernel code, that makes more sense.  I don't really know, if
> the apis are present in the older kernel trees, I don't have a problem
> having them be backported to stable kernel releases, as this isn't code
> that people are actually running on a "normal" system.

Wonderful, will peg test-driver changes as stable then when this fits. I really
do think this will make test coverage better.

  Luis

Re: LTS testing with latest kselftests - some failures

[Luis R. Rodriguez]

On Mon, Jun 19, 2017 at 10:55:01PM +0800, Greg Kroah-Hartman wrote:
> On Mon, Jun 19, 2017 at 04:48:05PM +0200, Luis R. Rodriguez wrote:
> > On Sat, Jun 17, 2017 at 06:16:35AM +0200, Greg Kroah-Hartman wrote:
> > > On Fri, Jun 16, 2017 at 09:47:21PM +0200, Luis R. Rodriguez wrote:
> > > > Some of the knobs however are for extending tests for
> > > > existing APIs in older kernels, the async and custom fallback one are an
> > > > example.  There are a series of test cases later added which could help
> > > > test LTS kernels. Would Linaro pick these test driver enhancements to 
> > > > help
> > > > increase coverage of tests? Or is it not worth it? If its worth it then
> > > > what I was curious was how to help make this easier for this process to
> > > > bloom.
> > > 
> > > I don't understand, what do you mean by "pick these test driver
> > > enhancements"?  What kind of "knobs" are there in tests?  Shouldn't the
> > > tests "just work" with no kind of special configuration of the tests be
> > > needed?  No user is going to know to enable something special.
> > 
> > Test driver knobs, so for instance the async and custom patches referenced
> > enable the shell script to use the async api and the custom API.
> 
> Ah, testing kernel code, that makes more sense.  I don't really know, if
> the apis are present in the older kernel trees, I don't have a problem
> having them be backported to stable kernel releases, as this isn't code
> that people are actually running on a "normal" system.

Wonderful, will peg test-driver changes as stable then when this fits. I really
do think this will make test coverage better.

  Luis

Re: LTS testing with latest kselftests - some failures

[Greg Kroah-Hartman]

On Mon, Jun 19, 2017 at 04:48:05PM +0200, Luis R. Rodriguez wrote:
> On Sat, Jun 17, 2017 at 06:16:35AM +0200, Greg Kroah-Hartman wrote:
> > On Fri, Jun 16, 2017 at 09:47:21PM +0200, Luis R. Rodriguez wrote:
> > > Some of the knobs however are for extending tests for
> > > existing APIs in older kernels, the async and custom fallback one are an
> > > example.  There are a series of test cases later added which could help
> > > test LTS kernels. Would Linaro pick these test driver enhancements to help
> > > increase coverage of tests? Or is it not worth it? If its worth it then
> > > what I was curious was how to help make this easier for this process to
> > > bloom.
> > 
> > I don't understand, what do you mean by "pick these test driver
> > enhancements"?  What kind of "knobs" are there in tests?  Shouldn't the
> > tests "just work" with no kind of special configuration of the tests be
> > needed?  No user is going to know to enable something special.
> 
> Test driver knobs, so for instance the async and custom patches referenced
> enable the shell script to use the async api and the custom API.

Ah, testing kernel code, that makes more sense.  I don't really know, if
the apis are present in the older kernel trees, I don't have a problem
having them be backported to stable kernel releases, as this isn't code
that people are actually running on a "normal" system.

thanks,

greg k-h

Re: LTS testing with latest kselftests - some failures

[Greg Kroah-Hartman]

On Mon, Jun 19, 2017 at 04:48:05PM +0200, Luis R. Rodriguez wrote:
> On Sat, Jun 17, 2017 at 06:16:35AM +0200, Greg Kroah-Hartman wrote:
> > On Fri, Jun 16, 2017 at 09:47:21PM +0200, Luis R. Rodriguez wrote:
> > > Some of the knobs however are for extending tests for
> > > existing APIs in older kernels, the async and custom fallback one are an
> > > example.  There are a series of test cases later added which could help
> > > test LTS kernels. Would Linaro pick these test driver enhancements to help
> > > increase coverage of tests? Or is it not worth it? If its worth it then
> > > what I was curious was how to help make this easier for this process to
> > > bloom.
> > 
> > I don't understand, what do you mean by "pick these test driver
> > enhancements"?  What kind of "knobs" are there in tests?  Shouldn't the
> > tests "just work" with no kind of special configuration of the tests be
> > needed?  No user is going to know to enable something special.
> 
> Test driver knobs, so for instance the async and custom patches referenced
> enable the shell script to use the async api and the custom API.

Ah, testing kernel code, that makes more sense.  I don't really know, if
the apis are present in the older kernel trees, I don't have a problem
having them be backported to stable kernel releases, as this isn't code
that people are actually running on a "normal" system.

thanks,

greg k-h

Re: LTS testing with latest kselftests - some failures

[Luis R. Rodriguez]

On Sat, Jun 17, 2017 at 06:16:35AM +0200, Greg Kroah-Hartman wrote:
> On Fri, Jun 16, 2017 at 09:47:21PM +0200, Luis R. Rodriguez wrote:
> > Some of the knobs however are for extending tests for
> > existing APIs in older kernels, the async and custom fallback one are an
> > example.  There are a series of test cases later added which could help
> > test LTS kernels. Would Linaro pick these test driver enhancements to help
> > increase coverage of tests? Or is it not worth it? If its worth it then
> > what I was curious was how to help make this easier for this process to
> > bloom.
> 
> I don't understand, what do you mean by "pick these test driver
> enhancements"?  What kind of "knobs" are there in tests?  Shouldn't the
> tests "just work" with no kind of special configuration of the tests be
> needed?  No user is going to know to enable something special.

Test driver knobs, so for instance the async and custom patches referenced
enable the shell script to use the async api and the custom API.

  Luis

Re: LTS testing with latest kselftests - some failures

[Luis R. Rodriguez]

On Sat, Jun 17, 2017 at 06:16:35AM +0200, Greg Kroah-Hartman wrote:
> On Fri, Jun 16, 2017 at 09:47:21PM +0200, Luis R. Rodriguez wrote:
> > Some of the knobs however are for extending tests for
> > existing APIs in older kernels, the async and custom fallback one are an
> > example.  There are a series of test cases later added which could help
> > test LTS kernels. Would Linaro pick these test driver enhancements to help
> > increase coverage of tests? Or is it not worth it? If its worth it then
> > what I was curious was how to help make this easier for this process to
> > bloom.
> 
> I don't understand, what do you mean by "pick these test driver
> enhancements"?  What kind of "knobs" are there in tests?  Shouldn't the
> tests "just work" with no kind of special configuration of the tests be
> needed?  No user is going to know to enable something special.

Test driver knobs, so for instance the async and custom patches referenced
enable the shell script to use the async api and the custom API.

  Luis

Re: LTS testing with latest kselftests - some failures

[Greg Kroah-Hartman]

On Fri, Jun 16, 2017 at 09:47:21PM +0200, Luis R. Rodriguez wrote:
> Some of the knobs however are for extending tests for
> existing APIs in older kernels, the async and custom fallback one are an
> example.  There are a series of test cases later added which could help
> test LTS kernels. Would Linaro pick these test driver enhancements to help
> increase coverage of tests? Or is it not worth it? If its worth it then
> what I was curious was how to help make this easier for this process to
> bloom.

I don't understand, what do you mean by "pick these test driver
enhancements"?  What kind of "knobs" are there in tests?  Shouldn't the
tests "just work" with no kind of special configuration of the tests be
needed?  No user is going to know to enable something special.

Make the tests "just work" please, because given the large number of
them, no one is going to know to look for special things.

thanks,

greg k-h

Re: LTS testing with latest kselftests - some failures

[Greg Kroah-Hartman]

On Fri, Jun 16, 2017 at 09:47:21PM +0200, Luis R. Rodriguez wrote:
> Some of the knobs however are for extending tests for
> existing APIs in older kernels, the async and custom fallback one are an
> example.  There are a series of test cases later added which could help
> test LTS kernels. Would Linaro pick these test driver enhancements to help
> increase coverage of tests? Or is it not worth it? If its worth it then
> what I was curious was how to help make this easier for this process to
> bloom.

I don't understand, what do you mean by "pick these test driver
enhancements"?  What kind of "knobs" are there in tests?  Shouldn't the
tests "just work" with no kind of special configuration of the tests be
needed?  No user is going to know to enable something special.

Make the tests "just work" please, because given the large number of
them, no one is going to know to look for special things.

thanks,

greg k-h

Re: LTS testing with latest kselftests - some failures

[Fengguang Wu]

On Fri, Jun 16, 2017 at 06:46:51PM +0200, Luis R. Rodriguez wrote:

Kees, please review 47e0bbb7fa98 below.
Brian, please review be4a1326d12c below.

On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote:

Hello Greg, Shuah,

While testing 4.4.y and 4.9.y LTS kernels with latest kselftest,


To be clear it seems like you are taking the latest upstream ksefltest and run
it against older stable kernels. Furthermore you seem to only run the shell
script tests but are using older kselftests drivers? Is this all correct?
Otherwise it is unclear how you are running into the issues below.

Does 0-day so the same? I thought 0-day takes just the kselftest from each tree
submitted. That *seemed* to me like the way it was designed. Shuah ?


Yes in 0-day, we run the kselftest code corresponding to the current kernel.

Thanks,
Fengguang

Re: LTS testing with latest kselftests - some failures

[Fengguang Wu]

On Fri, Jun 16, 2017 at 06:46:51PM +0200, Luis R. Rodriguez wrote:

Kees, please review 47e0bbb7fa98 below.
Brian, please review be4a1326d12c below.

On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote:

Hello Greg, Shuah,

While testing 4.4.y and 4.9.y LTS kernels with latest kselftest,


To be clear it seems like you are taking the latest upstream ksefltest and run
it against older stable kernels. Furthermore you seem to only run the shell
script tests but are using older kselftests drivers? Is this all correct?
Otherwise it is unclear how you are running into the issues below.

Does 0-day so the same? I thought 0-day takes just the kselftest from each tree
submitted. That *seemed* to me like the way it was designed. Shuah ?


Yes in 0-day, we run the kselftest code corresponding to the current kernel.

Thanks,
Fengguang

Re: LTS testing with latest kselftests - some failures

[Luis R. Rodriguez]

On Fri, Jun 16, 2017 at 09:29:52PM +0200, Greg Kroah-Hartman wrote:
> On Fri, Jun 16, 2017 at 06:46:51PM +0200, Luis R. Rodriguez wrote:
> > Kees, please review 47e0bbb7fa98 below.
> > Brian, please review be4a1326d12c below.
> > 
> > On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote:
> > > Hello Greg, Shuah,
> > > 
> > > While testing 4.4.y and 4.9.y LTS kernels with latest kselftest,
> > 
> > To be clear it seems like you are taking the latest upstream ksefltest and 
> > run
> > it against older stable kernels. Furthermore you seem to only run the shell
> > script tests but are using older kselftests drivers? Is this all correct?
> > Otherwise it is unclear how you are running into the issues below.
> > 
> > Does 0-day so the same? I thought 0-day takes just the kselftest from each 
> > tree
> > submitted. That *seemed* to me like the way it was designed. Shuah ?
> > 
> > What's the name of *this* testing effort BTW? Is this part of the overall
> > kselftest ? Or is this something Linaro does for LTS kernels ? If there
> > is a name to your effort can you document it here so that others are aware:
> 
> It's a "test LTS kernels to make sure Greg didn't break anything" type
> of testing effort that Linaro is helping out with.

OK so its "standard" :)

> This could also be called, "it's about time someone did this..." :)

Good to know!

> > > we found a couple more test failures due to test-kernel mismatch:
> > > 
> > > 1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to
> > > tests, and related updates to lib/test_firmware.c to improve the
> > > tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable
> > > 4.9 misses the second update.
> > 
> > <-- snip, skipped 2. and 3. -->
> > 
> > > For all the 3 listed above, we will try and update the tests to 
> > > gracefully exit.
> > 
> > Hmm, this actually raises a good kselftest question:
> > 
> > I *though* kselftests were running tests on par with the kernels, so we 
> > would
> > *not* take latest upstream kselftests to test against older kernels. Is this
> > incorrect?
> 
> That is incorrect.  Your test should always degrade gracefully if the
> feature is not present in the kernel under test.

OK perfect, now I know to look for knobs in the shell tests to ensure this
doesn't happen again.

Some of the knobs however are for extending tests for
existing APIs in older kernels, the async and custom fallback one are an
example.  There are a series of test cases later added which could help
test LTS kernels. Would Linaro pick these test driver enhancements to help
increase coverage of tests? Or is it not worth it? If its worth it then
what I was curious was how to help make this easier for this process to
bloom.

> If the test is for a
> bug that was fixed, then that fix should also go to a stable kernel
> release.

Indeed, that was perfectly clear.

  Luis

Re: LTS testing with latest kselftests - some failures

[Luis R. Rodriguez]

On Fri, Jun 16, 2017 at 09:29:52PM +0200, Greg Kroah-Hartman wrote:
> On Fri, Jun 16, 2017 at 06:46:51PM +0200, Luis R. Rodriguez wrote:
> > Kees, please review 47e0bbb7fa98 below.
> > Brian, please review be4a1326d12c below.
> > 
> > On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote:
> > > Hello Greg, Shuah,
> > > 
> > > While testing 4.4.y and 4.9.y LTS kernels with latest kselftest,
> > 
> > To be clear it seems like you are taking the latest upstream ksefltest and 
> > run
> > it against older stable kernels. Furthermore you seem to only run the shell
> > script tests but are using older kselftests drivers? Is this all correct?
> > Otherwise it is unclear how you are running into the issues below.
> > 
> > Does 0-day so the same? I thought 0-day takes just the kselftest from each 
> > tree
> > submitted. That *seemed* to me like the way it was designed. Shuah ?
> > 
> > What's the name of *this* testing effort BTW? Is this part of the overall
> > kselftest ? Or is this something Linaro does for LTS kernels ? If there
> > is a name to your effort can you document it here so that others are aware:
> 
> It's a "test LTS kernels to make sure Greg didn't break anything" type
> of testing effort that Linaro is helping out with.

OK so its "standard" :)

> This could also be called, "it's about time someone did this..." :)

Good to know!

> > > we found a couple more test failures due to test-kernel mismatch:
> > > 
> > > 1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to
> > > tests, and related updates to lib/test_firmware.c to improve the
> > > tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable
> > > 4.9 misses the second update.
> > 
> > <-- snip, skipped 2. and 3. -->
> > 
> > > For all the 3 listed above, we will try and update the tests to 
> > > gracefully exit.
> > 
> > Hmm, this actually raises a good kselftest question:
> > 
> > I *though* kselftests were running tests on par with the kernels, so we 
> > would
> > *not* take latest upstream kselftests to test against older kernels. Is this
> > incorrect?
> 
> That is incorrect.  Your test should always degrade gracefully if the
> feature is not present in the kernel under test.

OK perfect, now I know to look for knobs in the shell tests to ensure this
doesn't happen again.

Some of the knobs however are for extending tests for
existing APIs in older kernels, the async and custom fallback one are an
example.  There are a series of test cases later added which could help
test LTS kernels. Would Linaro pick these test driver enhancements to help
increase coverage of tests? Or is it not worth it? If its worth it then
what I was curious was how to help make this easier for this process to
bloom.

> If the test is for a
> bug that was fixed, then that fix should also go to a stable kernel
> release.

Indeed, that was perfectly clear.

  Luis

Re: LTS testing with latest kselftests - some failures

[Greg Kroah-Hartman]

On Fri, Jun 16, 2017 at 06:46:51PM +0200, Luis R. Rodriguez wrote:
> Kees, please review 47e0bbb7fa98 below.
> Brian, please review be4a1326d12c below.
> 
> On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote:
> > Hello Greg, Shuah,
> > 
> > While testing 4.4.y and 4.9.y LTS kernels with latest kselftest,
> 
> To be clear it seems like you are taking the latest upstream ksefltest and run
> it against older stable kernels. Furthermore you seem to only run the shell
> script tests but are using older kselftests drivers? Is this all correct?
> Otherwise it is unclear how you are running into the issues below.
> 
> Does 0-day so the same? I thought 0-day takes just the kselftest from each 
> tree
> submitted. That *seemed* to me like the way it was designed. Shuah ?
> 
> What's the name of *this* testing effort BTW? Is this part of the overall
> kselftest ? Or is this something Linaro does for LTS kernels ? If there
> is a name to your effort can you document it here so that others are aware:

It's a "test LTS kernels to make sure Greg didn't break anything" type
of testing effort that Linaro is helping out with.

This could also be called, "it's about time someone did this..." :)

> > we found a couple more test failures due to test-kernel mismatch:
> > 
> > 1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to
> > tests, and related updates to lib/test_firmware.c to improve the
> > tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable
> > 4.9 misses the second update.
> 
> <-- snip, skipped 2. and 3. -->
> 
> > For all the 3 listed above, we will try and update the tests to gracefully 
> > exit.
> 
> Hmm, this actually raises a good kselftest question:
> 
> I *though* kselftests were running tests on par with the kernels, so we would
> *not* take latest upstream kselftests to test against older kernels. Is this
> incorrect?

That is incorrect.  Your test should always degrade gracefully if the
feature is not present in the kernel under test.  If the test is for a
bug that was fixed, then that fix should also go to a stable kernel
release.

thanks,

greg k-h

Re: LTS testing with latest kselftests - some failures

[Greg Kroah-Hartman]

On Fri, Jun 16, 2017 at 06:46:51PM +0200, Luis R. Rodriguez wrote:
> Kees, please review 47e0bbb7fa98 below.
> Brian, please review be4a1326d12c below.
> 
> On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote:
> > Hello Greg, Shuah,
> > 
> > While testing 4.4.y and 4.9.y LTS kernels with latest kselftest,
> 
> To be clear it seems like you are taking the latest upstream ksefltest and run
> it against older stable kernels. Furthermore you seem to only run the shell
> script tests but are using older kselftests drivers? Is this all correct?
> Otherwise it is unclear how you are running into the issues below.
> 
> Does 0-day so the same? I thought 0-day takes just the kselftest from each 
> tree
> submitted. That *seemed* to me like the way it was designed. Shuah ?
> 
> What's the name of *this* testing effort BTW? Is this part of the overall
> kselftest ? Or is this something Linaro does for LTS kernels ? If there
> is a name to your effort can you document it here so that others are aware:

It's a "test LTS kernels to make sure Greg didn't break anything" type
of testing effort that Linaro is helping out with.

This could also be called, "it's about time someone did this..." :)

> > we found a couple more test failures due to test-kernel mismatch:
> > 
> > 1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to
> > tests, and related updates to lib/test_firmware.c to improve the
> > tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable
> > 4.9 misses the second update.
> 
> <-- snip, skipped 2. and 3. -->
> 
> > For all the 3 listed above, we will try and update the tests to gracefully 
> > exit.
> 
> Hmm, this actually raises a good kselftest question:
> 
> I *though* kselftests were running tests on par with the kernels, so we would
> *not* take latest upstream kselftests to test against older kernels. Is this
> incorrect?

That is incorrect.  Your test should always degrade gracefully if the
feature is not present in the kernel under test.  If the test is for a
bug that was fixed, then that fix should also go to a stable kernel
release.

thanks,

greg k-h

Re: LTS testing with latest kselftests - some failures

[Alexander Alemayhu]

On Fri, Jun 16, 2017 at 01:08:04PM +0530, Sumit Semwal wrote:
> 
> Thanks, this was quite helpful, and so now bpf tests build on x86_64
> with current mainline for me. Perhaps we should document these
> somewhere, as dependencies?
>
There is already some documentation available[0], but something in the kernel
tree would be nice. Please send the patch(es) to netdev.

Thanks.

[0]: http://docs.cilium.io/en/latest/bpf/#development-environment

-- 
Mit freundlichen Grüßen

Alexander Alemayhu

Re: LTS testing with latest kselftests - some failures

[Alexander Alemayhu]

On Fri, Jun 16, 2017 at 01:08:04PM +0530, Sumit Semwal wrote:
> 
> Thanks, this was quite helpful, and so now bpf tests build on x86_64
> with current mainline for me. Perhaps we should document these
> somewhere, as dependencies?
>
There is already some documentation available[0], but something in the kernel
tree would be nice. Please send the patch(es) to netdev.

Thanks.

[0]: http://docs.cilium.io/en/latest/bpf/#development-environment

-- 
Mit freundlichen Grüßen

Alexander Alemayhu

Re: LTS testing with latest kselftests - some failures

[Luis R. Rodriguez]

Kees, please review 47e0bbb7fa98 below.
Brian, please review be4a1326d12c below.

On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote:
> Hello Greg, Shuah,
> 
> While testing 4.4.y and 4.9.y LTS kernels with latest kselftest,

To be clear it seems like you are taking the latest upstream ksefltest and run
it against older stable kernels. Furthermore you seem to only run the shell
script tests but are using older kselftests drivers? Is this all correct?
Otherwise it is unclear how you are running into the issues below.

Does 0-day so the same? I thought 0-day takes just the kselftest from each tree
submitted. That *seemed* to me like the way it was designed. Shuah ?

What's the name of *this* testing effort BTW? Is this part of the overall
kselftest ? Or is this something Linaro does for LTS kernels ? If there
is a name to your effort can you document it here so that others are aware:

https://bottest.wiki.kernel.org/

Replying below only the firmware stuff.

> we found a couple more test failures due to test-kernel mismatch:
> 
> 1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to
> tests, and related updates to lib/test_firmware.c to improve the
> tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable
> 4.9 misses the second update.

<-- snip, skipped 2. and 3. -->

> For all the 3 listed above, we will try and update the tests to gracefully 
> exit.

Hmm, this actually raises a good kselftest question:

I *though* kselftests were running tests on par with the kernels, so we would
*not* take latest upstream kselftests to test against older kernels. Is this
incorrect?

If this is indeed incorrect then indeed you have a problem and then I understand
this email, however this manual approach seems rather fragile. If I would have
understood this practice was expected I would have tried to design tests cases
a bit differently, but *also* it does beg the question about what to do when
the latest kselftest shell script does require some new knob form a test driver
which *is* fine to backport to the respective ksefltest C test-driver for an
older kernel. What makes this hard is that C test-drivers may depend on new API,
so you might have to do some manual work to backport some fancy new API in old
ways. This makes me question the value to this mismatch between shell and C
test-drivers on kselftests. Your effort seems to be all manual and empirical ?
Did we design kselftests with this in mind?  Even though using the latest
kselftest shell tests against older stable kernels with older kselftest C
drivers seems to be like a good idea (provided the above is resolved) your
current suggestion to just drop some tests seems worrisome and seems to
*invalidate* the gains of such effort and all the pains you are going through.

If you are just dropping patches / tests loosely your approach could be missing
out on valid tests which *may* have missed out on respective stable patches.

The test-firmware async knobs are a good example, and so is the firmware custom
fallback trigger.  These patches are just extending test coverage, so they help
test the existing old kernel API.

Its not worthy to Cc stable on those as they are not fixing a stable issue,
however their test at times may reveal an issue which a subsequent patch *does
fix* which *is* Cc stable.

An alternative to the way you are doing things, if I understood it correctly,
would be for us to consider evaluating pegging as stable candidates not only
ksefltest shell but kselftest C driver extensions, then instead of using the
latest ksefltests against older kernels you could just use the kselftest on the
respective old stable kernel, the *backport* effort becomes part of the stable
pipeline. Note I think this is very debatable  and I would not be surprised
if Greg does not like it, but its worth *considering* if there is indeed value 
to
your current attempted approach.

The alternative of course, is to only use ksefltest from each respective
kernel, under the assumption each stable fix does makes its way through.

So -- where is the metric value of your current approach? Do we have stats?

> I will also individually request subsystem authors / mailing lists for
> each of these towards help in improving these tests if required, but
> wanted to use this thread as a converging point.
> 
> Thanks and Best regards,
> Sumit.
> 
> 
> [1]: https://lkml.org/lkml/2015/12/8/816
> Patches added via [1]:
> eb910947c82f (test: firmware_class: add asynchronous request trigger)

This is an example extension to the C test driver which could be useful for
kselftest.

> be4a1326d12c (test: firmware_class: use kstrndup() where appropriate)

I can't see this being a stable candidate, its unclear why this has come up on
this thread ?

> 47e0bbb7fa98 (test: firmware_class: report errors properly on failure)

Hrm, come to think of it, this *might* have been a stable fix, however the
fix did not mention any specific about real issue with this. 

Re: LTS testing with latest kselftests - some failures

[Luis R. Rodriguez]

Kees, please review 47e0bbb7fa98 below.
Brian, please review be4a1326d12c below.

On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote:
> Hello Greg, Shuah,
> 
> While testing 4.4.y and 4.9.y LTS kernels with latest kselftest,

To be clear it seems like you are taking the latest upstream ksefltest and run
it against older stable kernels. Furthermore you seem to only run the shell
script tests but are using older kselftests drivers? Is this all correct?
Otherwise it is unclear how you are running into the issues below.

Does 0-day so the same? I thought 0-day takes just the kselftest from each tree
submitted. That *seemed* to me like the way it was designed. Shuah ?

What's the name of *this* testing effort BTW? Is this part of the overall
kselftest ? Or is this something Linaro does for LTS kernels ? If there
is a name to your effort can you document it here so that others are aware:

https://bottest.wiki.kernel.org/

Replying below only the firmware stuff.

> we found a couple more test failures due to test-kernel mismatch:
> 
> 1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to
> tests, and related updates to lib/test_firmware.c to improve the
> tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable
> 4.9 misses the second update.

<-- snip, skipped 2. and 3. -->

> For all the 3 listed above, we will try and update the tests to gracefully 
> exit.

Hmm, this actually raises a good kselftest question:

I *though* kselftests were running tests on par with the kernels, so we would
*not* take latest upstream kselftests to test against older kernels. Is this
incorrect?

If this is indeed incorrect then indeed you have a problem and then I understand
this email, however this manual approach seems rather fragile. If I would have
understood this practice was expected I would have tried to design tests cases
a bit differently, but *also* it does beg the question about what to do when
the latest kselftest shell script does require some new knob form a test driver
which *is* fine to backport to the respective ksefltest C test-driver for an
older kernel. What makes this hard is that C test-drivers may depend on new API,
so you might have to do some manual work to backport some fancy new API in old
ways. This makes me question the value to this mismatch between shell and C
test-drivers on kselftests. Your effort seems to be all manual and empirical ?
Did we design kselftests with this in mind?  Even though using the latest
kselftest shell tests against older stable kernels with older kselftest C
drivers seems to be like a good idea (provided the above is resolved) your
current suggestion to just drop some tests seems worrisome and seems to
*invalidate* the gains of such effort and all the pains you are going through.

If you are just dropping patches / tests loosely your approach could be missing
out on valid tests which *may* have missed out on respective stable patches.

The test-firmware async knobs are a good example, and so is the firmware custom
fallback trigger.  These patches are just extending test coverage, so they help
test the existing old kernel API.

Its not worthy to Cc stable on those as they are not fixing a stable issue,
however their test at times may reveal an issue which a subsequent patch *does
fix* which *is* Cc stable.

An alternative to the way you are doing things, if I understood it correctly,
would be for us to consider evaluating pegging as stable candidates not only
ksefltest shell but kselftest C driver extensions, then instead of using the
latest ksefltests against older kernels you could just use the kselftest on the
respective old stable kernel, the *backport* effort becomes part of the stable
pipeline. Note I think this is very debatable  and I would not be surprised
if Greg does not like it, but its worth *considering* if there is indeed value 
to
your current attempted approach.

The alternative of course, is to only use ksefltest from each respective
kernel, under the assumption each stable fix does makes its way through.

So -- where is the metric value of your current approach? Do we have stats?

> I will also individually request subsystem authors / mailing lists for
> each of these towards help in improving these tests if required, but
> wanted to use this thread as a converging point.
> 
> Thanks and Best regards,
> Sumit.
> 
> 
> [1]: https://lkml.org/lkml/2015/12/8/816
> Patches added via [1]:
> eb910947c82f (test: firmware_class: add asynchronous request trigger)

This is an example extension to the C test driver which could be useful for
kselftest.

> be4a1326d12c (test: firmware_class: use kstrndup() where appropriate)

I can't see this being a stable candidate, its unclear why this has come up on
this thread ?

> 47e0bbb7fa98 (test: firmware_class: report errors properly on failure)

Hrm, come to think of it, this *might* have been a stable fix, however the
fix did not mention any specific about real issue with this. 

Re: LTS testing with latest kselftests - some failures

[Sumit Semwal]

Hi Alexander,

On 16 June 2017 at 12:44, Alexander Alemayhu  wrote:
> Last time I saw similar kinds of errors gcc libraries were missing.
> Can you try rerunning after
>
>   apt-get install -y gcc-multilib g++-multilib

Thanks, this was quite helpful, and so now bpf tests build on x86_64
with current mainline for me. Perhaps we should document these
somewhere, as dependencies?

Best,
Sumit.

Re: LTS testing with latest kselftests - some failures

[Sumit Semwal]

Hi Alexander,

On 16 June 2017 at 12:44, Alexander Alemayhu  wrote:
> Last time I saw similar kinds of errors gcc libraries were missing.
> Can you try rerunning after
>
>   apt-get install -y gcc-multilib g++-multilib

Thanks, this was quite helpful, and so now bpf tests build on x86_64
with current mainline for me. Perhaps we should document these
somewhere, as dependencies?

Best,
Sumit.

Re: LTS testing with latest kselftests - some failures

[Alexander Alemayhu]

On Fri, Jun 16, 2017 at 10:01:37AM +0530, Sumit Semwal wrote:
> 
> make headers_install was missing, but it still didn't improve the
> build - here's the pastebin: https://paste.debian.net/971652/
>
Last time I saw similar kinds of errors gcc libraries were missing.
Can you try rerunning after 

  apt-get install -y gcc-multilib g++-multilib

You probably don't need or have most of them, but if the above doesn't
help, try

  apt-get install -y make gcc libssl-dev bc libelf-dev libcap-dev \
  clang gcc-multilib llvm libncurses5-dev git


-- 
Mit freundlichen Grüßen

Alexander Alemayhu

Re: LTS testing with latest kselftests - some failures

[Alexander Alemayhu]

On Fri, Jun 16, 2017 at 10:01:37AM +0530, Sumit Semwal wrote:
> 
> make headers_install was missing, but it still didn't improve the
> build - here's the pastebin: https://paste.debian.net/971652/
>
Last time I saw similar kinds of errors gcc libraries were missing.
Can you try rerunning after 

  apt-get install -y gcc-multilib g++-multilib

You probably don't need or have most of them, but if the above doesn't
help, try

  apt-get install -y make gcc libssl-dev bc libelf-dev libcap-dev \
  clang gcc-multilib llvm libncurses5-dev git


-- 
Mit freundlichen Grüßen

Alexander Alemayhu

Re: LTS testing with latest kselftests - some failures

[Sumit Semwal]

Hi Alexander,

On 16 June 2017 at 04:35, Alexander Alemayhu  wrote:
> On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote:
>>
>> 4. bpf tests: These seem to have build failures in mainline as well -
>> I also tried to build kselftest-next, but a simple 'make -C
>> tools/testing/selftests/bpf' seems to error out. Are there any special
>> instructions to build these? [I tried x86_64, arm64 cross-compile on x86_64]
>>
> Do you have the full failure output? If you haven't already you might
> also want to run 'make headers_install' in the top level directory.

make headers_install was missing, but it still didn't improve the
build - here's the pastebin: https://paste.debian.net/971652/

>
> --
> Mit freundlichen Grüßen
>
> Alexander Alemayhu

Best,
Sumit.

Re: LTS testing with latest kselftests - some failures

[Sumit Semwal]

Hi Alexander,

On 16 June 2017 at 04:35, Alexander Alemayhu  wrote:
> On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote:
>>
>> 4. bpf tests: These seem to have build failures in mainline as well -
>> I also tried to build kselftest-next, but a simple 'make -C
>> tools/testing/selftests/bpf' seems to error out. Are there any special
>> instructions to build these? [I tried x86_64, arm64 cross-compile on x86_64]
>>
> Do you have the full failure output? If you haven't already you might
> also want to run 'make headers_install' in the top level directory.

make headers_install was missing, but it still didn't improve the
build - here's the pastebin: https://paste.debian.net/971652/

>
> --
> Mit freundlichen Grüßen
>
> Alexander Alemayhu

Best,
Sumit.

Re: LTS testing with latest kselftests - some failures

[Alexander Alemayhu]

On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote:
> 
> 4. bpf tests: These seem to have build failures in mainline as well -
> I also tried to build kselftest-next, but a simple 'make -C
> tools/testing/selftests/bpf' seems to error out. Are there any special
> instructions to build these? [I tried x86_64, arm64 cross-compile on x86_64]
>
Do you have the full failure output? If you haven't already you might
also want to run 'make headers_install' in the top level directory.

-- 
Mit freundlichen Grüßen

Alexander Alemayhu

Re: LTS testing with latest kselftests - some failures

[Alexander Alemayhu]

On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote:
> 
> 4. bpf tests: These seem to have build failures in mainline as well -
> I also tried to build kselftest-next, but a simple 'make -C
> tools/testing/selftests/bpf' seems to error out. Are there any special
> instructions to build these? [I tried x86_64, arm64 cross-compile on x86_64]
>
Do you have the full failure output? If you haven't already you might
also want to run 'make headers_install' in the top level directory.

-- 
Mit freundlichen Grüßen

Alexander Alemayhu

Re: LTS testing with latest kselftests - some failures

[Shuah Khan]

Hi Sumit,

On 06/15/2017 11:56 AM, Sumit Semwal wrote:
> Hello Greg, Shuah,
> 
> While testing 4.4.y and 4.9.y LTS kernels with latest kselftest, we
> found a couple more test failures due to test-kernel mismatch:
> 
> 1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to
> tests, and related updates to lib/test_firmware.c to improve the
> tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable
> 4.9 misses the second update.

I will take a look the commit list your provided and see if it makes sense
to back-port. As we discussed earlier in this thread, fixes are something
that can be back-ported to include in the stable releases. Updates will not
be.

> 2. Bitmap test - test got added in 4.5, fails if test_bitmap.ko isn't present.

If you can send a patch to have it exit gracefully and change the message
to say unsupported feature, I can pull that into mainline can mark it for
stable inclusion if it qualifies.

> 
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

This is something I need to look at and consult with Security maintainers.

> 
> For all the 3 listed above, we will try and update the tests to gracefully 
> exit.

That is great.

> 
> 
> 4. bpf tests: These seem to have build failures in mainline as well -
> I also tried to build kselftest-next, but a simple 'make -C
> tools/testing/selftests/bpf' seems to error out. Are there any special
> instructions to build these? [I tried x86_64, arm64 cross-compile on x86_64]

Hmm. I didn't notice this one. I will check and see what's going on there.

> 
> 
> I will also individually request subsystem authors / mailing lists for
> each of these towards help in improving these tests if required, but
> wanted to use this thread as a converging point.

Please cc me and linux-kselftest mailing list.

Thanks for reporting the problems.

> 
> 
> [1]: https://lkml.org/lkml/2015/12/8/816
> Patches added via [1]:
> eb910947c82f (test: firmware_class: add asynchronous request trigger)
> be4a1326d12c (test: firmware_class: use kstrndup() where appropriate)
> 47e0bbb7fa98 (test: firmware_class: report errors properly on failure)
> 
> [2]: https://lkml.org/lkml/2017/1/23/440
> Patch added via [2]:
> 061132d2b9c9 (test_firmware: add test custom fallback trigger)
> 
> [3]: https://lkml.org/lkml/2016/6/9/627
> 
> 

thanks,
-- Shuah

Re: LTS testing with latest kselftests - some failures

[Shuah Khan]

Hi Sumit,

On 06/15/2017 11:56 AM, Sumit Semwal wrote:
> Hello Greg, Shuah,
> 
> While testing 4.4.y and 4.9.y LTS kernels with latest kselftest, we
> found a couple more test failures due to test-kernel mismatch:
> 
> 1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to
> tests, and related updates to lib/test_firmware.c to improve the
> tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable
> 4.9 misses the second update.

I will take a look the commit list your provided and see if it makes sense
to back-port. As we discussed earlier in this thread, fixes are something
that can be back-ported to include in the stable releases. Updates will not
be.

> 2. Bitmap test - test got added in 4.5, fails if test_bitmap.ko isn't present.

If you can send a patch to have it exit gracefully and change the message
to say unsupported feature, I can pull that into mainline can mark it for
stable inclusion if it qualifies.

> 
> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
> feature and test together.
> - This one also seems like a security hole being closed, and the
> 'feature' could be a candidate for stable backports, but Arnd tried
> that, and it was quite non-trivial. So perhaps  we'll need some help
> from the subsystem developers here.

This is something I need to look at and consult with Security maintainers.

> 
> For all the 3 listed above, we will try and update the tests to gracefully 
> exit.

That is great.

> 
> 
> 4. bpf tests: These seem to have build failures in mainline as well -
> I also tried to build kselftest-next, but a simple 'make -C
> tools/testing/selftests/bpf' seems to error out. Are there any special
> instructions to build these? [I tried x86_64, arm64 cross-compile on x86_64]

Hmm. I didn't notice this one. I will check and see what's going on there.

> 
> 
> I will also individually request subsystem authors / mailing lists for
> each of these towards help in improving these tests if required, but
> wanted to use this thread as a converging point.

Please cc me and linux-kselftest mailing list.

Thanks for reporting the problems.

> 
> 
> [1]: https://lkml.org/lkml/2015/12/8/816
> Patches added via [1]:
> eb910947c82f (test: firmware_class: add asynchronous request trigger)
> be4a1326d12c (test: firmware_class: use kstrndup() where appropriate)
> 47e0bbb7fa98 (test: firmware_class: report errors properly on failure)
> 
> [2]: https://lkml.org/lkml/2017/1/23/440
> Patch added via [2]:
> 061132d2b9c9 (test_firmware: add test custom fallback trigger)
> 
> [3]: https://lkml.org/lkml/2016/6/9/627
> 
> 

thanks,
-- Shuah

LTS testing with latest kselftests - some failures

[Sumit Semwal]

Hello Greg, Shuah,

While testing 4.4.y and 4.9.y LTS kernels with latest kselftest, we
found a couple more test failures due to test-kernel mismatch:

1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to
tests, and related updates to lib/test_firmware.c to improve the
tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable
4.9 misses the second update.

2. Bitmap test - test got added in 4.5, fails if test_bitmap.ko isn't present.

3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
feature and test together.
- This one also seems like a security hole being closed, and the
'feature' could be a candidate for stable backports, but Arnd tried
that, and it was quite non-trivial. So perhaps  we'll need some help
from the subsystem developers here.

For all the 3 listed above, we will try and update the tests to gracefully exit.


4. bpf tests: These seem to have build failures in mainline as well -
I also tried to build kselftest-next, but a simple 'make -C
tools/testing/selftests/bpf' seems to error out. Are there any special
instructions to build these? [I tried x86_64, arm64 cross-compile on x86_64]


I will also individually request subsystem authors / mailing lists for
each of these towards help in improving these tests if required, but
wanted to use this thread as a converging point.

Thanks and Best regards,
Sumit.


[1]: https://lkml.org/lkml/2015/12/8/816
Patches added via [1]:
eb910947c82f (test: firmware_class: add asynchronous request trigger)
be4a1326d12c (test: firmware_class: use kstrndup() where appropriate)
47e0bbb7fa98 (test: firmware_class: report errors properly on failure)

[2]: https://lkml.org/lkml/2017/1/23/440
Patch added via [2]:
061132d2b9c9 (test_firmware: add test custom fallback trigger)

[3]: https://lkml.org/lkml/2016/6/9/627

LTS testing with latest kselftests - some failures

[Sumit Semwal]

Hello Greg, Shuah,

While testing 4.4.y and 4.9.y LTS kernels with latest kselftest, we
found a couple more test failures due to test-kernel mismatch:

1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to
tests, and related updates to lib/test_firmware.c to improve the
tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable
4.9 misses the second update.

2. Bitmap test - test got added in 4.5, fails if test_bitmap.ko isn't present.

3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] -
feature and test together.
- This one also seems like a security hole being closed, and the
'feature' could be a candidate for stable backports, but Arnd tried
that, and it was quite non-trivial. So perhaps  we'll need some help
from the subsystem developers here.

For all the 3 listed above, we will try and update the tests to gracefully exit.


4. bpf tests: These seem to have build failures in mainline as well -
I also tried to build kselftest-next, but a simple 'make -C
tools/testing/selftests/bpf' seems to error out. Are there any special
instructions to build these? [I tried x86_64, arm64 cross-compile on x86_64]


I will also individually request subsystem authors / mailing lists for
each of these towards help in improving these tests if required, but
wanted to use this thread as a converging point.

Thanks and Best regards,
Sumit.


[1]: https://lkml.org/lkml/2015/12/8/816
Patches added via [1]:
eb910947c82f (test: firmware_class: add asynchronous request trigger)
be4a1326d12c (test: firmware_class: use kstrndup() where appropriate)
47e0bbb7fa98 (test: firmware_class: report errors properly on failure)

[2]: https://lkml.org/lkml/2017/1/23/440
Patch added via [2]:
061132d2b9c9 (test_firmware: add test custom fallback trigger)

[3]: https://lkml.org/lkml/2016/6/9/627