Re: [LTP] Towards 4.14 LTS

[Cyril Hrubis]

Hi!
> For instance 4.13-rc just was added to the mix.
> 
> For test buckets, I???m currently dorking around with some make check targets
> for a few interesting packages. 

You may want to look into xfstests as well, we found a few kernel oopses
recently related to backported FS patches for SLES kernels.

-- 
Cyril Hrubis
chru...@suse.cz

Re: [LTP] Towards 4.14 LTS

[Cyril Hrubis]

Hi!
> For instance 4.13-rc just was added to the mix.
> 
> For test buckets, I???m currently dorking around with some make check targets
> for a few interesting packages. 

You may want to look into xfstests as well, we found a few kernel oopses
recently related to backported FS patches for SLES kernels.

-- 
Cyril Hrubis
chru...@suse.cz

Re: [LTP] Towards 4.14 LTS

[Tom Gall]

> On Nov 20, 2017, at 10:10 AM, Cyril Hrubis  wrote:
> 
> Hi!
>> So why didn???t we report these? As mentioned we???ve been tossing out dodgy
>> test cases to get to a clean baseline. We don???t need or want noise. 
>> 
>> For LTS, I want the system when it detects a failure to enable a quick 
>> bisect involving the affected test bucket. Given the nature of kernel 
>> bugs tho, there is that class of bug which only happens occasionally.
> 
> From my experience debugging kernel bugs requires an actuall human
> interaction and there is only certain level of automation that can be
> achieved. Don't take me wrong, automatic bisection and other bells and
> whistles are a nice to have, but at the end of the day you usually need
> someone to reproduce/look at the problem, possibly check the source
> code, report a bug, etc. Hence it does not make much sense to have an
> automated system without dedicated engineers assigned to review the test
> results.

You are entirely right automation only gets so far. We have a few lines
of defense that probably are worth a mention.

1) infra - sometimes results/runs need to be re-run for whatever reason.
2) triage - Crappy test case or something that is real?
3) kernel - bisecting etc

We don’t have huge dedicated teams for each category but likewise each 
has a team.

> -- 
> Cyril Hrubis
> chru...@suse.cz

Re: [LTP] Towards 4.14 LTS

[Tom Gall]

> On Nov 20, 2017, at 10:10 AM, Cyril Hrubis  wrote:
> 
> Hi!
>> So why didn???t we report these? As mentioned we???ve been tossing out dodgy
>> test cases to get to a clean baseline. We don???t need or want noise. 
>> 
>> For LTS, I want the system when it detects a failure to enable a quick 
>> bisect involving the affected test bucket. Given the nature of kernel 
>> bugs tho, there is that class of bug which only happens occasionally.
> 
> From my experience debugging kernel bugs requires an actuall human
> interaction and there is only certain level of automation that can be
> achieved. Don't take me wrong, automatic bisection and other bells and
> whistles are a nice to have, but at the end of the day you usually need
> someone to reproduce/look at the problem, possibly check the source
> code, report a bug, etc. Hence it does not make much sense to have an
> automated system without dedicated engineers assigned to review the test
> results.

You are entirely right automation only gets so far. We have a few lines
of defense that probably are worth a mention.

1) infra - sometimes results/runs need to be re-run for whatever reason.
2) triage - Crappy test case or something that is real?
3) kernel - bisecting etc

We don’t have huge dedicated teams for each category but likewise each 
has a team.

> -- 
> Cyril Hrubis
> chru...@suse.cz

Re: [LTP] Towards 4.14 LTS

[Cyril Hrubis]

Hi!
> So why didn???t we report these? As mentioned we???ve been tossing out dodgy
> test cases to get to a clean baseline. We don???t need or want noise. 
> 
> For LTS, I want the system when it detects a failure to enable a quick 
> bisect involving the affected test bucket. Given the nature of kernel 
> bugs tho, there is that class of bug which only happens occasionally.

>From my experience debugging kernel bugs requires an actuall human
interaction and there is only certain level of automation that can be
achieved. Don't take me wrong, automatic bisection and other bells and
whistles are a nice to have, but at the end of the day you usually need
someone to reproduce/look at the problem, possibly check the source
code, report a bug, etc. Hence it does not make much sense to have an
automated system without dedicated engineers assigned to review the test
results.

-- 
Cyril Hrubis
chru...@suse.cz

Re: [LTP] Towards 4.14 LTS

[Cyril Hrubis]

Hi!
> So why didn???t we report these? As mentioned we???ve been tossing out dodgy
> test cases to get to a clean baseline. We don???t need or want noise. 
> 
> For LTS, I want the system when it detects a failure to enable a quick 
> bisect involving the affected test bucket. Given the nature of kernel 
> bugs tho, there is that class of bug which only happens occasionally.

>From my experience debugging kernel bugs requires an actuall human
interaction and there is only certain level of automation that can be
achieved. Don't take me wrong, automatic bisection and other bells and
whistles are a nice to have, but at the end of the day you usually need
someone to reproduce/look at the problem, possibly check the source
code, report a bug, etc. Hence it does not make much sense to have an
automated system without dedicated engineers assigned to review the test
results.

-- 
Cyril Hrubis
chru...@suse.cz