https://lists.ubuntu.com/archives/kernel-team/2019-November/105243.html

** Description changed:

  == Justification ==
- In the ubuntu_kernel_selftests, the skipped test will uses return code 4 
(KSFT_SKIP).
+ In the ubuntu_kernel_selftests, the skipped test will uses return code 4
+ (KSFT_SKIP).
  
  However the code to handle this non-zero return code was not implemented
- in the kselftest framework. And this will generate some false-positives
+ in the kselftest framework. And this will generate some false-positive
  results as those skipped tests were treated as a failure.
  
  For example the raw_skew test in timers on 4.15:
-     1..0 # Skipped: The clock was adjusted externally. Shutdown NTPd or other 
time sync daemons
-     not ok 1..7 selftests: raw_skew [FAIL]
+     1..0 # Skipped: The clock was adjusted externally. Shutdown NTPd or other 
time sync daemons
+     not ok 1..7 selftests: raw_skew [FAIL]
  
  And the test_user_copy.sh test in user (failure could be found in KVM 
kernels):
-     user: module test_user_copy is not found [SKIP]
-     not ok 1..1 selftests: test_user_copy.sh [FAIL]
+     user: module test_user_copy is not found [SKIP]
+     not ok 1..1 selftests: test_user_copy.sh [FAIL]
+ 
+ Xenial is affected by this issue as well. Considering the fact that this
+ issue in Bionic affects more tests / kernel variants, and it's way
+ easier to fix this in Bionic, I decided to go for Bionic first.
  
  == Fix ==
  771cbc3b (selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers)
  a3355440 (selftests: Fix lib.mk run_tests target shell script)
  42b44c34 (selftests: lib.mk: cleanup RUN_TESTS define and make it readable)
  3f4435b5 (selftests: lib.mk: add SKIP handling to RUN_TESTS define)
  
  All these can be cherry-picked into Bionic.
  
- The crucial patch is commit 3f4435b5, the other three patches just paves
- the road for it to be cherry-picked.
+ The crucial patch is commit 3f4435b5, the other three patches are just
+ some minor fixes that pave the road for it to be cherry-picked.
  
  == Test ==
- Tested on a Bionic amd64 node, the result shows positive outcome.
+ Tested on a Bionic amd64 node, the result shows positive.
  
  A test summary before the patch:
- END GOOD        ubuntu_kernel_selftests.setup
- END GOOD        ubuntu_kernel_selftests.breakpoints
- END GOOD        ubuntu_kernel_selftests.cpu-hotplug
- END GOOD        ubuntu_kernel_selftests.efivarfs
- END GOOD        ubuntu_kernel_selftests.memfd
- END GOOD        ubuntu_kernel_selftests.memory-hotplug
- END GOOD        ubuntu_kernel_selftests.mount
- END GOOD        ubuntu_kernel_selftests.net
- END GOOD        ubuntu_kernel_selftests.ptrace
- END GOOD        ubuntu_kernel_selftests.seccomp
- END ERROR       ubuntu_kernel_selftests.timers
- END GOOD        ubuntu_kernel_selftests.powerpc
- END GOOD        ubuntu_kernel_selftests.user
- END GOOD        ubuntu_kernel_selftests.ftrace
+ END GOOD ubuntu_kernel_selftests.setup
+ END GOOD ubuntu_kernel_selftests.breakpoints
+ END GOOD ubuntu_kernel_selftests.cpu-hotplug
+ END GOOD ubuntu_kernel_selftests.efivarfs
+ END GOOD ubuntu_kernel_selftests.memfd
+ END GOOD ubuntu_kernel_selftests.memory-hotplug
+ END GOOD ubuntu_kernel_selftests.mount
+ END GOOD ubuntu_kernel_selftests.net
+ END GOOD ubuntu_kernel_selftests.ptrace
+ END GOOD ubuntu_kernel_selftests.seccomp
+ END ERROR ubuntu_kernel_selftests.timers
+ END GOOD ubuntu_kernel_selftests.powerpc
+ END GOOD ubuntu_kernel_selftests.user
+ END GOOD ubuntu_kernel_selftests.ftrace
  
  A test summary after the patch:
- END GOOD        ubuntu_kernel_selftests.setup
- END GOOD        ubuntu_kernel_selftests.breakpoints
- END GOOD        ubuntu_kernel_selftests.cpu-hotplug
- END GOOD        ubuntu_kernel_selftests.efivarfs
- END GOOD        ubuntu_kernel_selftests.memfd
- END GOOD        ubuntu_kernel_selftests.memory-hotplug
- END GOOD        ubuntu_kernel_selftests.mount
- END GOOD        ubuntu_kernel_selftests.net
- END GOOD        ubuntu_kernel_selftests.ptrace
- END GOOD        ubuntu_kernel_selftests.seccomp
- END GOOD        ubuntu_kernel_selftests.timers
- END GOOD        ubuntu_kernel_selftests.powerpc
- END GOOD        ubuntu_kernel_selftests.user
- END GOOD        ubuntu_kernel_selftests.ftrac
+ END GOOD ubuntu_kernel_selftests.setup
+ END GOOD ubuntu_kernel_selftests.breakpoints
+ END GOOD ubuntu_kernel_selftests.cpu-hotplug
+ END GOOD ubuntu_kernel_selftests.efivarfs
+ END GOOD ubuntu_kernel_selftests.memfd
+ END GOOD ubuntu_kernel_selftests.memory-hotplug
+ END GOOD ubuntu_kernel_selftests.mount
+ END GOOD ubuntu_kernel_selftests.net
+ END GOOD ubuntu_kernel_selftests.ptrace
+ END GOOD ubuntu_kernel_selftests.seccomp
+ END GOOD ubuntu_kernel_selftests.timers
+ END GOOD ubuntu_kernel_selftests.powerpc
+ END GOOD ubuntu_kernel_selftests.user
+ END GOOD ubuntu_kernel_selftests.ftrac
  
- The raw_skew test in timers is now marked as [SKIPPED] and no longer
- causing failure.
+ No tests were skipped unintentionally. And the raw_skew test in timers
+ is now marked as [SKIPPED] and no longer causing failure.
  
  A complete test report could be found in the bug report attachment.
  
  == Regression Potential ==
- Low, this patch set just added support to handle the skip return code and 
changes are limited to the kselftest framework itself, no actual impact to 
normal users.
+ Low, this patch set just added support to handle the skip return code
+ and changes are limited to the kselftest framework itself, no actual
+ impact to normal users.
  
  
  == Original bug report ==
  In the selftests tool, we are using a KSFT_SKIP=4 return code for those 
skipped tests in the latest upstream kernel tree.
  
  But the corresponding code to take care of this return value does not
  exist in Bionic and Xenial kernel, this will cause the test still being
  marked as failed:
  
  selftests: test_user_copy.sh
  ========================================
  user: module test_user_copy is not found [SKIP]
  not ok 1..1 selftests:  test_user_copy.sh [FAIL]

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1812352

Title:
  Handle the skip return code in kernel_selftests on Bionic

Status in ubuntu-kernel-tests:
  In Progress
Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Trusty:
  Invalid
Status in linux source package in Xenial:
  Invalid
Status in linux source package in Bionic:
  In Progress

Bug description:
  == Justification ==
  In the ubuntu_kernel_selftests, the skipped test will uses return code 4
  (KSFT_SKIP).

  However the code to handle this non-zero return code was not implemented
  in the kselftest framework. And this will generate some false-positive
  results as those skipped tests were treated as a failure.

  For example the raw_skew test in timers on 4.15:
      1..0 # Skipped: The clock was adjusted externally. Shutdown NTPd or other 
time sync daemons
      not ok 1..7 selftests: raw_skew [FAIL]

  And the test_user_copy.sh test in user (failure could be found in KVM 
kernels):
      user: module test_user_copy is not found [SKIP]
      not ok 1..1 selftests: test_user_copy.sh [FAIL]

  Xenial is affected by this issue as well. Considering the fact that
  this issue in Bionic affects more tests / kernel variants, and it's
  way easier to fix this in Bionic, I decided to go for Bionic first.

  == Fix ==
  771cbc3b (selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers)
  a3355440 (selftests: Fix lib.mk run_tests target shell script)
  42b44c34 (selftests: lib.mk: cleanup RUN_TESTS define and make it readable)
  3f4435b5 (selftests: lib.mk: add SKIP handling to RUN_TESTS define)

  All these can be cherry-picked into Bionic.

  The crucial patch is commit 3f4435b5, the other three patches are just
  some minor fixes that pave the road for it to be cherry-picked.

  == Test ==
  Tested on a Bionic amd64 node, the result shows positive.

  A test summary before the patch:
  END GOOD ubuntu_kernel_selftests.setup
  END GOOD ubuntu_kernel_selftests.breakpoints
  END GOOD ubuntu_kernel_selftests.cpu-hotplug
  END GOOD ubuntu_kernel_selftests.efivarfs
  END GOOD ubuntu_kernel_selftests.memfd
  END GOOD ubuntu_kernel_selftests.memory-hotplug
  END GOOD ubuntu_kernel_selftests.mount
  END GOOD ubuntu_kernel_selftests.net
  END GOOD ubuntu_kernel_selftests.ptrace
  END GOOD ubuntu_kernel_selftests.seccomp
  END ERROR ubuntu_kernel_selftests.timers
  END GOOD ubuntu_kernel_selftests.powerpc
  END GOOD ubuntu_kernel_selftests.user
  END GOOD ubuntu_kernel_selftests.ftrace

  A test summary after the patch:
  END GOOD ubuntu_kernel_selftests.setup
  END GOOD ubuntu_kernel_selftests.breakpoints
  END GOOD ubuntu_kernel_selftests.cpu-hotplug
  END GOOD ubuntu_kernel_selftests.efivarfs
  END GOOD ubuntu_kernel_selftests.memfd
  END GOOD ubuntu_kernel_selftests.memory-hotplug
  END GOOD ubuntu_kernel_selftests.mount
  END GOOD ubuntu_kernel_selftests.net
  END GOOD ubuntu_kernel_selftests.ptrace
  END GOOD ubuntu_kernel_selftests.seccomp
  END GOOD ubuntu_kernel_selftests.timers
  END GOOD ubuntu_kernel_selftests.powerpc
  END GOOD ubuntu_kernel_selftests.user
  END GOOD ubuntu_kernel_selftests.ftrac

  No tests were skipped unintentionally. And the raw_skew test in timers
  is now marked as [SKIPPED] and no longer causing failure.

  A complete test report could be found in the bug report attachment.

  == Regression Potential ==
  Low, this patch set just added support to handle the skip return code
  and changes are limited to the kselftest framework itself, no actual
  impact to normal users.

  
  == Original bug report ==
  In the selftests tool, we are using a KSFT_SKIP=4 return code for those 
skipped tests in the latest upstream kernel tree.

  But the corresponding code to take care of this return value does not
  exist in Bionic and Xenial kernel, this will cause the test still
  being marked as failed:

  selftests: test_user_copy.sh
  ========================================
  user: module test_user_copy is not found [SKIP]
  not ok 1..1 selftests:  test_user_copy.sh [FAIL]

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1812352/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to     : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to