Hi Shuah,
At Kernel Recipes, we talked about selftests, and the fact it is
difficult to have some net tests supporting old kernels. You asked for
examples, here are two of them, with a bit of delay, sorry for that.
- The best example is certainly the Packetdrill tests (tools/testing
selftests/net/packetdrill). These tests execute some system calls
(e.g. socket creation, connect, send data, close), inject some
packets, and check the kernel behaves as expected by validating
sent packets and internal MIB counters, e.g. buffer size, etc.
=> The kernel can behave differently -- optimisations for the packet
size, buffer size, etc. while the userspace interface is the same.
=> It is difficult to keep a compatibility with all kernel versions
when validating internal behaviours: the userspace doesn't (need
to) care about how data are exchanged, internal buffer size, etc.
as long as data are moved from one end to another.
- For MPTCP that I co-maintain: features have been slowly added during
the upstream process. Some features appeared without changing the
userspace behaviour, e.g. with MPTCP, we can have multiple paths
between the client and the server, but the userspace applications
don't have to care about how paths are handled. When MPTCP got
introduced, only one path could be used. When multiple ones could be
used quickly after, there was no API change by design: the same
syscalls were used: socket(), connect/listen(), write/read(), close().
Most of the time, we try to introduce new MIB counters linked to new
features, but sometimes that's not the case, e.g. the path-manager got
optimised not to create duplicated paths, or to support paths with a
mix of IPv4 and IPv6 for the same MPTCP connections. These internal
changes didn't get expose to userspace. To have the latest selftests
being able to run on older kernels, we look at /proc/kallsyms, but
that's not always enough, we fallback to the kernel version in some
rare cases. See: tools/testing/selftests/net/mptcp/mptcp_join.sh, and
look for mptcp_lib_kallsyms_*, mptcp_lib_kversion_* and
mptcp_lib_get_counter. That's not easy to support all.
I don't know if it is specific to net tests, but when validating
internal behaviours, it is difficult to support all kernel versions. We
could expose more capabilities to the userspace, but that's not
something common here with the net tests, because (normal) userspace
programs don't have worry about some internal behaviours: that's why
some protocols are supported by the kernel, not to have to worry about
internal behaviours. If we want net selftests to be executed on older
kernel versions, we will need quite a lot of modifications, which will
be hard to maintain.
Do you think the documentation can be updated not to suppose all
selftests can be executed on any kernel versions?
"Kselftest from mainline can be run on older stable kernels."
https://docs.kernel.org/dev-tools/kselftest.html
For the net subsystem, it might be better to backport new regressions
tests linked to a bug fix, than trying to execute the latest selftests
on older kernels.
Cheers,
Matt
--
Sponsored by the NGI0 Core fund.
