Hi all,
The last couple of days I have been discussing with you via e-mail and
IRC (much thanks to those involved) about current issues of autopkgtest
running tests for britney. I'll summarize my view of it in this e-mail
with the pro's and con's of different solutions and my proposed way
forward (for now).
ISSUES
There are a couple of issues that were the main topic of the discussion.
I'll summarize here so it is clear how I see the issue;
0. autopkgtest needs too much assistance (both in Debian and in Ubuntu)
for cases it should, IMHO, handle automatically. Part of this problem is
in dependency issues that albeit technically present, most of the time
aren't practical problems. Fixing those is my preferred solution. This
is even more true in Debian, as Ubuntu didn't want to bother Debian
maintainers with that and didn't want to carry those patches either.
1. britney triggers tests for packages that are at that moment
non-installable from unstable (especially, or maybe only, during
transitions).
2. autopkgtest enables testing migration candidates by using all
packages from testing except for the binary packages of the candidate
which come from unstable. It achieves that via apt pinning. However,
sometimes this candidate has dependencies that need to come from
unstable. The current solution for that situation is that autopkgtest
contains a fallback which removes all the pinning. This has the drawback
that all packages from unstable are allowed to be installed, including
potentially broken packages or broken combinations.
3. autopkgtest has a needs-recommends restriction which is documented to
"Enable installation of recommended packages in apt for the test
dependencies. This does not affect build dependencies." The way this is
currently implemented is via apt's APT::Install-Recommends option. This
implementation leads to more packages installed than I believe is
desirable for this option. On top of that, installing recommends may
hide real dependency issues.
4. apt doesn't install recommends if an external solver is used. #900989
[no-recommends]
5. autopkgtest (or apt) is silent when recommends can't be installed.
Typically, tests that need the recommends will just fail, but it will
not be obvious why this is. The failure is correct, but the error
message will waste peoples time in debugging. So in my opinion it is a
must to improve the error handling of this situation. The FAIL will
remain a FAIL, but avoids wasting peoples time. #896698 [no-fail]
SOLUTIONS
1.a. As already discussed with nthykier some months ago, britney could
do some *recursive* installability checks before requesting a test. This
will drive down the number of false negatives, especially during
transitions. What is more, it will reduce the stress on autopkgtest in
combination with apt to come up with the right solution while there is none.
pro:
- less failures that just need a redo later-> less time spent by humans
to check
con:
- (some) duplication of logic, as the real installability check of
britney happens in phase 2, so after the policy checks.
note:
- doesn't need to be perfect, just better than now
2.a. If autopkgtest is told which packages are needed from unstable
(even with the exact version) to have a coherent set, it doesn't need to
guess what is a reasonable solution for britney. Therefore britney could
output a set of packages from unstable per test, instead of just the
current trigger.
pro:
- autopkgtest doesn't need to guess what britney wants
- apt will double check installability of the solution as it is not free
anymore to do something else.
- with versions autopkgtest could fail if it doesn't test the version
that britney expected it would find,
con:
- new additional logic in britney (albeit probably similar to the
installability logic, haven't checked)
2.b. the fallback in autopkgtest could be removed. autopkgtest should
rely on it being told what packages are allowed.
pro:
- removes a hack
- easier to understand and track for manual debuggers what is going on
- less code (it already exists, so not a big one)
- guarantees there is only one entity (britney) that decides what to
test
con:
- maybe exposes unknown issues currently "hidden" behind this fallback
2.c. apt's internal solver only considers so called candidate versions.
External solvers like aspcud exist, that can optionally search for any
solution as long as installability is guaranteed.
pro:
- use a solver that is better suited for the non-standard use case we
have at hand. I.e. one that can find solutions for the request:
"install the following packages a, b, .. z from unstable and testing,
while taking *as much as possible* from testing and while taking
binary packages from source Q from unstable if they are to be
installed."
- solves LP: #1760810 [worker-crash] (worker crashes due to removal of
packages needed to function)
con:
- non-standard resolver
- apt doesn't install
