As background, our autopkgtest workers can only access the wider internet via a http/s proxy. DNS resolution also works, but other services most likely will not.
A while ago I advised some folks to make a test 'skippable', where it was trying to access resources that couldn't be accessed due to this. This patch was duly forwarded to the Debian maintainer - thanks - as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=969057 you can read that log for the full discussion, but the tl;dr is that they feel that the "needs-internet" restriction which is already present in the package should cover this case. I've got to say that I am kind of convinced by that. README.package-tests.rst says: The test needs unrestricted internet access and also If a test does use the internet outside of the pre-configured apt domain, the test must be marked with the needs-internet restriction. Using the internet usually makes tests less reliable, so this should be kept to a minimum. But for many packages their main purpose is to interact with remote web services and thus their testing should actually cover those too, to ensure that the distribution package keeps working with their corresponding web service. Given this, I think we're on dodgy ground asking the maintainer to make changes. We have two choices that I can see: * Accept that some tests which need to access external services will fail, and we might have to hint (ignore) those failures. * Run autopkgtest with "--needs-internet=try", which means that we run the test but treat it as flaky if it fails. I prefer the latter since it will result in less work for the release team, but it does come with a caveat which is that needs-internet tests will essentially no longer be able to fail. Given that we don't have a restriction which expresses our particular flavour of internet access, tests which only use http/s, and so are totally fine for us to run fully currently, will regress in coverage. We won't be able to detect legit failures any more. In the medium term, if scheduled, we could try to work on a way to express our type of internet access and then have packages use that*, but here I'd like to limit the discussion to solutions we can implement now, please. Would appreciate opinions. Cheers, -- Iain Lane [ [email protected] ] Debian Developer [ [email protected] ] Ubuntu Developer [ [email protected] ] * To avoid restriction proliferation, I am thinking that parameterised restrictions like "needs-internet=http-https-only" might be a neat way to solve this. Just an idea right now, not discussed with autopkgtest maintainers.
signature.asc
Description: PGP signature
-- Ubuntu-release mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
