> On 6 Nov 2022, at 08:15, Michał Górny <mgo...@gentoo.org> wrote: > > Hi, everyone. > > Arch testing's relying on automation a lot these days. Not saying > that's bad, if it improves the state of affairs. However, I have some > concerns, based on what I've seen lately.
Thanks for starting this discussion, I think others have felt this way too. > > On top of that, it seems that most of it still relies on proprietary > software and we have no clue how *exactly* it works, and it's really, > really hard to get a straight answer. > arthurzam, jsmolic, and I are using https://github.com/arthurzam/tattoo. > So, my questions are: > > 1. Is "runtime testing required" field being respected? Obviously not > every package can be (sufficiently) tested via FEATURES=test, so we've > added that fields. However, if arch testers just ignore it and push > things stable based on pure build testing... Not right now. We discussed it on #gentoo-dev maybe 2 months ago or so but concluded we needed nattka support to fix up our automation. That had two parts: 1. https://github.com/projg2/nattka/issues/72 & https://github.com/projg2/nattka/pull/73 (done) 2. https://github.com/arthurzam/tattoo/issues/1 (not done) > > 2. How are kernels being tested? Given the speed with which new gentoo- > sources stablereqs are handled, I really feel like "arch testing" there > means "checking if sources install", and have little to do with working > kernels. > I usually blacklist gentoo-sources because I can't actually test it, or if I do stable it, I've tried to run them for real. For gentoo-kernel, I run src_test. > 3. How does the automation handle packages that aren't trivially > installable? I recall that in the past stablereqs were stalled for > months without a single comment because automation couldn't figure out > how to proceed, and nobody bothered reporting a problem. It doesn't, really. I think at the very least we need to try do world upgrades after adding the package list to package.accept_keywords, as stuff like LLVM and GNOME bugs won't work without it (blockers etc). Best, sam
signature.asc
Description: Message signed with OpenPGP
