Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On Tue, Nov 08, 2022 at 09:43:19AM +0100, Agostino Sarubbo wrote: > Hi, > > Whatever outside the arch testing (like tinderbox) is off topic here since it > is a completely different argument. > > To make John Helmert III happy, I just switched to tatt; so my actual > workflow is tatt + nattka and there is nothing more. > > If there are unanswered questions about the arch testing, please let me > know. > > Agostino Great, thanks! signature.asc Description: PGP signature
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On martedì 8 novembre 2022 14:26:18 CET Michał Górny wrote: > If the code was > public, I could try figuring it out and perhaps even fixing it. Stable requests are handled by many people. o, since your requests were ignored by all members and sam said that him, arthurzam, jsmolic are using tattoo, my guess is that you have already everything in place. Quoting the relevant part from sam's email: Agostino
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On Mon, 2022-11-07 at 19:23 -0500, Rich Freeman wrote: > On Mon, Nov 7, 2022 at 6:16 PM Sam James wrote: > > > > > On 7 Nov 2022, at 06:07, Oskari Pirhonen wrote: > > > > > > On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote: > > > > I would be in favour of stepping up the social contract and actually > > > > prohibiting this kind of things, we had that before too, the nattka you > > > > mgorny wrote is replacement for old bugzilla bot that was ... > > > > closedsource and perished, though nattka now have way more features than > > > > the old thing ever had. > > > > > > As a user, I think it would be really cool if there was a requirement > > > that all infra and infra-adjacent stuff was free software. > > > > > > I feel like I've read that Debian already has something like this. While > > > doing some quick searches I didn't find a full-on requirement, but all > > > their infra bits I did find were powered by free software. The most > > > relevant ones being buildd [1] and debci [2]. Additionally, the debci > > > docs has inctructions on reproducing tests yourself [3] which is a nice > > > extra IMO. > > > > Gentoo has > > https://www.gentoo.org/get-started/philosophy/social-contract.html. > > [...] > > I think the key is something that was brought up earlier in the > thread: is this causing problems? We're talking about handling arch testing and not tinderboxing in general but yes, this is causing problems. If someone's running automation that takes care of a significant portion of arch testing, it effectively leads to monopolized arch testing. Other arch testers don't need to do anything, so they eventually stop paying attention and everyone assumes "X will take care of it anyway". Now, the first problem is the bus factor. If X stops doing arch testing, requests pile up. It takes time before others resume their work. If the software used to do the automation was proprietary, others have to start over. Of course, this is better now that we have an alternative. The second problem is that we don't really know *how* things are processed. As I've said, it happened to me before that stablereqs were ignored for months. My guess is that the automation couldn't figure out how to process them, so it skipped them, silently. I still don't know what was the problem, or how to avoid it in the future. If the code was public, I could try figuring it out and perhaps even fixing it. -- Best regards, Michał Górny
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
Hi, Whatever outside the arch testing (like tinderbox) is off topic here since it is a completely different argument. To make John Helmert III happy, I just switched to tatt; so my actual workflow is tatt + nattka and there is nothing more. If there are unanswered questions about the arch testing, please let me know. Agostino
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On Mon, Nov 07, 2022 at 08:26:15 +0200, Joonas Niilola wrote: > I _believe_ ago's tinderbox isn't being paid by the GF _anymore_ due to > this reason, but he keeps it running with his own expenses. I don't mind > this as long as the results are desirable and not phony. I still see a > lot of value in most of ago's work. > Don't get me wrong, I really appreaciate the work that all of you do to keep my favorite distro running. And maintaining the level of quality that I've grown to expect from Gentoo over the years too :) > It is unfortunate we don't get to see the engine behind and copy it, > since I'd be really interested in using his automated bug search / > report tool. Even tattoo (https://github.com/arthurzam/tattoo) lacks > that at the moment. > Perhaps he would be open to the idea of releasing some parts of it? (pun intended) - Oskari signature.asc Description: PGP signature
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On 8.11.2022 2.23, Rich Freeman wrote: > > Of course if somebody wants to contribute to 100% FOSS tinderbox > efforts that would be even better. Perhaps if our 100% FOSS tinderbox > efforts addressed our needs very well, then nobody would want to > bother with the proprietary reports, or generating them. IMO it would > be better to create the FOSS solution before abandoning the > proprietary one. Doing otherwise is basically burning bridges - it > can be motivating in a sense but not really ideal. I'd love to have a > 100% FOSS solution around all of this, but I appreciate what has been > created and can hardly criticize volunteers for failing to make it > happen, especially since I haven't contributed to that myself. > https://gitweb.gentoo.org/proj/tinderbox-cluster.git/ https://wiki.gentoo.org/wiki/Project:Tinderbox-cluster Hopefully soon. -- juippis OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On 06/11/2022 10.34, Sam James wrote: > > ... > > That had two parts: > 1. https://github.com/projg2/nattka/issues/72 & > https://github.com/projg2/nattka/pull/73 (done) > 2. https://github.com/arthurzam/tattoo/issues/1 (not done) I was waiting for nattka-0.4 (which returns the field value) and was hoping to wait until it becomes stable, so it will be able to upgrade nattka on all devboxes easily and just use new API. Seeing this conversation, made me understand that it is desired to do it ASAP, so I added it now with a little ugly code around (import guards to check that nattka supports the new field). So currently tattoo skips all bugs marked with Manual testing. I also plan to create a small dashboard showing special cases of Arch Testing bugs, such as: 1. Bugs with only one arch remaining 2. Bugs blocked by blocker bug 3. Bugs without any info and activity (not blocked, untouched, not done) This is in hopes to have a more organized and priority bugs, to mitigate cases when something somewhere failed, and we lose the bug until pinged. My current plan is to have a script that generated a HTML file that I will host on ~dev space, and will have a scheduled process to regenerate the HTML. I'm still planning the solution and how to schedule it, so no promises :) -- Arthur Zamarin arthur...@gentoo.org Gentoo Linux developer (Python, Arch Teams, pkgcore stack, GURU) OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On Mon, Nov 7, 2022 at 7:34 PM John Helmert III wrote: > > On Mon, Nov 07, 2022 at 07:23:33PM -0500, Rich Freeman wrote: > > Proprietary tools do contribute to this since they can > > generate results that are harder to reproduce, but if they are clear > > and accurate and actionable it could still be a net-positive. > > In some cases, yes, this is exactly the problem. This was one of the > bugs reported in the now-deleted issue tracking repository on Github. > This was hinted at earlier in the thread. My goal wasn't to say whether it was or was not an issue, but bring the focus more on the tangible impact, as I think that will probably help to make this less about philosophy and more about impact. I think that is more likely to create action (whether that is a policy change or improvement to the tooling). -- Rich
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On Mon, Nov 07, 2022 at 07:23:33PM -0500, Rich Freeman wrote: > On Mon, Nov 7, 2022 at 6:16 PM Sam James wrote: > > > > > On 7 Nov 2022, at 06:07, Oskari Pirhonen wrote: > > > > > > On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote: > > >> I would be in favour of stepping up the social contract and actually > > >> prohibiting this kind of things, we had that before too, the nattka you > > >> mgorny wrote is replacement for old bugzilla bot that was ... > > >> closedsource and perished, though nattka now have way more features than > > >> the old thing ever had. > > > > > > As a user, I think it would be really cool if there was a requirement > > > that all infra and infra-adjacent stuff was free software. > > > > > > I feel like I've read that Debian already has something like this. While > > > doing some quick searches I didn't find a full-on requirement, but all > > > their infra bits I did find were powered by free software. The most > > > relevant ones being buildd [1] and debci [2]. Additionally, the debci > > > docs has inctructions on reproducing tests yourself [3] which is a nice > > > extra IMO. > > > > Gentoo has > > https://www.gentoo.org/get-started/philosophy/social-contract.html. > > I feel like something like a dev-run tinderbox is a bit out of the > scope of that. > > Suppose I file a bug against a package, pointing out some issue in it. > How do you know I didn't use some proprietary static code analysis > tool to discover that error? Does it even really matter? The bug > speaks for itself. It is like worrying about whether somebody who > filed a bug was running Windows or another proprietary OS or browser > on their desktop. > > Well, a tinderbox is just an automated process for doing just that. > We don't require any dev to use a proprietary tinderbox before > committing. It is something that individual devs choose to use for > themselves, automating the testing workflow and possibly the > submission of bugs. > > I think the key is something that was brought up earlier in the > thread: is this causing problems? If somebody is running some tool > against the repository and automatically filing bugs, and those bugs > are not useful/actionable and waste the time of volunteers, then that > is a problem. Proprietary tools do contribute to this since they can > generate results that are harder to reproduce, but if they are clear > and accurate and actionable it could still be a net-positive. In some cases, yes, this is exactly the problem. This was one of the bugs reported in the now-deleted issue tracking repository on Github. > Of course if somebody wants to contribute to 100% FOSS tinderbox > efforts that would be even better. Perhaps if our 100% FOSS tinderbox > efforts addressed our needs very well, then nobody would want to > bother with the proprietary reports, or generating them. IMO it would > be better to create the FOSS solution before abandoning the > proprietary one. Doing otherwise is basically burning bridges - it > can be motivating in a sense but not really ideal. I'd love to have a > 100% FOSS solution around all of this, but I appreciate what has been > created and can hardly criticize volunteers for failing to make it > happen, especially since I haven't contributed to that myself. > > -- > Rich > signature.asc Description: PGP signature
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
> On 8 Nov 2022, at 00:23, Rich Freeman wrote: > > On Mon, Nov 7, 2022 at 6:16 PM Sam James wrote: >> >>> On 7 Nov 2022, at 06:07, Oskari Pirhonen wrote: >>> >>> On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote: I would be in favour of stepping up the social contract and actually prohibiting this kind of things, we had that before too, the nattka you mgorny wrote is replacement for old bugzilla bot that was ... closedsource and perished, though nattka now have way more features than the old thing ever had. >>> >>> As a user, I think it would be really cool if there was a requirement >>> that all infra and infra-adjacent stuff was free software. >>> >>> I feel like I've read that Debian already has something like this. While >>> doing some quick searches I didn't find a full-on requirement, but all >>> their infra bits I did find were powered by free software. The most >>> relevant ones being buildd [1] and debci [2]. Additionally, the debci >>> docs has inctructions on reproducing tests yourself [3] which is a nice >>> extra IMO. >> >> Gentoo has >> https://www.gentoo.org/get-started/philosophy/social-contract.html. > > I feel like something like a dev-run tinderbox is a bit out of the > scope of that. I intentionally didn't comment on the scope for now, but I'm glad you did. > > Suppose I file a bug against a package, pointing out some issue in it. > How do you know I didn't use some proprietary static code analysis > tool to discover that error? Does it even really matter? The bug > speaks for itself. It is like worrying about whether somebody who > filed a bug was running Windows or another proprietary OS or browser > on their desktop. > It matters if someone can't then reproduce the bug which happens somewhat often here. signature.asc Description: Message signed with OpenPGP
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On Mon, Nov 7, 2022 at 6:16 PM Sam James wrote: > > > On 7 Nov 2022, at 06:07, Oskari Pirhonen wrote: > > > > On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote: > >> I would be in favour of stepping up the social contract and actually > >> prohibiting this kind of things, we had that before too, the nattka you > >> mgorny wrote is replacement for old bugzilla bot that was ... > >> closedsource and perished, though nattka now have way more features than > >> the old thing ever had. > > > > As a user, I think it would be really cool if there was a requirement > > that all infra and infra-adjacent stuff was free software. > > > > I feel like I've read that Debian already has something like this. While > > doing some quick searches I didn't find a full-on requirement, but all > > their infra bits I did find were powered by free software. The most > > relevant ones being buildd [1] and debci [2]. Additionally, the debci > > docs has inctructions on reproducing tests yourself [3] which is a nice > > extra IMO. > > Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html. I feel like something like a dev-run tinderbox is a bit out of the scope of that. Suppose I file a bug against a package, pointing out some issue in it. How do you know I didn't use some proprietary static code analysis tool to discover that error? Does it even really matter? The bug speaks for itself. It is like worrying about whether somebody who filed a bug was running Windows or another proprietary OS or browser on their desktop. Well, a tinderbox is just an automated process for doing just that. We don't require any dev to use a proprietary tinderbox before committing. It is something that individual devs choose to use for themselves, automating the testing workflow and possibly the submission of bugs. I think the key is something that was brought up earlier in the thread: is this causing problems? If somebody is running some tool against the repository and automatically filing bugs, and those bugs are not useful/actionable and waste the time of volunteers, then that is a problem. Proprietary tools do contribute to this since they can generate results that are harder to reproduce, but if they are clear and accurate and actionable it could still be a net-positive. Of course if somebody wants to contribute to 100% FOSS tinderbox efforts that would be even better. Perhaps if our 100% FOSS tinderbox efforts addressed our needs very well, then nobody would want to bother with the proprietary reports, or generating them. IMO it would be better to create the FOSS solution before abandoning the proprietary one. Doing otherwise is basically burning bridges - it can be motivating in a sense but not really ideal. I'd love to have a 100% FOSS solution around all of this, but I appreciate what has been created and can hardly criticize volunteers for failing to make it happen, especially since I haven't contributed to that myself. -- Rich
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
> On 7 Nov 2022, at 06:07, Oskari Pirhonen wrote: > > On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote: >> I would be in favour of stepping up the social contract and actually >> prohibiting this kind of things, we had that before too, the nattka you >> mgorny wrote is replacement for old bugzilla bot that was ... >> closedsource and perished, though nattka now have way more features than >> the old thing ever had. > > As a user, I think it would be really cool if there was a requirement > that all infra and infra-adjacent stuff was free software. > > I feel like I've read that Debian already has something like this. While > doing some quick searches I didn't find a full-on requirement, but all > their infra bits I did find were powered by free software. The most > relevant ones being buildd [1] and debci [2]. Additionally, the debci > docs has inctructions on reproducing tests yourself [3] which is a nice > extra IMO. Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html. Best, sam signature.asc Description: Message signed with OpenPGP
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On Mon, Nov 07, 2022 at 08:26:15AM +0200, Joonas Niilola wrote: > On 7.11.2022 8.07, Oskari Pirhonen wrote: > > On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote: > >> I would be in favour of stepping up the social contract and actually > >> prohibiting this kind of things, we had that before too, the nattka you > >> mgorny wrote is replacement for old bugzilla bot that was ... > >> closedsource and perished, though nattka now have way more features than > >> the old thing ever had. > > > > As a user, I think it would be really cool if there was a requirement > > that all infra and infra-adjacent stuff was free software. > > > > I feel like I've read that Debian already has something like this. While > > doing some quick searches I didn't find a full-on requirement, but all > > their infra bits I did find were powered by free software. The most > > relevant ones being buildd [1] and debci [2]. Additionally, the debci > > docs has inctructions on reproducing tests yourself [3] which is a nice > > extra IMO. > > > > [1] https://buildd.debian.org > > [2] https://ci.debian.net > > [3] https://ci.debian.net/doc/file.MAINTAINERS.html > > > > - Oskari > > I _believe_ ago's tinderbox isn't being paid by the GF _anymore_ due to > this reason, but he keeps it running with his own expenses. I don't mind > this as long as the results are desirable and not phony. I still see a > lot of value in most of ago's work. He's not using infra's AWS resources anymore, but he's still using the devbox.amd64.dev.gentoo.org VM. > It is unfortunate we don't get to see the engine behind and copy it, > since I'd be really interested in using his automated bug search / > report tool. Even tattoo (https://github.com/arthurzam/tattoo) lacks > that at the moment. > > -- juippis signature.asc Description: PGP signature
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On 7.11.2022 8.07, Oskari Pirhonen wrote: > On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote: >> I would be in favour of stepping up the social contract and actually >> prohibiting this kind of things, we had that before too, the nattka you >> mgorny wrote is replacement for old bugzilla bot that was ... >> closedsource and perished, though nattka now have way more features than >> the old thing ever had. > > As a user, I think it would be really cool if there was a requirement > that all infra and infra-adjacent stuff was free software. > > I feel like I've read that Debian already has something like this. While > doing some quick searches I didn't find a full-on requirement, but all > their infra bits I did find were powered by free software. The most > relevant ones being buildd [1] and debci [2]. Additionally, the debci > docs has inctructions on reproducing tests yourself [3] which is a nice > extra IMO. > > [1] https://buildd.debian.org > [2] https://ci.debian.net > [3] https://ci.debian.net/doc/file.MAINTAINERS.html > > - Oskari I _believe_ ago's tinderbox isn't being paid by the GF _anymore_ due to this reason, but he keeps it running with his own expenses. I don't mind this as long as the results are desirable and not phony. I still see a lot of value in most of ago's work. It is unfortunate we don't get to see the engine behind and copy it, since I'd be really interested in using his automated bug search / report tool. Even tattoo (https://github.com/arthurzam/tattoo) lacks that at the moment. -- juippis OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote: > I would be in favour of stepping up the social contract and actually > prohibiting this kind of things, we had that before too, the nattka you > mgorny wrote is replacement for old bugzilla bot that was ... > closedsource and perished, though nattka now have way more features than > the old thing ever had. As a user, I think it would be really cool if there was a requirement that all infra and infra-adjacent stuff was free software. I feel like I've read that Debian already has something like this. While doing some quick searches I didn't find a full-on requirement, but all their infra bits I did find were powered by free software. The most relevant ones being buildd [1] and debci [2]. Additionally, the debci docs has inctructions on reproducing tests yourself [3] which is a nice extra IMO. [1] https://buildd.debian.org [2] https://ci.debian.net [3] https://ci.debian.net/doc/file.MAINTAINERS.html - Oskari signature.asc Description: PGP signature
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On Sun, Nov 06, 2022 at 08:03:16PM +0100, Agostino Sarubbo wrote: > On domenica 6 novembre 2022 14:27:40 CET John Helmert III wrote: > > As far as I can tell, there's ONE person relying completely on a > > proprietary arch testing system. > > > > Ago, could you comment on this? What's blocking you from open sourcing > > your software? > > Hi, > > I already answered in the previous post: > > "I still use getatoms.py to fetch 'doable' stablereqs (it is on my todo > to switch to nattka). And I have a script the **simply** does emerge over the > list of > the packages. There is nothing obscure in it." > > I'm working in arch testing since 2009. In the past I relied on scripts done > by someone else > and every time there was an issue I got no response. And so you force that frustration on everyone else? Why? > At a certain point I decided to make my own script in language I know so I > can edit it when > is needed. None of this blocks you from open sourcing it. Is your reason for not open-sourcing your automation really that "There is nothing obscure in it"? You also ignored my other question: "I'll also point out that you removed the Github repository that you used to tell people to report issues with your CI at, while there were several outstanding issues. Why?" > Since few years we allow self stabilization from maintainer. Do we know how > and with > what they test? No because it is not required. > The requirement for test is that the package you are testing works as > expected. > > https://wiki.gentoo.org/wiki/Project:AMD64_Arch_Testers#Arch_tester.27s_policy[1] > > Agostino > > > [1] > https://wiki.gentoo.org/wiki/Project:AMD64_Arch_Testers#Arch_tester.27s_policy signature.asc Description: PGP signature
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On domenica 6 novembre 2022 14:27:40 CET John Helmert III wrote: > As far as I can tell, there's ONE person relying completely on a > proprietary arch testing system. > > Ago, could you comment on this? What's blocking you from open sourcing > your software? Hi, I already answered in the previous post: "I still use getatoms.py to fetch 'doable' stablereqs (it is on my todo to switch to nattka). And I have a script the **simply** does emerge over the list of the packages. There is nothing obscure in it." I'm working in arch testing since 2009. In the past I relied on scripts done by someone else and every time there was an issue I got no response. At a certain point I decided to make my own script in language I know so I can edit it when is needed. Since few years we allow self stabilization from maintainer. Do we know how and with what they test? No because it is not required. The requirement for test is that the package you are testing works as expected. https://wiki.gentoo.org/wiki/Project:AMD64_Arch_Testers#Arch_tester.27s_policy[1] Agostino [1] https://wiki.gentoo.org/wiki/Project:AMD64_Arch_Testers#Arch_tester.27s_policy
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On domenica 6 novembre 2022 09:15:40 CET Michał Górny wrote: > On top of that, it seems that most of it still relies on proprietary > software and we have no clue how *exactly* it works, and it's really, > really hard to get a straight answer. I'm speaking for myself. I still use getatoms.py to fetch 'doable' stablereqs (it is on my todo to switch to nattka). And I have a script the *simply* does emerge over the list of the packages. There is nothing obscure in it. > So, my questions are: > > 1. Is "runtime testing required" field being respected? Obviously not > every package can be (sufficiently) tested via FEATURES=test, so we've > added that fields. However, if arch testers just ignore it and push > things stable based on pure build testing... sam already provided the right answer. In addition, when we introduced runtime_testing and package_list fields we requested support in pybugz: https://github.com/williamh/pybugz/issues/105[1] There is no trace (into the github ticket) about runtime testing field because I discussed/ requested over irc. > 2. How are kernels being tested? Given the speed with which new gentoo- > sources stablereqs are handled, I really feel like "arch testing" there > means "checking if sources install", and have little to do with working > kernels. For amd64, I boot into the new kernel to verify that at least it boots. For other 'exotic' arches, since there is a lack of hardware, the rule was to verify that at least it builds (install if we want to use the right word). If you think that build only is not appropriate, I can skip kernel stablereqs. > 3. How does the automation handle packages that aren't trivially > installable? I recall that in the past stablereqs were stalled for > months without a single comment because automation couldn't figure out > how to proceed, and nobody bothered reporting a problem. I skip them from automation and from time to time I handle it manually. Agostino [1] https://github.com/williamh/pybugz/issues/105
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
On Sun, Nov 06, 2022 at 09:15:40AM +0100, Michał Górny wrote: > Hi, everyone. > > Arch testing's relying on automation a lot these days. Not saying > that's bad, if it improves the state of affairs. However, I have some > concerns, based on what I've seen lately. > > On top of that, it seems that most of it still relies on proprietary > software and we have no clue how *exactly* it works, and it's really, > really hard to get a straight answer. As far as I can tell, there's ONE person relying completely on a proprietary arch testing system. Ago, could you comment on this? What's blocking you from open sourcing your software? I'll also point out that you removed the Github repository that you used to tell people to report issues with your CI at, while there were several outstanding issues. Why? I have two (of three) issue titles in my browser history, but as I recall you never touched them: 2. Release source code 3. More information on the bug [1] https://github.com/asarubbo/ci > So, my questions are: > > 1. Is "runtime testing required" field being respected? Obviously not > every package can be (sufficiently) tested via FEATURES=test, so we've > added that fields. However, if arch testers just ignore it and push > things stable based on pure build testing... > > 2. How are kernels being tested? Given the speed with which new gentoo- > sources stablereqs are handled, I really feel like "arch testing" there > means "checking if sources install", and have little to do with working > kernels. > > 3. How does the automation handle packages that aren't trivially > installable? I recall that in the past stablereqs were stalled for > months without a single comment because automation couldn't figure out > how to proceed, and nobody bothered reporting a problem. > > -- > Best regards, > Michał Górny > > signature.asc Description: PGP signature
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
Hi, On 06/11/2022 09.15, Michał Górny wrote: On top of that, it seems that most of it still relies on proprietary software and we have no clue how*exactly* it works, and it's really, really hard to get a straight answer. I never understood how it become socially acceptable in open source project that Gentoo is that one of the important automation that does tinderboxing is closed source and gate kept, meaning no one can on their own reproduce the failures, which is utterly annoying when your normal testing just works, yet their testing methodology is a secret. I would be in favour of stepping up the social contract and actually prohibiting this kind of things, we had that before too, the nattka you mgorny wrote is replacement for old bugzilla bot that was ... closedsource and perished, though nattka now have way more features than the old thing ever had. -- Piotr.
Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
> On 6 Nov 2022, at 08:15, Michał Górny wrote: > > Hi, everyone. > > Arch testing's relying on automation a lot these days. Not saying > that's bad, if it improves the state of affairs. However, I have some > concerns, based on what I've seen lately. Thanks for starting this discussion, I think others have felt this way too. > > On top of that, it seems that most of it still relies on proprietary > software and we have no clue how *exactly* it works, and it's really, > really hard to get a straight answer. > arthurzam, jsmolic, and I are using https://github.com/arthurzam/tattoo. > So, my questions are: > > 1. Is "runtime testing required" field being respected? Obviously not > every package can be (sufficiently) tested via FEATURES=test, so we've > added that fields. However, if arch testers just ignore it and push > things stable based on pure build testing... Not right now. We discussed it on #gentoo-dev maybe 2 months ago or so but concluded we needed nattka support to fix up our automation. That had two parts: 1. https://github.com/projg2/nattka/issues/72 & https://github.com/projg2/nattka/pull/73 (done) 2. https://github.com/arthurzam/tattoo/issues/1 (not done) > > 2. How are kernels being tested? Given the speed with which new gentoo- > sources stablereqs are handled, I really feel like "arch testing" there > means "checking if sources install", and have little to do with working > kernels. > I usually blacklist gentoo-sources because I can't actually test it, or if I do stable it, I've tried to run them for real. For gentoo-kernel, I run src_test. > 3. How does the automation handle packages that aren't trivially > installable? I recall that in the past stablereqs were stalled for > months without a single comment because automation couldn't figure out > how to proceed, and nobody bothered reporting a problem. It doesn't, really. I think at the very least we need to try do world upgrades after adding the package list to package.accept_keywords, as stuff like LLVM and GNOME bugs won't work without it (blockers etc). Best, sam signature.asc Description: Message signed with OpenPGP
