On Mon, Nov 30, 2020 at 10:03:35AM +0100, Thomas Huth wrote: > On 27/11/2020 19.46, Philippe Mathieu-Daudé wrote: > > On 11/27/20 7:29 PM, Thomas Huth wrote: > >> On 27/11/2020 18.57, Philippe Mathieu-Daudé wrote: > >>> On 11/27/20 6:47 PM, Thomas Huth wrote: > >>>> On 27/11/2020 18.41, Philippe Mathieu-Daudé wrote: > >>>>> We lately realized that the Avocado framework was not designed > >>>>> to be regularly run on CI environments. Therefore, as of 5.2 > >>>>> we deprecate the gitlab-ci jobs using Avocado. To not disrupt > >>>>> current users, it is possible to keep the current behavior by > >>>>> setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable > >>>>> (see [*]). > >>>>> From now on, using these jobs (or adding new tests to them) > >>>>> is strongly discouraged. > >>>>> > >>>>> Tests based on Avocado will be ported to new job schemes during > >>>>> the next releases, with better documentation and templates. > >>>> > >>>> Why should we disable the jobs by default as long as there is no > >>>> replacement > >>>> available yet? > >>> > >>> Why keep it enabled if it is failing randomly > >> > >> We can still disable single jobs if they are not stable, but that's no > >> reason to disable all of them by default, is it? > >> > >>> if images hardcoded > >>> in tests are being removed from public servers, etc...? > >> > >> That's independent from Avocado, you'll always have that problem if you > >> want > >> to test with external images, unless you mirror them into a repository on > >> the same server (ie. gitlab), which, however, might not always be > >> possible... > >> > >>> They are not disabled, they are still runnable manually or setting > >>> QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE... > >> > >> And who do you think is going to set that variable? Hardly anybody, I > >> guess. > > > > Does that mean nobody cares about these tests? > > It's like with all the other tests: Most of the people do not really care > about them (if they are not the author of a test) unless the test fails > during "make check" / the gating CI of Peter. So IMHO the right way to go is > to finally get these in the gating CI, otherwise, if you now even disable > them in the gitlab-CI by default, they will bitrot completely.
That people don't care, and ignore it until Peter hits the failure during merge is a tragedy of the commons in itself. I think we need to set expectations that caring about tests is a key part of every contributor's responsibility, with subsystem maintainers leading by example: https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg04897.html We do need tests to be reliable though when we're treating them as gating. Hiding unreliable tests behind an env variable you have to opt-in to setting is not going to help that. IMHO unreliable tests should be just disabled entirely. If someone genuinely does care about the test then they can fix it and re-enable it at the same time. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
