On Fri, Nov 27, 2020 at 2:41 PM Philippe Mathieu-Daudé <phi...@redhat.com> wrote: > > We lately realized that the Avocado framework was not designed > to be regularly run on CI environments. Therefore, as of 5.2 > we deprecate the gitlab-ci jobs using Avocado. To not disrupt > current users, it is possible to keep the current behavior by > setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable > (see [*]). > From now on, using these jobs (or adding new tests to them) > is strongly discouraged.
When you say, "Avocado framework was not designed to be regularly run on CI environments", I feel your pain. Avocado is a really nice test framework, and I agree with you that running it locally is a little easier than running inside a CI environment. Debugging a job failure in the CI is not user-friendly; finding the command to reproduce a job failure locally is not user-friendly. I understand why you would like to remove the CI's acceptance tests, but I think your proposal is missing some arguments and some planning. If I read correctly, we share the same view that the CI and the software tests are two different things. Here you are proposing that we temporarily remove the CI's acceptance tests because it is not user-friendly to the devs. This does not mean the tests will be lost. It will be possible to run them locally or in the CI using the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable. > > Tests based on Avocado will be ported to new job schemes during > the next releases, with better documentation and templates. I understand you intend to make a more reliable and stable CI. Some wording on why the new job scheme will be better than what we have now and some planning on enabling the acceptance tests again in the CI may help evaluate if it is feasible or just the same as what we have today. It would be nice to hear from other subsystem maintainers their pain points about using the CI and how we can improve it. I hear you that Avocado needs to improve its interface to be more user friendly. As an Avocado developer, I would also like to hear from others where we can improve Avocado to make it less painful for the QEMU developers'.
