On Tue, May 11, 2021 at 08:48:44AM +0200, Philippe Mathieu-Daudé wrote: > +Stefan/Peter > > Not sure if following up this thread or start a new one, but I got > blocked again from Gitlab, tagged as a crypto miner for running QEMU > CI... > [1] > https://about.gitlab.com/handbook/support/workflows/investigate_blocked_pipeline.html#trends--high-priority-cases > > I pushed 5 different branches to my repository in less than 1h, > kicking 580 jobs [*]. > > I didn't try to stress Gitlab, it was a simple "one time in the month > rebase unmerged branches, push them before respining on the mailing > list". > > I'm considering changing my workflow: > - not push more than 2 branches per hour (I know 3/h works, so choose > a lower number, as we want to add more tests). > - merge multiple branches locally and push the merged result and > bisect / re-push on failure > - run less testing > - do not run testing > > This sounds counter productive and doesn't scale to a community of > contributors asked to use Gitlab. > > So far I don't have better idea than this series. > > Who is interested in sending patches to improve our workflow?
So we have a few scenarios for using the CI 1. Running gating CI before merging to master 2. Subsystem maintainers running CI before sending a PULL req 3. Contributors running CI before sending a patch series Right now we have the same jobs running in all three scenarios. Given the increasing restrictions on usage, we clearly need to cut down in general and also make it so that it is harder to accidentally burn all your available CI allowance. Currently we always run CI whenever pushing to gitlab. This is convenient but in retrospect it is overkill. People often push to gitlab simply as their backup strategy and thus don't need CI run every time. Not all changes require all possible jobs to by run, but it is hard to filter jobs when we're triggering them based on pushes, as the baseline against which file changes are identified is ill-defined. For scenario (1) we need all the jobs run to maximise quality. This is also a case where we're most likely to have custom runners available, so CI allowance is less of a concern. The job count still needs to be reasonable to avoid hitting issues at times when the merges are frequent (just before freeze). For scenario (2) subsys maintainers, we want them to minimize the liklihood that a pull request will fail scenario (1) and require a respin. Running all jobs achieces this but it is likely overkill. eg we have 24 cross compiler builds. If we expect most maintainers will have either x86-64 or aarch64 hardware for their primary dev platform, then the key benefit of cross compilers is getting coverage of - 32-bit - big endian - windows We don't need 24 jobs todo that. We could simply pick armel as the most relevant 32-bit arch and s390x as the most relevant big endian arch, and then the win32/64 platforms. IOW we could potentially only run 4-6 jobs instead of 24, and still get excellant arch coverage. Similarly for native builds we test quite alot of different distros. I think we probably can rationalize that down to just 2 distros, one covering oldest packages (Debian Stretch) and one covering newest (Fedora 34), and a "build everything" config. We have many other jobs that are testing various obscure combinations of configure args. I'd suggest these rarely fail for most pull requests so are overkill. For subsystem maintainers we could potentially get down to just 10-15 jobs if we're ambitious. Leave everything else as manual trigger only. Perhaps set all the jobs to only run on certain branch name patterns. eg perhaps "*-next" filter is common for subsystem maintainer's pending branches ? For general contributors a similarly short set of jobs to subsystem maintainers is viable. Perhaps again just let then use a "-next" branch. If we can enable manual triggers on any other branches that's good. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
