Hello, everyone.

TL;DR: I'm looking for some help to get pull-request CI running
distributed, i.e. on multiple hosts.

Quick history

The repo-mirror-ci project pretty much started as a simple cronjob
running on hardware contributed to the purpose by Todd Goodman
(considering how much this hardware has done for Gentoo, I'd suggest
giving him a honorary citizenship of Gentoo or something like
that ;-)). I can't say they were terrible but they weren't perfect
either. Due to lack of time, I mostly focused on improving them
moderately and adding new features.

Today, everything is still running on the same hardware. While
the hardware is real good, and I'm doing some effort to improve
the performance of the tools, I think it's quite close to its limits.

Right now, it is running three tasks every 20 minutes: updating repo
mirrors, running pkgcheck on Gentoo and processing one pull request.
However, if those tasks delay beyond those 20 minutes, the poor man's
scheduling based on cron + lockfiles may cause some of the executions
to be skipped.

In other words, the best case is processing 3 pull requests / h. This
is starting to be no longer sufficient, and causing significant delays
during periods of high contributor activity -- while my goal would be to
provide the results ASAP -- 10-20 minutes would be perfect.

The goal and needs

I think the best way forward would be to start distributing tasks
between more systems. While I don't really have time to make whole
repo-mirror-ci distributed, I think it should be feasible to start
splitting pull request processing out of it.

I would use some help to achieve that, esp. wrt distributed task

Can you think of any tools that could help me get the task done easily
and with as little of reinventing the wheel as possible? Right now, I
have just a lot of trivial shell script that checks pull request for
changes, checks them out, runs 'pmaint regen', pkgcheck, publishes all
kinds of results and statuses, and also compares QA results to
determine new failures created by the PR [1].

As I see it, checking for changes and submitting the results should
stay on the current host. However, it should be able to run all
the actual work on slaves and retrieve the results from them. I would
appreciate all the help with implementing that, plus possibly some
degree of control to make it possible to update pkgcheck on slaves

Please let me know if you're interested in helping. Thanks.


Best regards,
Michał Górny

Attachment: pgpU8Dli2RywM.pgp
Description: OpenPGP digital signature

Reply via email to