On Wed, Oct 27, 2021 at 02:43:46PM -0700, Stefano Stabellini wrote:
> On Wed, 27 Oct 2021, Anthony PERARD wrote:
> > > But we do have a severe problem at the moment with external sources: our
> > > "git clones" keep failing during the build on x86. That is definitely
> > > something worth improving (see my other email thread on the subject) and
> > > it is the main problem affecting gitlab-ci at the moment, I keep having
> > > to restart jobs almost daily to get the overall pipeline to "pass".
> > >
> > > If you have any ideas on how to stop fetching things using "git" from
> > > external repositories in gitlab-ci that would be fantastic :-)
> > > The only thing I could think of to "fix it" is moving all external repos
> > > to gitlab repositories mirrors.
> >
> > I don't think that would work, I've seen the initial clone/fetch of a
> > job fail as well, so from gitlab. If we could have a cache of those
> > external resources closer to the runners, that would be better.
>
> You mean like a git repository mirror inside the Rackspace network (the
> provider of the x86 runner), right? Then we would force the git client
> to go to the Rackspace mirror instead of directly to the target using
> "insteadOf".
That would seems the best to me. If we could install Ian's
git-cache-proxy that is used in osstest, that would be good I think.
Having a mirror instead might work too but that would mean figure out
which repo we would need a mirror of.
I did try a different alternative a while back, I tried to use gitlab's
caching capability:
automation: Cache sub-project git tree in build jobs
https://lore.kernel.org/xen-devel/20191219144217.305851-3-anthony.per...@citrix.com/
It mostly works but I'm not sure how useful it is as it seems there is
10 computers that would maintain 10 different caches, and most of them
for a short while.
> Is that what you meant? Doug, do you think it would work?
--
Anthony PERARD
