Re: CI container image interference between staging and staging-7.2
On Wed, Sep 13, 2023, 03:26 Michael Tokarev wrote:
> 13.09.2023 02:07, Stefan Hajnoczi wrote:
> > Hi,
> > TL;DR Michael: Please check that the staging-7.2 branch has Dan's
> > commit e28112d00703abd136e2411d23931f4f891c9244 ("gitlab: stable
> > staging branches publish containers in a separate tag").
> ...
>
> Mea cupla, Stefan. I'm always forgetting about the fact that ci controls
> don't work on older branches in one way or another. Sigh.
>
> The patch(es) you're talking about - I didn't pick them up for 7.2 (which
> was the branch in question this time, which interfered with your testing),
> thinking it would be ok. Yes, after this fiasco (which is the first one
> actually), it looks like I should re-consider doing this.
>
> It needs quite a few changes in there. And one of them is to actually
> look at QEMU_CI={0,1,2} variable when pushing staging-N.M branches. Right
> now - and this is what I forgot this time once again, - I used QEMU_CI=1
> so the job does not auto-start, but forgot that in 7.2 it auto-starts
> regardless of QEMU_CI value.
>
> I don't push staging-N.M branches often, usually doing all the CI on
> a my gitlab repository instead. And when I do push to qemu-project,
> I either intend to skip automatic job run, to run just the tests I'm
> interested in, or push it at a time when no other pipelines are to be
> run (which is easy due to time zone differences).
>
> But actually I'm a bit surprised this issue happened to begin with.
> Maybe something else is missing still. The thing is that after
> Daniel's changes, qemu/staging container tags should be named differently,
> no? Ah. No. Staging didn't change, it was staging-N.M which were
> renamed. Once again, I'm sorry for not thinking well enough about this, -
> after container tags renaming I was kinda sure main staging tags were
> different from old staging-N.M, which is not the case..
>
> Please excuse me for this trouble. Things like these usually takes quite
> some time to figure out.. :( I'll make sure this wont happen again,
> one way or another.
>
No worries!
Stefan
>
Re: CI container image interference between staging and staging-7.2
13.09.2023 02:07, Stefan Hajnoczi wrote:
Hi,
TL;DR Michael: Please check that the staging-7.2 branch has Dan's
commit e28112d00703abd136e2411d23931f4f891c9244 ("gitlab: stable
staging branches publish containers in a separate tag").
...
Mea cupla, Stefan. I'm always forgetting about the fact that ci controls
don't work on older branches in one way or another. Sigh.
The patch(es) you're talking about - I didn't pick them up for 7.2 (which
was the branch in question this time, which interfered with your testing),
thinking it would be ok. Yes, after this fiasco (which is the first one
actually), it looks like I should re-consider doing this.
It needs quite a few changes in there. And one of them is to actually
look at QEMU_CI={0,1,2} variable when pushing staging-N.M branches. Right
now - and this is what I forgot this time once again, - I used QEMU_CI=1
so the job does not auto-start, but forgot that in 7.2 it auto-starts
regardless of QEMU_CI value.
I don't push staging-N.M branches often, usually doing all the CI on
a my gitlab repository instead. And when I do push to qemu-project,
I either intend to skip automatic job run, to run just the tests I'm
interested in, or push it at a time when no other pipelines are to be
run (which is easy due to time zone differences).
But actually I'm a bit surprised this issue happened to begin with.
Maybe something else is missing still. The thing is that after
Daniel's changes, qemu/staging container tags should be named differently,
no? Ah. No. Staging didn't change, it was staging-N.M which were
renamed. Once again, I'm sorry for not thinking well enough about this, -
after container tags renaming I was kinda sure main staging tags were
different from old staging-N.M, which is not the case..
Please excuse me for this trouble. Things like these usually takes quite
some time to figure out.. :( I'll make sure this wont happen again,
one way or another.
Thank!
/mjt
Re: CI container image interference between staging and staging-7.2
On 13/9/23 01:07, Stefan Hajnoczi wrote:
Hi,
TL;DR Michael: Please check that the staging-7.2 branch has Dan's
commit e28112d00703abd136e2411d23931f4f891c9244 ("gitlab: stable
staging branches publish containers in a separate tag").
I couldn't explain a check-cfi-x86_64 failure
(https://gitlab.com/qemu-project/qemu/-/jobs/5072006964), so I reran
build-cfi-x86_64 to see if it has an effect on its dependencies.
To my surprise the rerun of build-cfi-x86_64 failed:
https://gitlab.com/qemu-project/qemu/-/jobs/5072087783
The first run was successful:
https://gitlab.com/qemu-project/qemu/-/jobs/5071532799
Diffing the output shows that the software versions are different. The
successful run has Python 3.11.5 and meson 1.0.1 while the failed run
has Python 3.10.8 and meson 0.63.3.
I think staging and staging-7.2 pipelines are interfering with each
other. My understanding is that build-cfi-x86_64 uses
registry.gitlab.com/qemu-project/qemu/qemu/fedora:latest and that
should be built from fedora:38. Python 3.10.8 is what Fedora 35 uses.
The staging-7.2 branch's fedora.docker file uses fedora:35.
Isn't it clarified by this commit?
commit d4c7a56539d0bdb6fccf60af94d528613cbc7c4b
Author: Daniel P. Berrangé
Date: Thu Jun 8 17:40:14 2023 +0100
gitlab: centralize the container tag name
We use a fixed container tag of 'latest' so that contributors' forks
don't end up with an ever growing number of containers as they work
on throwaway feature branches.
This fixed tag causes problems running CI upstream in stable staging
branches, however, because the stable staging branch will publish
old container content that clashes with that needed by primary
staging branch. This makes it impossible to reliably run CI
pipelines in parallel in upstream for different staging branches.
This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to
change which tag container publishing uses. Initially it can be set
by contributors as a git push option if they want to override the
default use of 'latest' eg
git push gitlab -o ci.variable=QEMU_CONTAINER_TAG=fish
this is useful if contributors need to run pipelines for different
branches concurrently in their forks.
Reviewed-by: Michael Tokarev
Reviewed-by: Richard Henderson
Signed-off-by: Daniel P. Berrangé
Message-Id: <20230608164018.2520330-2-berra...@redhat.com>
Signed-off-by: Thomas Huth
Michael, did you forgot to set QEMU_CONTAINER_TAG? Maybe we can add
a tiny scripts/make-stable-release to not forget?
CI container image interference between staging and staging-7.2
Hi,
TL;DR Michael: Please check that the staging-7.2 branch has Dan's
commit e28112d00703abd136e2411d23931f4f891c9244 ("gitlab: stable
staging branches publish containers in a separate tag").
I couldn't explain a check-cfi-x86_64 failure
(https://gitlab.com/qemu-project/qemu/-/jobs/5072006964), so I reran
build-cfi-x86_64 to see if it has an effect on its dependencies.
To my surprise the rerun of build-cfi-x86_64 failed:
https://gitlab.com/qemu-project/qemu/-/jobs/5072087783
The first run was successful:
https://gitlab.com/qemu-project/qemu/-/jobs/5071532799
Diffing the output shows that the software versions are different. The
successful run has Python 3.11.5 and meson 1.0.1 while the failed run
has Python 3.10.8 and meson 0.63.3.
I think staging and staging-7.2 pipelines are interfering with each
other. My understanding is that build-cfi-x86_64 uses
registry.gitlab.com/qemu-project/qemu/qemu/fedora:latest and that
should be built from fedora:38. Python 3.10.8 is what Fedora 35 uses.
The staging-7.2 branch's fedora.docker file uses fedora:35.
Stefan
