Hi Michal, > On 7 Nov 2022, at 07:50, Michal Orzel <michal.or...@amd.com> wrote: > > Hi Bertrand and Stefano, > > On 31/10/2022 16:00, Bertrand Marquis wrote: >> >> >> Hi Michal, >> >>> On 31 Oct 2022, at 14:39, Michal Orzel <michal.or...@amd.com> wrote: >>> >>> Hi Bertrand, >>> >>> On 31/10/2022 15:00, Bertrand Marquis wrote: >>>> >>>> >>>> This patch series is a first attempt to check if we could use Yocto in >>>> gitlab ci to build and run xen on qemu for arm, arm64 and x86. >>>> >>>> The first patch is creating a container with all elements required to >>>> build Yocto, a checkout of the yocto layers required and an helper >>>> script to build and run xen on qemu with yocto. >>>> >>>> The second patch is creating containers with a first build of yocto done >>>> so that susbsequent build with those containers would only rebuild what >>>> was changed and take the rest from the cache. >>>> >>>> The third patch is adding a way to easily clean locally created >>>> containers. >>>> >>>> This is is mainly for discussion and sharing as there are still some >>>> issues/problem to solve: >>>> - building the qemu* containers can take several hours depending on the >>>> network bandwith and computing power of the machine where those are >>>> created >>> This is not really an issue as the build of the containers occurs on the >>> local >>> machines before pushing them to registry. Also, building the containers >>> will only be required for new Yocto releases. >>> >>>> - produced containers containing the cache have a size between 8 and >>>> 12GB depending on the architecture. We might need to store the build >>>> cache somewhere else to reduce the size. If we choose to have one >>>> single image, the needed size is around 20GB and we need up to 40GB >>>> during the build, which is why I splitted them. >>>> - during the build and run, we use a bit more then 20GB of disk which is >>>> over the allowed size in gitlab >>> As we could see during v2 testing, we do not have any space restrictions >>> on the Xen GitLab and I think we already decided to have the Yocto >>> integrated into our CI. >> >> Right, I should have modified this chapter to be coherent with your latest >> tests. >> Sorry for that. >> >>> >>> I will do some testing and get back to you with results + review. > I did some testing and here are the results: > > In the current form this series will fail when running CI because the Yocto > containers > are based on "From ubuntu:22.04" (there is no platform prefix), which means > that the containers > are built for the host architecture (in my case and in 99% of the cases of > the local build it will > be x86). In Gitlab we have 2 runners (arm64 and x86_64). This means that all > the test jobs would need > to specify x86_64 as a tag when keeping the current behavior. > After I built all the containers on my x86 machine, I pushed them to registry > and the pipeline was successful: > https://gitlab.com/xen-project/people/morzel/xen-orzelmichal/-/pipelines/686853939 > > Here is the diff on patch no. 3 to make the series work (using x86 tag and > small improvement to include needs: []): > ``` > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml > index 5c620fefce59..52cccec6f904 100644 > --- a/automation/gitlab-ci/test.yaml > +++ b/automation/gitlab-ci/test.yaml > @@ -65,6 +65,9 @@ > paths: > - 'logs/*' > when: always > + needs: [] > + tags: > + - x86_64 > > # Test jobs > build-each-commit-gcc: > @@ -206,19 +209,13 @@ yocto-qemuarm64: > extends: .yocto-test > variables: > YOCTO_BOARD: qemuarm64 > - tags: > - - arm64 > > yocto-qemuarm: > extends: .yocto-test > variables: > YOCTO_BOARD: qemuarm > - tags: > - - arm32 > > yocto-qemux86-64: > extends: .yocto-test > variables: > YOCTO_BOARD: qemux86-64 > - tags: > - - x86_64 > ``` > > Now, the logical way would be to build x86 yocto container for x86, arm64 for > arm64 and arm32 on arm64 or x86. > I tried building the container qemuarm64 specifying target arm64 on x86. > After 15h, only 70% of the Yocto build > was completed and there was an error with glibc (the local build of the > container for the host arch takes on my machine max 2h). > This enormous amount of time is due to the qemu docker emulation that happens > behind the scenes (I checked on 2 different machines). > > So we have 3 solutions: > 1) Build and run these containers for/on x86_64: > - local users can build the containers on local machines that are almost > always x86 based, in short period of time, > - "everyone" can build/push the containers once there is a new Yocto release > - slightly slower CI build time > 2) Build and run these containers for specific architectures: > - almost no go for local users using x86 machine (unless using more than 16 > threads (which I used) and willing to wait 2 days for the build) > - faster CI build time (arm64 runner is faster than x86 one) > - someone with arm64 based machine (not that common) would have to build and > push the containers > 3) Try to use CI to build and push the containers to registry > - it could be possible but what about local users
I might have a solution 4. The containers we pushed to gitlab are publicly accessible ? Shouldn’t we push both versions there and then document how someone can retrieve and use those containers instead of building them ? This way we will do the push but a normal user would just retrieve the containers and use them so building time would have less impact and we could build both flavours of the containers. > > Regardless of what we chose, we need to keep in mind that the biggest > advantage to the Yocto build/run is that > it allows/should allow local users to perform basic testing for all the Xen > supported architectures. This is because > everything happens in one place with one command. Agree. Bertrand > > ~Michal
