Hi, On Tue, Mar 24, 2020 at 11:00 AM Martin Grigorov <martin.grigo...@gmail.com> wrote:
> Hi Guillaume, > > On Mon, Mar 23, 2020 at 8:01 PM Guillaume Quintard < > guilla...@varnish-software.com> wrote: > >> Hi Martin, >> >> Thank you for that. >> A few remarks and questions: >> - how much time does the "docker build" step takes? We can possibly speed >> things up by push images to the dockerhub, as they don't need to change >> very often. >> > > Definitely such optimization would be a good thing to do! > At the moment, with 'machine' executor it fetches the base image and then > builds all the Docker layers again and again. > Here are the timings: > 1) Spinning up a VM - around 10secs > 2) prepare env variables - 0secs > 3) checkout code (varnish-cache) - 5secs > 4) activate QEMU - 2secs > 5) build packages > 5.1) x86 deb - 3m 30secs > 5.2) x86 rpm - 2m 50secs > 5.3) aarch64 rpm - 35mins > 5.4) aarch64 deb - 45mins > > >> - any reason why you clone pkg-varnish-cache in each job? The idea was to >> have it cloned once in tar-pkg-tools for consistency and reproducibility, >> which we lose here. >> > > I will extract the common steps once I see it working. This is my first > CircleCI project and I still find my ways in it! > > >> - do we want to change things for the amd64 platforms for the sake of >> consistency? >> > > So far there is nothing specific for amd4 or aarch64, except the base > Docker images. > For example make-deb-packages.sh is reused for both amd64 and aarch64 > builds. Same for -rpm- and now for -apk- (alpine). > > Once I feel the change is almost finished I will open a Pull Request for > more comments! > > Martin > > >> >> -- >> Guillaume Quintard >> >> >> On Mon, Mar 23, 2020 at 6:25 AM Martin Grigorov < >> martin.grigo...@gmail.com> wrote: >> >>> Hi, >>> >>> >>> On Wed, Mar 18, 2020 at 5:31 PM Martin Grigorov < >>> martin.grigo...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> On Thu, Mar 12, 2020 at 4:35 PM Martin Grigorov < >>>> martin.grigo...@gmail.com> wrote: >>>> >>>>> Hi Guillaume, >>>>> >>>>> On Thu, Mar 12, 2020 at 3:23 PM Guillaume Quintard < >>>>> guilla...@varnish-software.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Offering arm64 packages requires a few things: >>>>>> - arm64-compatible code (all good in >>>>>> https://github.com/varnishcache/varnish-cache) >>>>>> - arm64-compatible package framework (all good in >>>>>> https://github.com/varnishcache/pkg-varnish-cache) >>>>>> - infrastructure to build the packages (uhoh, see below) >>>>>> - infrastructure to store and deliver ( >>>>>> https://packagecloud.io/varnishcache) >>>>>> >>>>>> So, everything is in place, expect for the third point. At the >>>>>> moment, there are two concurrent CI implementations: >>>>>> - travis: >>>>>> https://github.com/varnishcache/varnish-cache/blob/master/.travis.yml >>>>>> It's >>>>>> the historical one, and currently only runs compilation+test for OSX >>>>>> >>>>> >>>>> Actually it tests Linux AMD64 and ARM64 too. >>>>> >>>>> >>>>>> - circleci: >>>>>> https://github.com/varnishcache/varnish-cache/blob/master/.circleci/config.yml >>>>>> the >>>>>> new kid on the block, that builds all the packages and distchecks for all >>>>>> the packaged platforms >>>>>> >>>>>> The issue is that cirecleci doesn't support arm64 containers (for >>>>>> now?), so we would need to re-implement the packaging logic in Travis. >>>>>> It's >>>>>> not a big problem, but it's currently not a priority on my side. >>>>>> >>>>>> However, I am totally ready to provide help if someone wants to take >>>>>> that up. The added benefit it that Travis would be able to handle >>>>>> everything and we can retire the circleci experiment >>>>>> >>>>> >>>>> I will take a look in the coming days and ask you if I need help! >>>>> >>>> >>>> I've took a look at the current setup and here is what I've found as >>>> problems and possible solutions: >>>> >>>> 1) Circle CI >>>> 1.1) problem - the 'machine' and 'Docker' executors run on x86_64, so >>>> there is no way to build the packages in a "native" environment >>>> 1.2) possible solutions >>>> 1.2.1) use multiarch cross build >>>> 1.2.2) use 'machine' executor that registers QEMU via >>>> https://hub.docker.com/r/multiarch/qemu-user-static/ and then builds >>>> and runs a custom Docker image that executes a shell script with the build >>>> steps >>>> It will look something like >>>> https://github.com/yukimochi-containers/alpine-vpnserver/blob/69bb0a612c9df3e4ba78064d114751b760f0df9d/.circleci/config.yml#L19-L38 >>>> but >>>> instead of uploading the Docker image as a last step it will run it. >>>> The RPM and DEB build related code from current config.yml will be >>>> extracted into shell scripts which will be copied in the custom Docker >>>> images >>>> >>>> From these two possible ways I have better picture in my head how to do >>>> 1.2.2, but I don't mind going deep in 1.2.1 if this is what you'd prefer. >>>> >>> >>> I've decided to stay with Circle CI and use 'machine' executor with QEMU. >>> >>> The changed config.yml could be seen at >>> https://github.com/martin-g/varnish-cache/tree/feature/aarch64-packages/.circleci >>> and >>> the build at >>> https://app.circleci.com/pipelines/github/martin-g/varnish-cache/71/workflows/3a275d79-62a9-48b4-9aef-1585de1c87c8 >>> The builds on x86 arch take 3-4 mins, but for aarch64 (emulation!) >>> ~40mins >>> For now the jobs just build the .deb & .rpm packages for CentOS 7 and >>> Ubuntu 18.04, both amd64 and aarch64. >>> TODOs: >>> - migrate Alpine >>> >> Build on Alpine aarch64 fails with: ... automake: this behaviour will change in future Automake versions: they will automake: unconditionally cause object files to be placed in the same subdirectory automake: of the corresponding sources. automake: project, to avoid future incompatibilities. parallel-tests: installing 'build-aux/test-driver' lib/libvmod_debug/Makefile.am:12: warning: libvmod_debug_la_LDFLAGS multiply defined in condition TRUE ... lib/libvmod_debug/automake_boilerplate.am:19: ... 'libvmod_debug_la_LDFLAGS' previously defined here lib/libvmod_debug/Makefile.am:9: 'lib/libvmod_debug/ automake_boilerplate.am' included from here + autoconf + CONFIG_SHELL=/bin/sh + export CONFIG_SHELL + ./configure '--prefix=/opt/varnish' '--mandir=/opt/varnish/man' --enable-maintainer-mode --enable-developer-warnings --enable-debugging-symbols --enable-dependency-tracking --with-persistent-storage --quiet configure: WARNING: dot not found - build will fail if svg files are out of date. configure: WARNING: No system jemalloc found, using system malloc configure: error: Could not find backtrace() support Does anyone know a workaround ? I use multiarch/alpine:aarch64-edge as a base Docker image Martin > - store the packages as CircleCI artifacts >>> - anything else that is still missing >>> >>> Adding more architectures would be as easy as adding a new Dockerfile >>> with a base image from the respective type. >>> >>> Martin >>> >>> >>>> 2) Travis CI >>>> 2.1) problems >>>> 2.1.1) generally Travis is slower than Circle! >>>> Althought if we use CircleCI 'machine' executor it will be slower than >>>> the current 'Docker' executor! >>>> 2.1.2) Travis supports only Ubuntu >>>> Current setup at CircleCI uses CentOS 7. >>>> I guess the build steps won't have problems on Ubuntu. >>>> >>>> 3) GitHub Actions >>>> GH Actions does not support ARM64 but it supports self hosted ARM64 >>>> runners >>>> 3.1) The problem is that there is no way to make a self hosted runner >>>> really private. I.e. if someone forks Varnish Cache any commit in the fork >>>> will trigger builds on the arm64 node. There is no way to reserve the >>>> runner only for commits against >>>> https://github.com/varnishcache/varnish-cache >>>> >>>> Do you see other problems or maybe different ways ? >>>> Do you have preferences which way to go ? >>>> >>>> Regards, >>>> Martin >>>> >>>> >>>>> >>>>> Regards, >>>>> Martin >>>>> >>>>> >>>>>> >>>>>> -- >>>>>> Guillaume Quintard >>>>>> _______________________________________________ >>>>>> varnish-dev mailing list >>>>>> varnish-dev@varnish-cache.org >>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >>>>>> >>>>>
_______________________________________________ varnish-dev mailing list varnish-dev@varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
