Hello, Here is the PR: https://github.com/varnishcache/varnish-cache/pull/3263 I will add some more documentation about the new setup. Any feedback is welcome!
Regards, Martin On Wed, Mar 25, 2020 at 9:55 PM Martin Grigorov <[email protected]> wrote: > Hi, > > On Wed, Mar 25, 2020, 20:15 Guillaume Quintard < > [email protected]> wrote: > >> is that script running as root? >> > > Yes. > I also added 'USER root' to its Dockerfile and '-u 0' to 'docker run' > arguments but it still doesn't work. > The x86 build is OK. > It must be something in the base docker image. > I've disabled the Alpine aarch64 job for now. > I'll send a PR tomorrow! > > Regards, > Martin > > >> -- >> Guillaume Quintard >> >> >> On Wed, Mar 25, 2020 at 2:30 AM Martin Grigorov < >> [email protected]> wrote: >> >>> Hi, >>> >>> I've moved 'dist' job to be executed in parallel with 'tar_pkg_tools' >>> and the results from both are shared in the workspace for the actual >>> packing jobs. >>> Now the new error for aarch64-apk job is: >>> >>> abuild: varnish >>> varnish: Updating the sha512sums in APKBUILD... >>> ]0; DEBUG: 4 >>> ]0;abuild: varnish >>> varnish: Building /varnish 6.4.0-r1 (using abuild >>> 3.5.0-r0) started Wed, 25 Mar 2020 09:22:02 +0000 >>> >>> varnish: Checking sanity of /package/APKBUILD... >>> >>> WARNING: varnish: No maintainer >>> >>> varnish: Analyzing dependencies... >>> 0% % >>> ############################################>>> varnish: Installing for >>> build: build-base gcc libc-dev libgcc pcre-dev ncurses-dev libedit-dev >>> py-docutils linux-headers libunwind-dev python py3-sphinx >>> Waiting for repository lock >>> ERROR: Unable to lock database: Bad file descriptor >>> ERROR: Failed to open apk database: Bad file descriptor >>> >>> ERROR: varnish: builddeps failed >>> ]0; >>> varnish: Uninstalling dependencies... >>> Waiting for repository lock >>> ERROR: Unable to lock database: Bad file descriptor >>> ERROR: Failed to open apk database: Bad file descriptor >>> >>> Google suggested to do this: >>> rm -rf /var/cache/apk >>> mkdir /var/cache/apk >>> >>> It fails at 'abuild -r' - >>> https://github.com/martin-g/varnish-cache/blob/b62c357b389c0e1e31e9c001cbffb55090c2e49f/.circleci/make-apk-packages.sh#L61 >>> >>> Any hints ? >>> >>> Martin >>> >>> On Wed, Mar 25, 2020 at 2:39 AM Guillaume Quintard < >>> [email protected]> wrote: >>> >>>> Hi, >>>> >>>> So, you are pointing at the `dist` job, whose sole role is to provide >>>> us with a dist tarball, so we don't need that command line to work for >>>> everyone, just for that specific platform. >>>> >>>> On the other hand, >>>> https://github.com/varnishcache/varnish-cache/blob/4f9d8bed6b24bf9ee900c754f37615fdba1c44db/.circleci/config.yml#L168 >>>> is >>>> closer to what you want, `distcheck` will be call on all platform, and you >>>> can see that it has the `--with-unwind` argument. >>>> -- >>>> Guillaume Quintard >>>> >>>> >>>> On Tue, Mar 24, 2020 at 3:05 PM Martin Grigorov < >>>> [email protected]> wrote: >>>> >>>>> >>>>> >>>>> On Tue, Mar 24, 2020, 17:19 Guillaume Quintard < >>>>> [email protected]> wrote: >>>>> >>>>>> Compare your configure line with what's currently in use (or the >>>>>> apkbuild file), there are a few options (with-unwind, without-jemalloc, >>>>>> etc.) That need to be set >>>>>> >>>>> >>>>> The configure line comes from "./autogen.des": >>>>> https://github.com/varnishcache/varnish-cache/blob/4f9d8bed6b24bf9ee900c754f37615fdba1c44db/autogen.des#L35-L42 >>>>> It is called at: >>>>> >>>>> https://github.com/varnishcache/varnish-cache/blob/4f9d8bed6b24bf9ee900c754f37615fdba1c44db/.circleci/config.yml#L40 >>>>> In my branch at: >>>>> >>>>> https://github.com/martin-g/varnish-cache/blob/4b4626ee9cc366b032a45f27b54d77176125ef03/.circleci/make-apk-packages.sh#L26 >>>>> >>>>> It fails only on aarch64 for Alpine Linux. The x86_64 build for Alpine >>>>> is fine. >>>>> AARCH64 for CentOS 7 and Ubuntu 18.04 are also fine. >>>>> >>>>> Martin >>>>> >>>>> >>>>>> On Tue, Mar 24, 2020, 08:05 Martin Grigorov < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> On Tue, Mar 24, 2020 at 11:00 AM Martin Grigorov < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Guillaume, >>>>>>>> >>>>>>>> On Mon, Mar 23, 2020 at 8:01 PM Guillaume Quintard < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Martin, >>>>>>>>> >>>>>>>>> Thank you for that. >>>>>>>>> A few remarks and questions: >>>>>>>>> - how much time does the "docker build" step takes? We can >>>>>>>>> possibly speed things up by push images to the dockerhub, as they >>>>>>>>> don't >>>>>>>>> need to change very often. >>>>>>>>> >>>>>>>> >>>>>>>> Definitely such optimization would be a good thing to do! >>>>>>>> At the moment, with 'machine' executor it fetches the base image >>>>>>>> and then builds all the Docker layers again and again. >>>>>>>> Here are the timings: >>>>>>>> 1) Spinning up a VM - around 10secs >>>>>>>> 2) prepare env variables - 0secs >>>>>>>> 3) checkout code (varnish-cache) - 5secs >>>>>>>> 4) activate QEMU - 2secs >>>>>>>> 5) build packages >>>>>>>> 5.1) x86 deb - 3m 30secs >>>>>>>> 5.2) x86 rpm - 2m 50secs >>>>>>>> 5.3) aarch64 rpm - 35mins >>>>>>>> 5.4) aarch64 deb - 45mins >>>>>>>> >>>>>>>> >>>>>>>>> - any reason why you clone pkg-varnish-cache in each job? The idea >>>>>>>>> was to have it cloned once in tar-pkg-tools for consistency and >>>>>>>>> reproducibility, which we lose here. >>>>>>>>> >>>>>>>> >>>>>>>> I will extract the common steps once I see it working. This is my >>>>>>>> first CircleCI project and I still find my ways in it! >>>>>>>> >>>>>>>> >>>>>>>>> - do we want to change things for the amd64 platforms for the sake >>>>>>>>> of consistency? >>>>>>>>> >>>>>>>> >>>>>>>> So far there is nothing specific for amd4 or aarch64, except the >>>>>>>> base Docker images. >>>>>>>> For example make-deb-packages.sh is reused for both amd64 and >>>>>>>> aarch64 builds. Same for -rpm- and now for -apk- (alpine). >>>>>>>> >>>>>>>> Once I feel the change is almost finished I will open a Pull >>>>>>>> Request for more comments! >>>>>>>> >>>>>>>> Martin >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Guillaume Quintard >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Mar 23, 2020 at 6:25 AM Martin Grigorov < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Mar 18, 2020 at 5:31 PM Martin Grigorov < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> On Thu, Mar 12, 2020 at 4:35 PM Martin Grigorov < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Guillaume, >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 12, 2020 at 3:23 PM Guillaume Quintard < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Offering arm64 packages requires a few things: >>>>>>>>>>>>> - arm64-compatible code (all good in >>>>>>>>>>>>> https://github.com/varnishcache/varnish-cache) >>>>>>>>>>>>> - arm64-compatible package framework (all good in >>>>>>>>>>>>> https://github.com/varnishcache/pkg-varnish-cache) >>>>>>>>>>>>> - infrastructure to build the packages (uhoh, see below) >>>>>>>>>>>>> - infrastructure to store and deliver ( >>>>>>>>>>>>> https://packagecloud.io/varnishcache) >>>>>>>>>>>>> >>>>>>>>>>>>> So, everything is in place, expect for the third point. At the >>>>>>>>>>>>> moment, there are two concurrent CI implementations: >>>>>>>>>>>>> - travis: >>>>>>>>>>>>> https://github.com/varnishcache/varnish-cache/blob/master/.travis.yml >>>>>>>>>>>>> It's >>>>>>>>>>>>> the historical one, and currently only runs compilation+test for >>>>>>>>>>>>> OSX >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Actually it tests Linux AMD64 and ARM64 too. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> - circleci: >>>>>>>>>>>>> https://github.com/varnishcache/varnish-cache/blob/master/.circleci/config.yml >>>>>>>>>>>>> the >>>>>>>>>>>>> new kid on the block, that builds all the packages and distchecks >>>>>>>>>>>>> for all >>>>>>>>>>>>> the packaged platforms >>>>>>>>>>>>> >>>>>>>>>>>>> The issue is that cirecleci doesn't support arm64 containers >>>>>>>>>>>>> (for now?), so we would need to re-implement the packaging logic >>>>>>>>>>>>> in Travis. >>>>>>>>>>>>> It's not a big problem, but it's currently not a priority on my >>>>>>>>>>>>> side. >>>>>>>>>>>>> >>>>>>>>>>>>> However, I am totally ready to provide help if someone wants >>>>>>>>>>>>> to take that up. The added benefit it that Travis would be able >>>>>>>>>>>>> to handle >>>>>>>>>>>>> everything and we can retire the circleci experiment >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I will take a look in the coming days and ask you if I need >>>>>>>>>>>> help! >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I've took a look at the current setup and here is what I've >>>>>>>>>>> found as problems and possible solutions: >>>>>>>>>>> >>>>>>>>>>> 1) Circle CI >>>>>>>>>>> 1.1) problem - the 'machine' and 'Docker' executors run on >>>>>>>>>>> x86_64, so there is no way to build the packages in a "native" >>>>>>>>>>> environment >>>>>>>>>>> 1.2) possible solutions >>>>>>>>>>> 1.2.1) use multiarch cross build >>>>>>>>>>> 1.2.2) use 'machine' executor that registers QEMU via >>>>>>>>>>> https://hub.docker.com/r/multiarch/qemu-user-static/ and then >>>>>>>>>>> builds and runs a custom Docker image that executes a shell script >>>>>>>>>>> with the >>>>>>>>>>> build steps >>>>>>>>>>> It will look something like >>>>>>>>>>> https://github.com/yukimochi-containers/alpine-vpnserver/blob/69bb0a612c9df3e4ba78064d114751b760f0df9d/.circleci/config.yml#L19-L38 >>>>>>>>>>> but >>>>>>>>>>> instead of uploading the Docker image as a last step it will run it. >>>>>>>>>>> The RPM and DEB build related code from current config.yml will >>>>>>>>>>> be extracted into shell scripts which will be copied in the custom >>>>>>>>>>> Docker >>>>>>>>>>> images >>>>>>>>>>> >>>>>>>>>>> From these two possible ways I have better picture in my head >>>>>>>>>>> how to do 1.2.2, but I don't mind going deep in 1.2.1 if this is >>>>>>>>>>> what you'd >>>>>>>>>>> prefer. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I've decided to stay with Circle CI and use 'machine' executor >>>>>>>>>> with QEMU. >>>>>>>>>> >>>>>>>>>> The changed config.yml could be seen at >>>>>>>>>> https://github.com/martin-g/varnish-cache/tree/feature/aarch64-packages/.circleci >>>>>>>>>> and >>>>>>>>>> the build at >>>>>>>>>> https://app.circleci.com/pipelines/github/martin-g/varnish-cache/71/workflows/3a275d79-62a9-48b4-9aef-1585de1c87c8 >>>>>>>>>> The builds on x86 arch take 3-4 mins, but for aarch64 >>>>>>>>>> (emulation!) ~40mins >>>>>>>>>> For now the jobs just build the .deb & .rpm packages for CentOS 7 >>>>>>>>>> and Ubuntu 18.04, both amd64 and aarch64. >>>>>>>>>> TODOs: >>>>>>>>>> - migrate Alpine >>>>>>>>>> >>>>>>>>> >>>>>>> Build on Alpine aarch64 fails with: >>>>>>> ... >>>>>>> automake: this behaviour will change in future Automake versions: >>>>>>> they will >>>>>>> automake: unconditionally cause object files to be placed in the >>>>>>> same subdirectory >>>>>>> automake: of the corresponding sources. >>>>>>> automake: project, to avoid future incompatibilities. >>>>>>> parallel-tests: installing 'build-aux/test-driver' >>>>>>> lib/libvmod_debug/Makefile.am:12: warning: libvmod_debug_la_LDFLAGS >>>>>>> multiply defined in condition TRUE ... >>>>>>> lib/libvmod_debug/automake_boilerplate.am:19: ... >>>>>>> 'libvmod_debug_la_LDFLAGS' previously defined here >>>>>>> lib/libvmod_debug/Makefile.am:9: 'lib/libvmod_debug/ >>>>>>> automake_boilerplate.am' included from here >>>>>>> + autoconf >>>>>>> + CONFIG_SHELL=/bin/sh >>>>>>> + export CONFIG_SHELL >>>>>>> + ./configure '--prefix=/opt/varnish' '--mandir=/opt/varnish/man' >>>>>>> --enable-maintainer-mode --enable-developer-warnings >>>>>>> --enable-debugging-symbols --enable-dependency-tracking >>>>>>> --with-persistent-storage --quiet >>>>>>> configure: WARNING: dot not found - build will fail if svg files are >>>>>>> out of date. >>>>>>> configure: WARNING: No system jemalloc found, using system malloc >>>>>>> configure: error: Could not find backtrace() support >>>>>>> >>>>>>> Does anyone know a workaround ? >>>>>>> I use multiarch/alpine:aarch64-edge as a base Docker image >>>>>>> >>>>>>> Martin >>>>>>> >>>>>>> >>>>>>> >>>>>>>> - store the packages as CircleCI artifacts >>>>>>>>>> - anything else that is still missing >>>>>>>>>> >>>>>>>>>> Adding more architectures would be as easy as adding a new >>>>>>>>>> Dockerfile with a base image from the respective type. >>>>>>>>>> >>>>>>>>>> Martin >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> 2) Travis CI >>>>>>>>>>> 2.1) problems >>>>>>>>>>> 2.1.1) generally Travis is slower than Circle! >>>>>>>>>>> Althought if we use CircleCI 'machine' executor it will be >>>>>>>>>>> slower than the current 'Docker' executor! >>>>>>>>>>> 2.1.2) Travis supports only Ubuntu >>>>>>>>>>> Current setup at CircleCI uses CentOS 7. >>>>>>>>>>> I guess the build steps won't have problems on Ubuntu. >>>>>>>>>>> >>>>>>>>>>> 3) GitHub Actions >>>>>>>>>>> GH Actions does not support ARM64 but it supports self hosted >>>>>>>>>>> ARM64 runners >>>>>>>>>>> 3.1) The problem is that there is no way to make a self hosted >>>>>>>>>>> runner really private. I.e. if someone forks Varnish Cache any >>>>>>>>>>> commit in >>>>>>>>>>> the fork will trigger builds on the arm64 node. There is no way to >>>>>>>>>>> reserve >>>>>>>>>>> the runner only for commits against >>>>>>>>>>> https://github.com/varnishcache/varnish-cache >>>>>>>>>>> >>>>>>>>>>> Do you see other problems or maybe different ways ? >>>>>>>>>>> Do you have preferences which way to go ? >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Martin >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Martin >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Guillaume Quintard >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> varnish-dev mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> >>>>>>>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >>>>>>>>>>>>> >>>>>>>>>>>>
_______________________________________________ varnish-dev mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
