Re: [PATCH] CI: travis-ci: disable arm64 builds
Hi Илья, On Mon, Aug 9, 2021 at 12:45 PM Илья Шипицин wrote: > I'm using arm64 in Oracle Cloud Ampere A1 Compute | Oracle > <https://www.oracle.com/cloud/compute/arm/> > Yes! OCI has an ARM64 VM in their free tier! They also provide easy integration with GitHub Actions - https://blogs.oracle.com/cloud-infrastructure/post/announcing-github-actions-arm-runners-for-the-arm-compute-platform-on-oracle-cloud-infrastructure But the issue here is that GitHub Actions self-hosted runners are not recommended for public/OSS projects because bad people can execute malware on your VM via (meaningless) Pull Requests. > > > also, I've found promising approach (using ARM on Github Actions) Bump > Bootstrap version from 5.0.2 to 5.1.0 · phpmyadmin/phpmyadmin@c90affe > (github.com) <https://github.com/phpmyadmin/phpmyadmin/runs/3274334375> > This setup uses QEMU - https://github.com/phpmyadmin/phpmyadmin/blob/c90affe793b56fb6f5e1c7eed676ec9031fb1480/.github/workflows/tests.yml#L51 . It works but it is much slower than using a real ARM64 machine/VM. Martin > пн, 9 авг. 2021 г. в 14:29, Willy Tarreau : > >> Hi Martin, >> >> On Mon, Aug 09, 2021 at 11:04:34AM +0300, Martin Grigorov wrote: >> > TravisCI just announced some improvements related to 'arch: arm64' >> (using >> > Equnix Metal machines) - >> https://blog.travis-ci.com/2021-08-06-oss-equinix. >> >> Thanks for the info! >> >> > But I also had some similar problems with them recently and replaced the >> > config with 'arch: arm64-graviton2; group: edge; virt: vm;', i.e. AWS >> > Graviton2 machines. In my experience they behave more stable! >> >> Yeah, these machines are really fantastic. The real problem anyway is >> likely that nowadays everyone is interested in testing on arm and very >> few have one available, let alone even a cross-compiler, so I suspect >> that these days a lot of people enable arm builds in such CI environments >> because it's the only way they have to make sure their code builds there >> at all. >> >> Cheers, >> Willy >> >
Re: [PATCH] CI: travis-ci: disable arm64 builds
Hi Willy, On Mon, Aug 9, 2021 at 12:29 PM Willy Tarreau wrote: > Hi Martin, > > On Mon, Aug 09, 2021 at 11:04:34AM +0300, Martin Grigorov wrote: > > TravisCI just announced some improvements related to 'arch: arm64' (using > > Equnix Metal machines) - > https://blog.travis-ci.com/2021-08-06-oss-equinix. > > Thanks for the info! > > > But I also had some similar problems with them recently and replaced the > > config with 'arch: arm64-graviton2; group: edge; virt: vm;', i.e. AWS > > Graviton2 machines. In my experience they behave more stable! > > Yeah, these machines are really fantastic. The real problem anyway is > likely that nowadays everyone is interested in testing on arm and very > few have one available, let alone even a cross-compiler, so I suspect > that these days a lot of people enable arm builds in such CI environments > because it's the only way they have to make sure their code builds there > at all. > I am not sure whether you understood me. Or maybe I didn't understand you. Anyway, let me rephrase: TravisCI provides two types of ARM64 VMs - arm64 (powered by Equinix Metal) and arm64-graviton2 (by AWS Graviton2). You, as a user, can use any or both of them in your .travis.yml. I prefer the AWS Graviton2 instances because there are less issues with them. So, instead of disabling the ARM64 CI job I suggest to try with arm64-graviton2. Here is a sample setup for it - https://github.com/apache/wicket/blob/270a5a43970cd975539331b21a34bd83a59c9c39/.travis.yml#L18-L22 More info about it at https://blog.travis-ci.com/2020-09-11-arm-on-aws Martin > > Cheers, > Willy >
Re: [PATCH] CI: travis-ci: disable arm64 builds
Hi, On Sat, Aug 7, 2021 at 8:31 AM Willy Tarreau wrote: > Hi Ilya, > > On Tue, Aug 03, 2021 at 02:58:40PM +0500, ??? wrote: > > Hello, > > > > it looks like "something on travis-ci side". > > > > CC src/raw_sock.o > > gcc: fatal error: Killed signal terminated program cc1 > > compilation terminated. > > > > let us disable arm64 for a while. > TravisCI just announced some improvements related to 'arch: arm64' (using Equnix Metal machines) - https://blog.travis-ci.com/2021-08-06-oss-equinix. But I also had some similar problems with them recently and replaced the config with 'arch: arm64-graviton2; group: edge; virt: vm;', i.e. AWS Graviton2 machines. In my experience they behave more stable! Regards, Martin > > Yes I noticed a few of these lately, and sometimes it even looked like > impossible downloads. I suspect that github assigns a maximum runtime > to these VMs and that they're simply overloaded and victims of their > success. I could be wrong of course. > > Now applied, thank you! > Willy > >
Re: Load testing HAProxy 2.2 on x86_64 and aarch64 VMs
Hi Илья, I didn't have much success with Google Perf Tools. I've tried both with adding '-lprofiler' to LDFLAGS when building HAProxy and with LD_PRELOAD. In both cases on ARM64 it fails with: terminated by signal SIGSEGV (Address boundary error) On x86_64 there is no such issue but the produced result file is always empty. $ ldd haproxy linux-vdso.so.1 (0x7ffcfb5cd000) libprofiler.so.0 => /usr/local/lib/libprofiler.so.0 (0x7f2115507000)<<<<<<<<<<<<<<<<<< libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x7f21154cc000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x7f21154b) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f21154aa000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f211549f000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f211547c000) libssl.so.1.1 => /home/ubuntu/opt/lib/libssl.so.1.1 (0x7f21151e6000) libcrypto.so.1.1 => /home/ubuntu/opt/lib/libcrypto.so.1.1 (0x7f2114cf4000) liblua5.3.so.0 => /usr/lib/x86_64-linux-gnu/liblua5.3.so.0 (0x7f2114cb9000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f2114b6a000) libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x7f2114abd000) libpcreposix.so.3 => /usr/lib/x86_64-linux-gnu/libpcreposix.so.3 (0x7f2114ab8000) libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x7f2114a43000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x7f2114a28000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f2114836000) libunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8 (0x7f2114819000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x7f2114638000) /lib64/ld-linux-x86-64.so.2 (0x7f211552e000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x7f211460f000) liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 (0x7f21145ee000) libgcrypt.so.20 => /usr/lib/x86_64-linux-gnu/libgcrypt.so.20 (0x7f21144d) libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x7f21144ad000) CPUPROFILE=/tmp/haproxy-load-x64.prof haproxy -p pid.txt -f haproxy.cfg The above command creates /tmp/haproxy-load-x64.prof but it is never populated with any data. I've tried stopping HAProxy with signals INT, KILL and TERM. I even tried with env var CPUPROFILESIGNAL=12 to start/stop the profiler manually but again no success. I could share with you reports from Linux 'perf' command. Just let me know which events you'd be interested in! Regards, Martin On Sat, Jul 18, 2020 at 11:40 AM Илья Шипицин wrote: > Hello, Martin! > > Can you please compare load profiles using google perftools ? > > I never tried to use gperf on ARM64, also, my trial at Linaro is over, I > do not have an access to any ARM64 anymore. > in short, gperf can be found https://github.com/gperftools/gperftools > > please follow "CPU profiling part". > > it can collect cachegrind output, I attached example kcachegrind report > (you can sort by "self" time). > it would be interesting to compare amd64 <--> arm64 > > [image: Screenshot from 2020-07-18 13-35-27.png] > > > пт, 10 июл. 2020 г. в 19:00, Martin Grigorov : > >> Hello HAProxy community, >> >> I wanted to compare how the newly released HAProxy 2.2 (Congrats!) >> behaves under heavy load so I've ran some tests on my x86_64 and aarch64 >> VMs: >> >> >> https://medium.com/@martin.grigorov/compare-haproxy-performance-on-x86-64-and-arm64-cpu-architectures-bfd55d1d5566 >> >> Without much surprise the x86_64 VM gave better results! >> >> It is *not* a real use case scenario: the backends serve on GET / and >> return "Hello World", without any file/network operations. >> >> What is interesting though is that I can get 120-160K reqs/sec when >> hitting directly one of the backend servers, and only 20-40K reqs/sec when >> using HAProxy as a load balancer in front of them. >> >> I'd be happy to re-run the tests with any kind of improvements you may >> have! >> >> Regards, >> Martin >> >
Re: Load testing HAProxy 2.2 on x86_64 and aarch64 VMs
Hi Илья, I will do it sometimes this week! Regards, Martin On Sat, Jul 18, 2020 at 11:40 AM Илья Шипицин wrote: > Hello, Martin! > > Can you please compare load profiles using google perftools ? > > I never tried to use gperf on ARM64, also, my trial at Linaro is over, I > do not have an access to any ARM64 anymore. > in short, gperf can be found https://github.com/gperftools/gperftools > > please follow "CPU profiling part". > > it can collect cachegrind output, I attached example kcachegrind report > (you can sort by "self" time). > it would be interesting to compare amd64 <--> arm64 > > [image: Screenshot from 2020-07-18 13-35-27.png] > > > пт, 10 июл. 2020 г. в 19:00, Martin Grigorov : > >> Hello HAProxy community, >> >> I wanted to compare how the newly released HAProxy 2.2 (Congrats!) >> behaves under heavy load so I've ran some tests on my x86_64 and aarch64 >> VMs: >> >> >> https://medium.com/@martin.grigorov/compare-haproxy-performance-on-x86-64-and-arm64-cpu-architectures-bfd55d1d5566 >> >> Without much surprise the x86_64 VM gave better results! >> >> It is *not* a real use case scenario: the backends serve on GET / and >> return "Hello World", without any file/network operations. >> >> What is interesting though is that I can get 120-160K reqs/sec when >> hitting directly one of the backend servers, and only 20-40K reqs/sec when >> using HAProxy as a load balancer in front of them. >> >> I'd be happy to re-run the tests with any kind of improvements you may >> have! >> >> Regards, >> Martin >> >
Re: Log levels when logging to stdout
On Thu, Jul 16, 2020 at 10:22 AM Jerome Magnin wrote: > Hi Martin, > > On Thu, Jul 16, 2020 at 10:05:40AM +0300, Martin Grigorov wrote: > > > > I am using such logging configuration (HAProxy built from master branch): > > > > global > > log stdout format raw local0 err > > ... > > defaults > > log global > > option dontlog-normal > > option httplog > > option dontlognull > > ... > > > > But HAProxy still logs entries like the following: > > > > 0001:test_fe.clireq[001c:]: GET /haproxy-load HTTP/1.1 > > 0002:test_fe.clihdr[001e:]: host: 192.168.0.206:8080 > > 0006:test_fe.accept(0004)=0020 from [:::192.168.0.72:46768] > > ALPN= > > 0001:test_fe.clihdr[001c:]: host: 192.168.0.206:8080 > > 0002:test_fe.clihdr[001e:]: content-type: application/json > > 0004:test_fe.accept(0004)=0024 from [:::192.168.0.72:46754] > > ALPN= > > 0005:test_fe.accept(0004)=001f from [:::192.168.0.72:46766] > > ALPN= > > 0001:test_fe.clihdr[001c:]: content-type: application/json > > 0007:test_fe.accept(0004)=0025 from [:::192.168.0.72:46776] > > ALPN= > > 0003:test_fe.clireq[0022:]: GET /haproxy-load HTTP/1.1 > > 0008:test_fe.accept(0004)=001d from [:::192.168.0.72:46756] > > ALPN= > > 0004:test_fe.clireq[0024:]: GET /haproxy-load HTTP/1.1 > > 0003:test_fe.clihdr[0022:]: host: 192.168.0.206:8080 > > 0004:test_fe.clihdr[0024:]: host: 192.168.0.206:8080 > > > > Those do not look like errors but in any case I tried also with 'emerg' > log > > level instead of 'err' > > and this didn't change anything. > > This is haproxy debug output, available when you start with haproxy -d. > Thanks, Jerome! This was the problem ! Martin > > > > > Do I configure it in a wrong way ? > > I want HAProxy to log only when there is a problem. Because now it logs > few > > GBs of those when I load it and this affects the performance. > > https://medium.com/@martin.grigorov/hi-willy-476dee6439d3 > > How do you start haproxy ? > > -- > Jérôme >
Log levels when logging to stdout
Hello HAProxy community, I am using such logging configuration (HAProxy built from master branch): global log stdout format raw local0 err ... defaults log global option dontlog-normal option httplog option dontlognull ... But HAProxy still logs entries like the following: 0001:test_fe.clireq[001c:]: GET /haproxy-load HTTP/1.1 0002:test_fe.clihdr[001e:]: host: 192.168.0.206:8080 0006:test_fe.accept(0004)=0020 from [:::192.168.0.72:46768] ALPN= 0001:test_fe.clihdr[001c:]: host: 192.168.0.206:8080 0002:test_fe.clihdr[001e:]: content-type: application/json 0004:test_fe.accept(0004)=0024 from [:::192.168.0.72:46754] ALPN= 0005:test_fe.accept(0004)=001f from [:::192.168.0.72:46766] ALPN= 0001:test_fe.clihdr[001c:]: content-type: application/json 0007:test_fe.accept(0004)=0025 from [:::192.168.0.72:46776] ALPN= 0003:test_fe.clireq[0022:]: GET /haproxy-load HTTP/1.1 0008:test_fe.accept(0004)=001d from [:::192.168.0.72:46756] ALPN= 0004:test_fe.clireq[0024:]: GET /haproxy-load HTTP/1.1 0003:test_fe.clihdr[0022:]: host: 192.168.0.206:8080 0004:test_fe.clihdr[0024:]: host: 192.168.0.206:8080 Those do not look like errors but in any case I tried also with 'emerg' log level instead of 'err' and this didn't change anything. Do I configure it in a wrong way ? I want HAProxy to log only when there is a problem. Because now it logs few GBs of those when I load it and this affects the performance. https://medium.com/@martin.grigorov/hi-willy-476dee6439d3 Regards, Martin
Load testing HAProxy 2.2 on x86_64 and aarch64 VMs
Hello HAProxy community, I wanted to compare how the newly released HAProxy 2.2 (Congrats!) behaves under heavy load so I've ran some tests on my x86_64 and aarch64 VMs: https://medium.com/@martin.grigorov/compare-haproxy-performance-on-x86-64-and-arm64-cpu-architectures-bfd55d1d5566 Without much surprise the x86_64 VM gave better results! It is *not* a real use case scenario: the backends serve on GET / and return "Hello World", without any file/network operations. What is interesting though is that I can get 120-160K reqs/sec when hitting directly one of the backend servers, and only 20-40K reqs/sec when using HAProxy as a load balancer in front of them. I'd be happy to re-run the tests with any kind of improvements you may have! Regards, Martin
Broken builds after "REORG: dgram: rename proto_udp to dgram"
Hello, Just FYI: the build is broken since this commit: https://github.com/haproxy/haproxy/commit/7c18b54106ec21273aea3fe59ba23280e86821f5 https://travis-ci.com/github/haproxy/haproxy/builds Regards, Martin
Re: [PATCH] enable arm64 builds in travis-ci
Hi Willy, On Fri, May 15, 2020 at 6:07 PM Willy Tarreau wrote: > Ilya, > > > also, I'd suggest to purge travis-ci cache (if you are build in your own > > fork). > > some travis related issue might be related when something is took from > > cache (which was not supposed to happen) > > Could you please handle Martin's patch, possibly cut it into several > pieces if relevant and add a commit message indicating what it does > (and why) ? Martin is not at ease with Git (which is not a problem), > and it seems only him and you understand how the reasons of the changes > in his patch. At least it's totally unclear to me why there's a new > install target for arm64 and why there's a special "make" invocation > there. > Let me explain the change. At https://github.com/haproxy/haproxy/blob/a8dbdf3c4b463a3f3e018f0cd02fa0d8d179bc07/.travis.yml#L113-L117 you may see the default 'install' phase. At https://github.com/haproxy/haproxy/blob/a8dbdf3c4b463a3f3e018f0cd02fa0d8d179bc07/.travis.yml#L12-L19 is the default environment. They are used by every job from the matrix ( https://github.com/haproxy/haproxy/blob/a8dbdf3c4b463a3f3e018f0cd02fa0d8d179bc07/.travis.yml#L35 ). But each job can override the default environment and any of the phases (before_install, install, after_install, script). For the ARM64 build I overwrote the 'install' phase by copying the default one and removing the execution of the build_ssl() function (the one that builds OpenSSL from source) and I also overwrote the environment to update the values of SSL_INC and SSL_LIB variables. 'openssl' and 'libssl-dev' packages are already installed in the Ubuntu image used by TravisCI so there is nothing to install manually.. I've added a comment ( https://github.com/haproxy/haproxy/blob/a8dbdf3c4b463a3f3e018f0cd02fa0d8d179bc07/.travis.yml#L47) to remind us how it works. > Feel free to add your "purge cache" change as an extra patch if needed. > But in any case, please make sure it's still possible to follow the > impact of each change, because we've touched many things blindly for > a while on this arm64 issue and most of the changes were basically > "let's see if this helps", which is a real mess :-/ > > Thanks! > Willy >
Re: [PATCH] enable arm64 builds in travis-ci
Thank you for applying the patch! The ARM64 build passed at https://travis-ci.com/github/haproxy/haproxy/jobs/335338296 ! But it passes only for the builds which have 'haproxy-mirror' : https://travis-ci.com/github/haproxy/haproxy/builds I am not sure what exactly "haproxy-mirror" is in the TravisCI config. The builds which do not have "haproxy-mirror" next to the branch name also do not have ARM64 build at all, e.g. https://travis-ci.com/github/haproxy/haproxy/builds/166573019. One thing that I notice is that the successful link above has *jobs *and the failing one: *builds *before the job/build id. On Fri, May 15, 2020 at 9:53 PM Willy Tarreau wrote: > On Fri, May 15, 2020 at 11:44:48PM +0500, ??? wrote: > > commit message adjusted > > Many thanks for this, Ilya, now pushed! > > Willy >
Re: [PATCH] enable arm64 builds in travis-ci
Those are set to new values at https://github.com/haproxy/haproxy/pull/630/files#diff-354f30a63fb0907d4ad57269548329e3R51 On Fri, May 15, 2020 at 1:11 PM Илья Шипицин wrote: > or we'd better move SSL_LIB, SSL_INC to build-ssl.sh script > > пт, 15 мая 2020 г. в 15:09, Илья Шипицин : > >> probably, you also need to unset SSL_LIB and SSL_INC >> >> >> >> btw, I got an answer how to grant travis-ci rights (for triggering build >> manually) >> >> https://travis-ci.community/t/undocumented-require-admin-permissions/8530 >> >> >> пт, 15 мая 2020 г. в 14:57, Martin Grigorov : >> >>> Hi, >>> >>> I've created https://github.com/haproxy/haproxy/pull/630 >>> With this change the build passed successfully for 5 mins and 7 secs for >>> ARM64. >>> >>> Please let me know if you prefer me to send it as an attached .patch >>> file here. (I haven't used `git format-patch` before :-/). >>> >>> Martin >>> >>> On Mon, May 11, 2020 at 12:38 PM Илья Шипицин >>> wrote: >>> >>>> >>>> >>>> сб, 9 мая 2020 г. в 11:45, Willy Tarreau : >>>> >>>>> On Sat, May 09, 2020 at 08:11:27AM +0200, Vincent Bernat wrote: >>>>> > ? 8 mai 2020 14:25 +02, Willy Tarreau: >>>>> > >>>>> > >> > Let's increase the timeout to see if it has a chance to finish, >>>>> no ? >>>>> > >> > >>>>> > >> >>>>> > >> yes >>>>> > > >>>>> > > OK now pushed. It's really annoying to work blindly like this. The >>>>> > > build model Travis uses is broken by design. Requiring to commit >>>>> > > something for testing is utterly wrong. And doing so within the >>>>> > > project that's supposed to being test is further wrong. We already >>>>> > > have 44 patches only on .travis.yml! If this continues like this, >>>>> > > I predict that a "pre-CI" solution will appear to test if your >>>>> > > change is likely to trigger a travis error before it gets merged... >>>>> > >>>>> > You can push changes to a (throwable) branch instead. >>>>> >>>>> Good point, that can also be a solution. But it remains completely >>>>> hackish. It's basically abusing a versioning system to use it as a >>>>> messaging system to indicate "please build with this". >>>>> >>>>> Willy >>>>> >>>> >>>> >>>> I created several topics (no answer yet). >>>> >>>> as for travis-ci rights, it's totally undocumented. but I suspect >>>> travis grants >>>> rights based on github rights. i.e. github admin becomes travis admin >>>> as well. >>>> >>>> https://travis-ci.community/t/arm64-fails-with-non-clear-reason/8529 >>>> >>>> >>>> https://travis-ci.community/t/undocumented-require-admin-permissions/8530 >>>> >>>> >>>> https://travis-ci.community/t/undocumented-operation-requires-create-request-access-to-repository/8528 >>>> >>>> >>>
Re: [PATCH] enable arm64 builds in travis-ci
Hi, I've created https://github.com/haproxy/haproxy/pull/630 With this change the build passed successfully for 5 mins and 7 secs for ARM64. Please let me know if you prefer me to send it as an attached .patch file here. (I haven't used `git format-patch` before :-/). Martin On Mon, May 11, 2020 at 12:38 PM Илья Шипицин wrote: > > > сб, 9 мая 2020 г. в 11:45, Willy Tarreau : > >> On Sat, May 09, 2020 at 08:11:27AM +0200, Vincent Bernat wrote: >> > ? 8 mai 2020 14:25 +02, Willy Tarreau: >> > >> > >> > Let's increase the timeout to see if it has a chance to finish, no >> ? >> > >> > >> > >> >> > >> yes >> > > >> > > OK now pushed. It's really annoying to work blindly like this. The >> > > build model Travis uses is broken by design. Requiring to commit >> > > something for testing is utterly wrong. And doing so within the >> > > project that's supposed to being test is further wrong. We already >> > > have 44 patches only on .travis.yml! If this continues like this, >> > > I predict that a "pre-CI" solution will appear to test if your >> > > change is likely to trigger a travis error before it gets merged... >> > >> > You can push changes to a (throwable) branch instead. >> >> Good point, that can also be a solution. But it remains completely >> hackish. It's basically abusing a versioning system to use it as a >> messaging system to indicate "please build with this". >> >> Willy >> > > > I created several topics (no answer yet). > > as for travis-ci rights, it's totally undocumented. but I suspect travis > grants > rights based on github rights. i.e. github admin becomes travis admin as > well. > > https://travis-ci.community/t/arm64-fails-with-non-clear-reason/8529 > > https://travis-ci.community/t/undocumented-require-admin-permissions/8530 > > > https://travis-ci.community/t/undocumented-operation-requires-create-request-access-to-repository/8528 > >
Re: [PATCH] enable arm64 builds in travis-ci
On Fri, May 8, 2020 at 10:54 AM Илья Шипицин wrote: > > > пт, 8 мая 2020 г. в 12:26, Willy Tarreau : > >> On Fri, May 08, 2020 at 09:34:32AM +0300, Martin Grigorov wrote: >> > It must have started failing when you updated the version of OpenSSL. >> > .travis.yml caches ~/opt folder between builds. After the update to >> 1.1.1f >> > the build doesn't see the OpenSSL binaries in the cache anymore and >> tries >> > to download it and build it. >> > But as I've noticed in my attempt to build HAProxy with Docker+QEMU the >> > build of OpenSSL is taking too long. >> > The build of OpenSSL is wrapped with travis_wait to reduce the writes to >> > stdout but the default time for travis_wait is 20 mins and this is not >> > enough to build OpenSSL. >> >> That's very likely indeed. >> > > > it is not :) > I provided link to my fork, openssl build takes 640 sec > > >> >> > Due to >> > >> https://travis-ci.community/t/output-is-truncated-heavily-in-arm64-when-a-command-hangs/7630 >> > TravisCI >> > does not properly report that the problem is at build_ssl() step but >> shows >> > the last chunk of the buffered response and this confuses us all. >> >> Ah, excellent, precisely what I was looking for. And some indicate >> that the buffering further causes issues in the build system itself! >> >> > I think the build will become green if we extend travis_wait to a higher >> > value ( >> > >> https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received >> ). >> > I don't remember where I have read it but I think the upper limit is 120 >> > mins. >> > @Willy: could you please change >> > https://github.com/haproxy/haproxy/blob/master/.travis.yml#L112 to: >> > >> > travis_wait 120 bash -c 'scripts/build-ssl.sh >build-ssl.log 2>&1' || >> (cat >> > build-ssl.log && exit 1) >> > >> > i.e. add '120' after travis_wait >> >> We could, but 120 (2 hours) seems a bit extreme. It also means we can >> steal >> 2 hours of CPU there in case something goes wrong. >> > > travis-ci limits build at 60 min. > > >> >> > This should give it the time to download and install OpenSSL 1.1.1f and >> to >> > cache it. If the build passes once then the next builds should be much >> > faster because OpenSSL will be used from the cache. >> >> I'm wondering why instead we don't fallback on an already packaged version >> of OpenSSL for this platform. I mean, sure it's convenient to test the >> latest version but it's already tested on x86 and we could very well use >> any other version already packaged on the distro present there. This would >> solve the problem and even increase versions coverage. >> >> Ilya, what do you think ? >> > > yes, there are few options to think about. > I also provided options (numbered 1..4), Hopefully, I think on your > suggestions, Martin will think on my suggestion ... and we will decide what > next would be. > I still believe the problem is in the time needed to build OpenSSL, not in apt[-get]. We can increase temporarily the travis_wait to 60 just to cache the build of 1.1.1f and then remove '60' again. Although if this is the problem then 'travis_wait 60 ...' won't do any harm for the next builds because it will use the cache and return in few secs. I like the idea to install openssl with apt but I am not sure whether we can do this *only* for ARM64. > > >> >> Thanks, >> Willy >> >
Re: [PATCH] enable arm64 builds in travis-ci
Hi, I think I understand why it started failing. It must have started failing when you updated the version of OpenSSL. .travis.yml caches ~/opt folder between builds. After the update to 1.1.1f the build doesn't see the OpenSSL binaries in the cache anymore and tries to download it and build it. But as I've noticed in my attempt to build HAProxy with Docker+QEMU the build of OpenSSL is taking too long. The build of OpenSSL is wrapped with travis_wait to reduce the writes to stdout but the default time for travis_wait is 20 mins and this is not enough to build OpenSSL. Due to https://travis-ci.community/t/output-is-truncated-heavily-in-arm64-when-a-command-hangs/7630 TravisCI does not properly report that the problem is at build_ssl() step but shows the last chunk of the buffered response and this confuses us all. I think the build will become green if we extend travis_wait to a higher value ( https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received). I don't remember where I have read it but I think the upper limit is 120 mins. @Willy: could you please change https://github.com/haproxy/haproxy/blob/master/.travis.yml#L112 to: travis_wait 120 bash -c 'scripts/build-ssl.sh >build-ssl.log 2>&1' || (cat build-ssl.log && exit 1) i.e. add '120' after travis_wait This should give it the time to download and install OpenSSL 1.1.1f and to cache it. If the build passes once then the next builds should be much faster because OpenSSL will be used from the cache. Regards, Martin On Fri, May 8, 2020 at 9:18 AM Willy Tarreau wrote: > Hi Martin, > > On Fri, May 08, 2020 at 08:56:07AM +0300, Martin Grigorov wrote: > > Unfortunately it is not good: > > https://travis-ci.com/github/haproxy/haproxy/jobs/329657180 > > Indeed it's still not fixed on Travis' side. However what Ilya did > actually worked, in that the status is not reported as a global > build failure anymore. This allows us to continue to monitor if > and when this issue finally resolves on the build infrastructure. > It's also possible that they're not aware of the problem due to > too few people using arm64. If someone has contacts there it might > be worth checking with them. All we know for now is that it seems > to stop moving while setting up libpcre2. Maybe there's a bug in > a script in this package, resulting in a prompt for a question > which never gets a response :-/ But that's something we can't > check since we don't have access to an interactive shell there > to diagnose. > > Willy >
Re: [PATCH] enable arm64 builds in travis-ci
Hi all, On Thu, May 7, 2020 at 11:56 PM Willy Tarreau wrote: > Hi Ilya, > > On Thu, May 07, 2020 at 09:19:48PM +0500, ??? wrote: > > Hello, > > > > let us enable arm64 builds back. > > Good idea, just merged now. Let's see how that ends up now. > Unfortunately it is not good: https://travis-ci.com/github/haproxy/haproxy/jobs/329657180 Martin > > Thanks, > Willy > >
Re: [PATCH] fix errored ARM64 builds in travis-ci
Hi, I've just created a PR (https://github.com/haproxy/haproxy/pull/617/files) that introduces testing on ARM64/AARCH64 at GitHub Actions. It almost works! There are few tests that fail. Any help finding the reason is very welcome! Martin On Mon, Mar 23, 2020 at 11:12 AM Martin Grigorov wrote: > Hi Илья, > > On Sun, Mar 22, 2020 at 2:46 PM Илья Шипицин wrote: > >> Martin, >> >> as the one of the most interested in ARM64 builds, I've got news for you >> >> >> can you try >> >> travis_wait 30 bash -c 'scripts/build-ssl.sh >build-ssl.log 2>&1' || (cat >> build-ssl.log && exit 1) >> >> in travis ? (please not "travis_wait 30" instead of "travis_wait") >> > > it is running at the moment here: > https://travis-ci.org/github/martin-g/haproxy/builds/665770469 > > >> >> >> also, it might be important to clear travis cache from time to time. >> as for myself, "travis_wait 30" helped me to resolve similar issue on >> another project (in my own fork haproxy on arm64 builds just fine) >> >> ср, 18 мар. 2020 г. в 23:35, Илья Шипицин : >> >>> well, there are several topics on travis-ci forum related to "output on >>> ARM64 got truncated in the mid of ..." >>> Let us disable ARM64 travis-ci builds for few months. >>> >>> Martin, I'll play with hosted github runner in order to find a way how >>> we can limit its builds to allowed only. >>> >>> ср, 18 мар. 2020 г. в 18:57, Martin Grigorov : >>> >>>> >>>> Current master's build passed the problematic point in my TravisCI >>>> project: https://travis-ci.org/github/martin-g/haproxy/jobs/663953359 >>>> Note: I use TravisCI .org while HAProxy's official project is at .com: >>>> https://travis-ci.com/github/haproxy/haproxy >>>> I also think this is a problem on TravisCI's end. >>>> >>>> Martin >>>> >>>> On Wed, Mar 18, 2020 at 3:43 PM Илья Шипицин >>>> wrote: >>>> >>>>> I will disable PR builds. >>>>> >>>>> On Wed, Mar 18, 2020, 6:27 PM Willy Tarreau wrote: >>>>> >>>>>> On Wed, Mar 18, 2020 at 06:21:15PM +0500, ??? wrote: >>>>>> > let us calm down a bit :) >>>>>> >>>>>> Agreed, especially since the build on PRs already happens and already >>>>>> adds noise. >>>>>> >>>>>> > yes, I still believe it is because of buffering. I might have missed >>>>>> > something. >>>>>> > unless I will repair it, I'll drop arm64 support on travis (and we >>>>>> will >>>>>> > switch to self hosted github action runner) >>>>>> >>>>>> OK. >>>>>> >>>>>> Willy >>>>>> >>>>>
Re: Failing tests if USE_OPENSSL=1 is omitted in the FLAGS
Hi Илья, On Wed, May 6, 2020 at 11:59 AM Илья Шипицин wrote: > do you run tests on GH arm64 agents ? is it dedicated (your own) agents > attached to your repo ? can you give a link ? > I use Docker + QEMU with GH hosted runner. You can see the current diff at https://github.com/haproxy/haproxy/compare/master...martin-g:feature/test-aarch64-on-github-actions 'windows-latest.yml` is temporary disabled until I'm ready with my experiments. As explained at https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories using self-hosted runner is not secure if they are used for Pull Requests. They can be used for pushes to `master` branch though since this is supposed to be a reviewed code! But the tests failures I reported below fail on my local machine (x86_64) and my ARM64 VM. I do not use Docker and/or QEMU here. Martin > ср, 6 мая 2020 г. в 13:22, Martin Grigorov : > >> Hello HAProxy team, >> >> While working on a PR to build & test HAProxy on AARCH64 at GitHub >> Actions I've noticed a strange behavior for some of the tests. >> >> To reduce the time of the build I've removed USE_OPENSSL=1 from the FLAGS >> [1] passed to "make". >> The build passed successfully, some of the tests are skipped because they >> depend on SSL library, e.g.: >> >> ... >> Add test: reg-tests/lua/txn_get_priv.vtc >> Add test: reg-tests/lua/h_txn_get_priv.vtc >> Skip reg-tests/ssl/wrong_ctx_storage.vtc because haproxy is not >> compiled with the required option OPENSSL >> Skip reg-tests/ssl/ssl_client_auth.vtc because haproxy is not compiled >> with the required option OPENSSL >> Skip reg-tests/ssl/add_ssl_crt-list.vtc because haproxy is not compiled >> with the required option OPENSSL >> Skip reg-tests/ssl/set_ssl_cert.vtc because haproxy is not compiled >> with the required option OPENSSL >> ... >> >> but few tests just fail: >> >> Testing with haproxy version: 2.2-dev7 >> #top TEST reg-tests/lua/txn_get_priv.vtc FAILED (5.008) exit=2 >> #top TEST reg-tests/compression/lua_validation.vtc TIMED OUT (kill >> -9) >> #top TEST reg-tests/compression/lua_validation.vtc FAILED (10.009) >> signal=9 >> 2 tests failed, 0 tests skipped, 64 tests passed >> ## Gathering results ## >> ## Test case: reg-tests/compression/lua_validation.vtc ## >> ## test results in: >> "/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.7fffd08a" >> ## Test case: reg-tests/lua/txn_get_priv.vtc ## >> ## test results in: >> "/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.065f2677" >> c0 HTTP rx timeout (fd:6 5000 ms) >> h1 Bad exit status: 0x0100 exit 0x1 signal 0 core 0 >> Makefile:995: recipe for target 'reg-tests' failed >> make: *** [reg-tests] Error 1 >> >> These tests fail both on AARCH64 and x86_64 consistently until I add back >> USE_OPENSSL=1 >> to the flags. >> >> Is there an issue with these tests ? >> >> 1. >> https://github.com/haproxy/haproxy/blob/fafa13dd6549ee431f41dc3c1857855974d79bea/.travis.yml#L14 >> >> Regards, >> Martin >> >
Failing tests if USE_OPENSSL=1 is omitted in the FLAGS
Hello HAProxy team, While working on a PR to build & test HAProxy on AARCH64 at GitHub Actions I've noticed a strange behavior for some of the tests. To reduce the time of the build I've removed USE_OPENSSL=1 from the FLAGS [1] passed to "make". The build passed successfully, some of the tests are skipped because they depend on SSL library, e.g.: ... Add test: reg-tests/lua/txn_get_priv.vtc Add test: reg-tests/lua/h_txn_get_priv.vtc Skip reg-tests/ssl/wrong_ctx_storage.vtc because haproxy is not compiled with the required option OPENSSL Skip reg-tests/ssl/ssl_client_auth.vtc because haproxy is not compiled with the required option OPENSSL Skip reg-tests/ssl/add_ssl_crt-list.vtc because haproxy is not compiled with the required option OPENSSL Skip reg-tests/ssl/set_ssl_cert.vtc because haproxy is not compiled with the required option OPENSSL ... but few tests just fail: Testing with haproxy version: 2.2-dev7 #top TEST reg-tests/lua/txn_get_priv.vtc FAILED (5.008) exit=2 #top TEST reg-tests/compression/lua_validation.vtc TIMED OUT (kill -9) #top TEST reg-tests/compression/lua_validation.vtc FAILED (10.009) signal=9 2 tests failed, 0 tests skipped, 64 tests passed ## Gathering results ## ## Test case: reg-tests/compression/lua_validation.vtc ## ## test results in: "/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.7fffd08a" ## Test case: reg-tests/lua/txn_get_priv.vtc ## ## test results in: "/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.065f2677" c0 HTTP rx timeout (fd:6 5000 ms) h1 Bad exit status: 0x0100 exit 0x1 signal 0 core 0 Makefile:995: recipe for target 'reg-tests' failed make: *** [reg-tests] Error 1 These tests fail both on AARCH64 and x86_64 consistently until I add back USE_OPENSSL=1 to the flags. Is there an issue with these tests ? 1. https://github.com/haproxy/haproxy/blob/fafa13dd6549ee431f41dc3c1857855974d79bea/.travis.yml#L14 Regards, Martin
Re: regtest: abns should work now :-)
Hi everyone, On Mon, Mar 23, 2020 at 11:11 AM Martin Grigorov wrote: > Hi Илья, > > On Mon, Mar 23, 2020 at 10:52 AM Илья Шипицин > wrote: > >> well, I tried to repro abns failures on x86_64 >> I chose MS Azure VM of completely different size, both number of CPU and >> RAM. >> it was never reproduced, say on 1000 execution in loop. >> >> so, I decided "it looks like something with memory aligning". >> also, I tried to run arm64 emulation on virtualbox. no luck yet. >> > > > Have you tried with multiarch Docker ? > > 1) execute > docker run --rm --privileged multiarch/qemu-user-static:register --reset > to register QEMU > > 2) create Dockerfile > for Centos use: FROM multiarch/centos:7-aarch64-clean > for Ubuntu use: FROM multiarch/ubuntu-core:arm64-bionic > > 3) enjoy :-) > Here is a PR for Varnish Cache project where I use Docker + QEMU to build and package for several Linux distros and two architectures: https://github.com/varnishcache/varnish-cache/pull/3263 They use CircleCI but I guess the same approach can be applied on GitHub Actions. If you are interested in this approach I could give it a try. Regards, Martin > > >> >> пн, 23 мар. 2020 г. в 13:43, Willy Tarreau : >> >>> Hi Ilya, >>> >>> I think this time I managed to fix the ABNS test. To make a long story >>> short, it was by design extremely sensitive to the new process's startup >>> time, which is increased with larger FD counts and/or less powerful VMs >>> and/or noisy neighbors. This explains why it started to misbehave with >>> the commit which relaxed the maxconn limitations. A starting process >>> stealing a few ms of CPU from the old one could make its keep-alive >>> timeout expire before it got a new request on a reused connection, >>> resulting in an empty response as reported by the client. >>> >>> I'm going to issue dev5 now. s390x is currently down but all x86 ones >>> build and run fine for now. >>> >>> Cheers, >>> Willy >>> >>
Re: [PATCH] fix errored ARM64 builds in travis-ci
Hi Илья, On Sun, Mar 22, 2020 at 2:46 PM Илья Шипицин wrote: > Martin, > > as the one of the most interested in ARM64 builds, I've got news for you > > > can you try > > travis_wait 30 bash -c 'scripts/build-ssl.sh >build-ssl.log 2>&1' || (cat > build-ssl.log && exit 1) > > in travis ? (please not "travis_wait 30" instead of "travis_wait") > it is running at the moment here: https://travis-ci.org/github/martin-g/haproxy/builds/665770469 > > > also, it might be important to clear travis cache from time to time. > as for myself, "travis_wait 30" helped me to resolve similar issue on > another project (in my own fork haproxy on arm64 builds just fine) > > ср, 18 мар. 2020 г. в 23:35, Илья Шипицин : > >> well, there are several topics on travis-ci forum related to "output on >> ARM64 got truncated in the mid of ..." >> Let us disable ARM64 travis-ci builds for few months. >> >> Martin, I'll play with hosted github runner in order to find a way how we >> can limit its builds to allowed only. >> >> ср, 18 мар. 2020 г. в 18:57, Martin Grigorov : >> >>> >>> Current master's build passed the problematic point in my TravisCI >>> project: https://travis-ci.org/github/martin-g/haproxy/jobs/663953359 >>> Note: I use TravisCI .org while HAProxy's official project is at .com: >>> https://travis-ci.com/github/haproxy/haproxy >>> I also think this is a problem on TravisCI's end. >>> >>> Martin >>> >>> On Wed, Mar 18, 2020 at 3:43 PM Илья Шипицин >>> wrote: >>> >>>> I will disable PR builds. >>>> >>>> On Wed, Mar 18, 2020, 6:27 PM Willy Tarreau wrote: >>>> >>>>> On Wed, Mar 18, 2020 at 06:21:15PM +0500, ??? wrote: >>>>> > let us calm down a bit :) >>>>> >>>>> Agreed, especially since the build on PRs already happens and already >>>>> adds noise. >>>>> >>>>> > yes, I still believe it is because of buffering. I might have missed >>>>> > something. >>>>> > unless I will repair it, I'll drop arm64 support on travis (and we >>>>> will >>>>> > switch to self hosted github action runner) >>>>> >>>>> OK. >>>>> >>>>> Willy >>>>> >>>>
Re: regtest: abns should work now :-)
Hi Илья, On Mon, Mar 23, 2020 at 10:52 AM Илья Шипицин wrote: > well, I tried to repro abns failures on x86_64 > I chose MS Azure VM of completely different size, both number of CPU and > RAM. > it was never reproduced, say on 1000 execution in loop. > > so, I decided "it looks like something with memory aligning". > also, I tried to run arm64 emulation on virtualbox. no luck yet. > Have you tried with multiarch Docker ? 1) execute docker run --rm --privileged multiarch/qemu-user-static:register --reset to register QEMU 2) create Dockerfile for Centos use: FROM multiarch/centos:7-aarch64-clean for Ubuntu use: FROM multiarch/ubuntu-core:arm64-bionic 3) enjoy :-) > > пн, 23 мар. 2020 г. в 13:43, Willy Tarreau : > >> Hi Ilya, >> >> I think this time I managed to fix the ABNS test. To make a long story >> short, it was by design extremely sensitive to the new process's startup >> time, which is increased with larger FD counts and/or less powerful VMs >> and/or noisy neighbors. This explains why it started to misbehave with >> the commit which relaxed the maxconn limitations. A starting process >> stealing a few ms of CPU from the old one could make its keep-alive >> timeout expire before it got a new request on a reused connection, >> resulting in an empty response as reported by the client. >> >> I'm going to issue dev5 now. s390x is currently down but all x86 ones >> build and run fine for now. >> >> Cheers, >> Willy >> >
Re: [PATCH] fix errored ARM64 builds in travis-ci
Hi, On Wed, Mar 18, 2020 at 3:29 PM Willy Tarreau wrote: > On Wed, Mar 18, 2020 at 06:21:15PM +0500, ??? wrote: > > let us calm down a bit :) > > Agreed, especially since the build on PRs already happens and already > adds noise. > > > yes, I still believe it is because of buffering. I might have missed > > something. > > unless I will repair it, I'll drop arm64 support on travis (and we will > > switch to self hosted github action runner) > There is one major problem with GitHub Actions self hosted runners at the moment - they are not really private. I.e. if someone forks HAProxy and pushes something into their fork it will trigger builds on your private node, i.e. it will consume its resources. There is no way to say "this is my private node and I want it to build only after commits in https://github.com/haproxy/haproxy; If you find a way around this issue and you need an ARM64 VM then just let me know! Regards, Martin > > OK. > > Willy > >
Re: [PATCH] switch to clang-9 in Linux/travis-ci builds
Hi, On Sat, Mar 14, 2020 at 11:26 AM Willy Tarreau wrote: > Hi Ilya, > > On Fri, Jan 24, 2020 at 11:46:45AM +0500, ??? wrote: > > Hello, > > > > let us use clang-9 instead of default clang-7 for linux builds. > > It seems I missed this one. Now applied carefully, we'll see. If it > causes new failures, we'll adjust accordingly. > All looks good here on aarch64 with this change! Martin > > Willy > >
Re: [PATCH]: BUILD link to lib atomic on ARM
Hi David, On my ARM64 VM `uname -m` returns: $ uname -m aarch64 Should your change take 'aarch64' into account as well ? Martin On Sun, Mar 15, 2020 at 3:34 PM David CARLIER wrote: > Oups sorry I really forgot :-) > > On Sun, 15 Mar 2020 at 13:32, Martin Grigorov > wrote: > >> Hi >> >> On Sun, Mar 15, 2020 at 3:03 PM Aleksandar Lazic >> wrote: >> >>> On 15.03.20 11:33, David CARLIER wrote: >>> > Hi >>> > >>> > Here a little patch proposal to fix build on ARM. >>> > >>> > Regards. >>> >>> Ähm, maybe my mail client hide the Patch because I can't see it ;-)? >>> >> >> It seems David forgot to attach it or the attachment didn't make it for >> other reason. I also don't see it. >> >> Martin >> >> >>> >>> Regards >>> Aleks >>> >>>
Re: [PATCH] enable DEBUG_STRICT=1 for all kind of CI builds
Hello, I've just tested the change on my ARM64 VM and the build is successful! Regards, Martin On Sun, Mar 15, 2020 at 9:12 AM Илья Шипицин wrote: > Hello, > > I added DEBUG_STRICT=1 to all builds. > > Ilya Shipitcin >
Re: [PATCH]: BUILD link to lib atomic on ARM
Hi On Sun, Mar 15, 2020 at 3:03 PM Aleksandar Lazic wrote: > On 15.03.20 11:33, David CARLIER wrote: > > Hi > > > > Here a little patch proposal to fix build on ARM. > > > > Regards. > > Ähm, maybe my mail client hide the Patch because I can't see it ;-)? > It seems David forgot to attach it or the attachment didn't make it for other reason. I also don't see it. Martin > > Regards > Aleks > >
Re: Tests timeout on my ARM64 test VM
Hi Willy, On Fri, Mar 13, 2020 at 10:03 PM Willy Tarreau wrote: > Hi Martin, > > On Fri, Mar 13, 2020 at 12:35:12PM +0200, Martin Grigorov wrote: > > Hi , > > > > Suddenly today the build is again green here! > > I didn't make any changes to my testing setup. > > It must be something on the OS level but I wasn't able to figure out what > > makes the HAProxy tests timeout in the last several days. > > We've had issues with the abns test on other platforms in the past, > namely s390x and ppc64le. It used to occasionally break on x86_64 as > well but far less frequently. It was affected by two bugs that were > solved yesterday after a few days of investigation and testing. We've > seen yet another failure again on ppc64 while it was expected not to > fail, so I'd be careful before claiming victory. However the abns > test is extremely time sensitive and uses short delays around 15ms to > try to trigger the issue, and in a VM it is possible to see this > happen from time to time due to noisy neighbors. That's why I'm > staying extremely prudent on the verdict. The PPC64 machine I tested > on is provided by Minicloud and is a VM running on a real CPU, so it's > much less affected by timing issues. I've run the test several hundreds > of times in a row and couldn't make it fail anymore. > > So don't worry too much if it appeared and disappeared. The change > that emphasized it was the increase in default maxconn (304e17eb8), > apparently just due to a slightly longer startup time! And the ones > expected to have fixed it are between bdb00c5d and 4b3f27b included. > > Note that I didn't manage to make it fail on arm64 (real machine, > SolidRun's Macchiatobin). > > Hoping this clarifies the situation. > Thank you for this explanation! The problem here was that the tests were failing even with older commits. I've tried to git bisect the problem but no matter how far back in Git history I went the problem was still there. But the logs from the tests runs from just few days back were OK, no errors & timeouts. Also the ARM64 tests on TravisCI were OK. And Travis's ARM64 nodes are less powerful than mine VM. Those are the reasons I believe that the problem was at my VM. I just needed help with reading HAProxy's tests error logs. I didn't know how to approach debugging them. Regards, Martin > > Regards, > Willy >
Re: Tests timeout on my ARM64 test VM
Hi Илья, Suddenly today the build is again green here! I didn't make any changes to my testing setup. It must be something on the OS level but I wasn't able to figure out what makes the HAProxy tests timeout in the last several days. Regards, Martin On Wed, Mar 11, 2020 at 4:13 PM Martin Grigorov wrote: > > > On Wed, Mar 11, 2020 at 3:06 PM Илья Шипицин wrote: > >> I will a look during next weekend >> > > Thank you, Илья! > > >> >> BTW, I've managed to get Linaro VM :) >> > > Congrats! :-) > > >> >> On Wed, Mar 11, 2020, 5:40 PM Martin Grigorov >> wrote: >> >>> Hi, >>> >>> On Mon, Mar 9, 2020 at 10:22 PM Martin Grigorov < >>> martin.grigo...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I am not sure what have changed on my test ARM64 VM but the reg tests >>>> started timing out. >>>> Everything is fine on my dev machine (x86_64) and at Travis ( >>>> https://travis-ci.com/haproxy/haproxy). >>>> I don't think it is ARM64 related. Most probably some OS setting or >>>> something. >>>> I've rebooted the system just to make sure it is not some busy port or >>>> opened file descriptor but >>>> it still fails the same way. >>>> >>>> Does someone see in the attached logs what could be the problem? >>>> >>> >>> Anyone can help me here ? >>> >>> Martin >>> >>> >>>> Thank you! >>>> >>>> Martin >>>> >>>
Re: Tests timeout on my ARM64 test VM
On Wed, Mar 11, 2020 at 3:06 PM Илья Шипицин wrote: > I will a look during next weekend > Thank you, Илья! > > BTW, I've managed to get Linaro VM :) > Congrats! :-) > > On Wed, Mar 11, 2020, 5:40 PM Martin Grigorov > wrote: > >> Hi, >> >> On Mon, Mar 9, 2020 at 10:22 PM Martin Grigorov < >> martin.grigo...@gmail.com> wrote: >> >>> Hi, >>> >>> I am not sure what have changed on my test ARM64 VM but the reg tests >>> started timing out. >>> Everything is fine on my dev machine (x86_64) and at Travis ( >>> https://travis-ci.com/haproxy/haproxy). >>> I don't think it is ARM64 related. Most probably some OS setting or >>> something. >>> I've rebooted the system just to make sure it is not some busy port or >>> opened file descriptor but >>> it still fails the same way. >>> >>> Does someone see in the attached logs what could be the problem? >>> >> >> Anyone can help me here ? >> >> Martin >> >> >>> Thank you! >>> >>> Martin >>> >>
Re: Tests timeout on my ARM64 test VM
Hi, On Mon, Mar 9, 2020 at 10:22 PM Martin Grigorov wrote: > Hi, > > I am not sure what have changed on my test ARM64 VM but the reg tests > started timing out. > Everything is fine on my dev machine (x86_64) and at Travis ( > https://travis-ci.com/haproxy/haproxy). > I don't think it is ARM64 related. Most probably some OS setting or > something. > I've rebooted the system just to make sure it is not some busy port or > opened file descriptor but > it still fails the same way. > > Does someone see in the attached logs what could be the problem? > Anyone can help me here ? Martin > Thank you! > > Martin >
Re: Lua detection on aarch64
Hi Илья, On Wed, Jan 29, 2020 at 11:34 AM Илья Шипицин wrote: > Hello, > > I started to work on rpm packages. > I also applied at Linaro for arm64 vm (request is not completed yet). > > > interesting that Lua is not detected: > > > https://copr-be.cloud.fedoraproject.org/results/chipitsine/haproxy-rpm/fedora-31-aarch64/01207488-haproxy/builder-live.log.gz > > Martin, do you have arm64, can you check that ? > I've had the same problem on my VM. I've tried to debug it and I think the problem was somewhere here: https://github.com/haproxy/haproxy/blob/160287b6760584eb1dc20d9032d6d49ed051ca0b/Makefile#L547 Maybe the string concatenation with the quotes cause it. At Travis it seems to work fine. I've worked it around by specifying manually LUA_INC=/usr/include/lua5.3/: make -j3 CC=$CC V=0 TARGET=$TARGET $FLAGS DEBUG_CFLAGS="$DEBUG_CFLAGS" LDFLAGS="$LDFLAGS -L$SSL_LIB -Wl,-rpath,$SSL_LIB" 51DEGREES_SRC="$FIFTYONEDEGREES_SRC" EXTRA_OBJS="$EXTRA_OBJS" LUA_INC=/usr/include/lua5.3/ > > cheers, > Ilya Shipitcin >
Re: Disabling regtests in Travis ?
On Fri, Jan 24, 2020 at 6:43 PM Willy Tarreau wrote: > On Fri, Jan 24, 2020 at 09:12:58PM +0500, ??? wrote: > > >> + - make reg-tests VTEST_PROGRAM=../vtest/vtest > > >> REGTESTS_TYPES=default,bug,devel > > >> > > > > > > let us try that. > > OK, now pushed. > > > > I will have a look at "racy" tests. > > > Maybe we'll enable them on Github Actions. > > > > > > > > the good thing about Github Actions, it is possible to attach own build > > agents. So, if we > > have dedicated hardware and we not want to depend on travis-ci > neighbours, > > it might be an option. > > That's good to know, even if I doubt we'd need it, at least it > opens possibilities. > The regtests run fine on my ARM64 VM. I run them daily. If HAProxy team decides to move to GitHub Actions and to use an external build agent for ARM64 then just ping me! Regards, Martin > > Willy > >
Re: [PATCH] introduce ARM64 travis-ci builds
Thank you, Илья! On Sun, Jan 19, 2020 at 9:20 AM Илья Шипицин wrote: > hello, > > sometimes arm64 builds fails, I think it is good chance to introduce > regular builds > and fix them. > > also, few small improvements. > > cheers, > Ilya Shipicin >
Re: ARM(64) builds
Hi, On Sat, Jan 18, 2020, 22:10 Илья Шипицин wrote: > tests on ARM64 randomly fail > https://travis-ci.com/chipitsine/haproxy/jobs/277236120 > > (after restart there's a good chance to success) > I have the same observation on TravisCI. I think the reason is that their arm64 instances are less powerful than the amd64 ones. At https://docs.travis-ci.com/user/multi-cpu-architectures it is said: While available to all Open Source repositories, the concurrency available for multiple CPU arch-based jobs is limited during the alpha period. Not sure how much limited it is though. Today this test failed once on my VM: #top TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (1.111) exit=2 1 tests failed, 0 tests skipped, 34 tests passed ## Gathering results ## ## Test case: reg-tests/seamless-reload/abns_socket.vtc ## ## test results in: "/tmp/haregtests-2020-01-19_08-06-39.45Hchw/vtc.8496.328d7f95" c1 HTTP rx failed (fd:6 read: Connection reset by peer) Makefile:964: recipe for target 'reg-tests' failed make: *** [reg-tests] Error 1 but the next 4 runs were successul. > сб, 18 янв. 2020 г. в 09:52, Martin Grigorov : > >> >> >> On Fri, Jan 17, 2020 at 11:17 PM Martin Grigorov < >> martin.grigo...@gmail.com> wrote: >> >>> >>> >>> On Fri, Jan 17, 2020, 23:12 William Lallemand >>> wrote: >>> >>>> On Fri, Jan 17, 2020 at 08:50:27PM +0200, Martin Grigorov wrote: >>>> > Testing with haproxy version: 2.2-dev0-70c5b0-123 >>>> >>>> This binary was built with code from 1 week ago, it's normal that the >>>> test does >>>> not work since the fix was made this week. >>>> >>> >>> I'm using the same steps to build HAProxy as from .travis-ci.yml >>> I guess I have to add "make clean" in the beginning. >>> I'll try it tomorrow! Thanks! >>> >> >> That was it! >> Everything is back to normal now! >> Thank you, William! >> >> >>> >>> >>>> -- >>>> William Lallemand >>>> >>>
Re: ARM(64) builds
On Fri, Jan 17, 2020 at 11:17 PM Martin Grigorov wrote: > > > On Fri, Jan 17, 2020, 23:12 William Lallemand > wrote: > >> On Fri, Jan 17, 2020 at 08:50:27PM +0200, Martin Grigorov wrote: >> > Testing with haproxy version: 2.2-dev0-70c5b0-123 >> >> This binary was built with code from 1 week ago, it's normal that the >> test does >> not work since the fix was made this week. >> > > I'm using the same steps to build HAProxy as from .travis-ci.yml > I guess I have to add "make clean" in the beginning. > I'll try it tomorrow! Thanks! > That was it! Everything is back to normal now! Thank you, William! > > >> -- >> William Lallemand >> >
Re: ARM(64) builds
On Fri, Jan 17, 2020, 23:12 William Lallemand wrote: > On Fri, Jan 17, 2020 at 08:50:27PM +0200, Martin Grigorov wrote: > > Testing with haproxy version: 2.2-dev0-70c5b0-123 > > This binary was built with code from 1 week ago, it's normal that the test > does > not work since the fix was made this week. > I'm using the same steps to build HAProxy as from .travis-ci.yml I guess I have to add "make clean" in the beginning. I'll try it tomorrow! Thanks! > -- > William Lallemand >
Re: ARM(64) builds
Hi Илья, On Fri, Jan 17, 2020 at 5:43 PM Илья Шипицин wrote: > > > пт, 17 янв. 2020 г. в 19:33, Martin Grigorov : > >> >> >> On Fri, Jan 17, 2020 at 4:13 PM Martin Grigorov < >> martin.grigo...@gmail.com> wrote: >> >>> Hi all, >>> >>> Today's build consistently fails on my ARM64 VM: >>> >>> ## Starting vtest ## >>> Testing with haproxy version: 2.2-dev0-70c5b0-123 >>> #top TEST reg-tests/mcli/mcli_start_progs.vtc FAILED (3.004) exit=2 >>> 1 tests failed, 0 tests skipped, 34 tests passed >>> ## Gathering results ## >>> ## Test case: reg-tests/mcli/mcli_start_progs.vtc ## >>> ## test results in: >>> "/tmp/haregtests-2020-01-17_14-01-45.SGkYcJ/vtc.12807.6adfff44" >>> h1 haproxy h1 PID file check failed: >>> h1 Bad exit status: 0x0100 exit 0x1 signal 0 core 0 >>> Makefile:964: recipe for target 'reg-tests' failed >>> make: *** [reg-tests] Error 1 >>> >> >> git bisect blaims this commit: >> >> 25b569302167e71b32e569a2366027e8e320e80a is the first bad commit >> commit 25b569302167e71b32e569a2366027e8e320e80a >> Author: William Lallemand >> Date: Tue Jan 14 15:38:43 2020 +0100 >> >> REGTEST: mcli/mcli_start_progs: start 2 programs >> >> This regtest tests the issue #446 by starting 2 programs and checking >> if >> they exist in the "show proc" of the master CLI. >> >> Should be backported as far as 2.0. >> >> >> https://travis-ci.com/haproxy/haproxy is green >> https://cirrus-ci.com/github/haproxy/haproxy is green >> and what is even more interesting is that >> https://travis-ci.org/martin-g/haproxy/builds (my fork with enabled >> ARM64 testing on TravisCI) also just passed (after few failures due to >> timing issues (I guess) >> > > > timing out might happen because of > https://github.com/haproxy/haproxy/commit/ac8147446c7a3d1aa607042bc782095b03bc8dc4 > your fork is 16 commits behind current master. try to rebase to master > Thanks! I've updated HAProxy on my ARM64 VM but didn't update my fork at GitHub so it was behind. I've just did it and the build again passed successfully: https://travis-ci.org/martin-g/haproxy/builds/638568217 > > >> >> >>> Regards, >>> Martin >>> >>> On Fri, Jan 17, 2020 at 9:22 AM Илья Шипицин >>> wrote: >>> >>>> привет! >>>> >>>> пт, 17 янв. 2020 г. в 11:42, Martin Grigorov >>> >: >>>> >>>>> Привет Илья, >>>>> >>>>> On Thu, Jan 16, 2020 at 10:37 AM Илья Шипицин >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> чт, 16 янв. 2020 г. в 13:26, Martin Grigorov < >>>>>> martin.grigo...@gmail.com>: >>>>>> >>>>>>> Hi Илья, >>>>>>> >>>>>>> On Thu, Jan 16, 2020 at 10:19 AM Илья Шипицин >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> Hello, Martin! >>>>>>>> >>>>>>>> btw, just curious, how is Apache Foundation (or you personally) >>>>>>>> interested in all that ? >>>>>>>> please do not blame me, I really like to know. >>>>>>>> >>>>>>> >>>>>> ok, so you work in some company that is interested in haproxy on >>>>>> ARM64. >>>>>> you do not want to tell it's name, at least is it legal ? is it >>>>>> related to some government ? >>>>>> if "no" and "no", I guess most people won't ask any more questions :) >>>>>> >>>>> >>>>> It is legal and I do not work for a government of any country! >>>>> >>>>> >>>>>> >>>>>> personally, I do not work at Haproxy Inc, I use haproxy, sometimes I >>>>>> contribute to it. >>>>>> Please do not consider me as an "official representative". >>>>>> >>>>>> >>>>>> I'm interested in testing haproxy on ARM64, I planned to do so. I can >>>>>> priorierize it according to your interest to it. >>>>>> and yes, I can accept hardware donati
Re: ARM(64) builds
Hi William,
On Fri, Jan 17, 2020 at 4:46 PM William Lallemand
wrote:
> On Fri, Jan 17, 2020 at 04:33:22PM +0200, Martin Grigorov wrote:
> >
> > git bisect blaims this commit:
> >
> > 25b569302167e71b32e569a2366027e8e320e80a is the first bad commit
> > commit 25b569302167e71b32e569a2366027e8e320e80a
> > Author: William Lallemand
> > Date: Tue Jan 14 15:38:43 2020 +0100
> >
> > REGTEST: mcli/mcli_start_progs: start 2 programs
>
> Well that's the commit which introduces the vtc file so that's normal.
>
> >
> > https://travis-ci.com/haproxy/haproxy is green
> > https://cirrus-ci.com/github/haproxy/haproxy is green
> > and what is even more interesting is that
> > https://travis-ci.org/martin-g/haproxy/builds (my fork with enabled
> ARM64
> > testing on TravisCI) also just passed (after few failures due to timing
> > issues (I guess)
>
> I don't see anything regarding mcli_start_progs.vtc in this links, can you
> provide the output of
> `make reg-tests -- --debug reg-tests/mcli/mcli_start_progs.vtc` ?
>
Please find attached the output.
this two lines look bad in it:
*** h1 debug|[ALERT] 016/184150 (7188) : parsing [cur--1:0] : proxy
'MASTER', another server named 'cur--1' was already defined at line 0,
please use distinct names.
*** h1 debug|[ALERT] 016/184150 (7188) : Fatal errors found in
configuration.
$ grep -rnH 'cur'
/tmp/haregtests-2020-01-17_18-41-50.A6zVLb/vtc.7182.700b0bcd/
/tmp/haregtests-2020-01-17_18-41-50.A6zVLb/vtc.7182.700b0bcd/LOG:71:*** h1
debug|[ALERT] 016/184150 (7188) : parsing [cur--1:0] : proxy 'MASTER',
another server named 'cur--1' was already defined at line 0, please use
distinct names.
│ File:
/tmp/haregtests-2020-01-17_18-41-50.A6zVLb/vtc.7182.700b0bcd/h1/cfg
───┼─
1 │ global
2 │ stats socket
"/tmp/haregtests-2020-01-17_18-41-50.A6zVLb/vtc.7182.700b0bcd/h1/stats.sock"
level admin mode 600
3 │ stats socket "fd@${cli}" level admin
4 │
5 │ global
6 │ nbproc 1
7 │ defaults
8 │ mode http
9 │ option http-use-htx
10 │ timeout connect 1s
11 │ timeout client 1s
12 │ timeout server 1s
13 │
14 │ frontend myfrontend
15 │ bind "fd@${my_fe}"
16 │ default_backend test
17 │
18 │ backend test
19 │ server www1 127.0.0.1:39247
20 │
21 │ program foo
22 │ command sleep 10
23 │
24 │ program bar
25 │ command sleep 10
Please let me know if you need more information!
Thanks
>
> --
> William Lallemand
>
env VTEST_PROGRAM=../vtest/vtest make reg-tests -- --debug
reg-tests/mcli/mcli_start_progs.vtc
## Preparing to run tests ##
Testing with haproxy version: 2.2-dev0-70c5b0-123
Target : linux-glibc
Options : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT -PCRE2
-PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE
-STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL
+GETADDRINFO -OPENSSL -LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY
+TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER
+PRCTL +THREAD_DUMP -EVPORTS
## Gathering tests to run ##
Add test: reg-tests/mcli/mcli_start_progs.vtc
## Starting vtest ##
Testing with haproxy version: 2.2-dev0-70c5b0-123
dT 0.000
*top TEST reg-tests/mcli/mcli_start_progs.vtc starting
top extmacro def pwd=/home/ubuntu/git/haproxy/haproxy
top extmacro def no-htx=
top extmacro def localhost=127.0.0.1
top extmacro def bad_backend=127.0.0.1 45249
top extmacro def bad_ip=192.0.2.255
top macro def testdir=/home/ubuntu/git/haproxy/haproxy/reg-tests/mcli
top macro def
tmpdir=/tmp/haregtests-2020-01-17_18-41-50.A6zVLb/vtc.7182.700b0bcd
** top === varnishtest "Try to start a master CLI with 2 programs"
*top VTEST Try to start a master CLI with 2 programs
** top === feature ignore_unknown_macro
** top === server s1 {
** s1 Starting server
s1 macro def s1_addr=127.0.0.1
s1 macro def s1_port=39247
s1 macro def s1_sock=127.0.0.1 39247
*s1 Listen on 127.0.0.1 39247
** top === haproxy h1 -W -S -conf {
** s1 Started on 127.0.0.1 39247 (1 iterations)
h1 macro def h1_closed_sock=127.0.0.1 33279
h1 macro def h1_closed_addr=127.0.0.1
h1 macro def h1_closed_port=33279
dT
Re: ARM(64) builds
On Fri, Jan 17, 2020 at 4:13 PM Martin Grigorov wrote: > Hi all, > > Today's build consistently fails on my ARM64 VM: > > ## Starting vtest ## > Testing with haproxy version: 2.2-dev0-70c5b0-123 > #top TEST reg-tests/mcli/mcli_start_progs.vtc FAILED (3.004) exit=2 > 1 tests failed, 0 tests skipped, 34 tests passed > ## Gathering results ## > ## Test case: reg-tests/mcli/mcli_start_progs.vtc ## > ## test results in: > "/tmp/haregtests-2020-01-17_14-01-45.SGkYcJ/vtc.12807.6adfff44" > h1 haproxy h1 PID file check failed: > h1 Bad exit status: 0x0100 exit 0x1 signal 0 core 0 > Makefile:964: recipe for target 'reg-tests' failed > make: *** [reg-tests] Error 1 > git bisect blaims this commit: 25b569302167e71b32e569a2366027e8e320e80a is the first bad commit commit 25b569302167e71b32e569a2366027e8e320e80a Author: William Lallemand Date: Tue Jan 14 15:38:43 2020 +0100 REGTEST: mcli/mcli_start_progs: start 2 programs This regtest tests the issue #446 by starting 2 programs and checking if they exist in the "show proc" of the master CLI. Should be backported as far as 2.0. https://travis-ci.com/haproxy/haproxy is green https://cirrus-ci.com/github/haproxy/haproxy is green and what is even more interesting is that https://travis-ci.org/martin-g/haproxy/builds (my fork with enabled ARM64 testing on TravisCI) also just passed (after few failures due to timing issues (I guess) > Regards, > Martin > > On Fri, Jan 17, 2020 at 9:22 AM Илья Шипицин wrote: > >> привет! >> >> пт, 17 янв. 2020 г. в 11:42, Martin Grigorov : >> >>> Привет Илья, >>> >>> On Thu, Jan 16, 2020 at 10:37 AM Илья Шипицин >>> wrote: >>> >>>> >>>> >>>> чт, 16 янв. 2020 г. в 13:26, Martin Grigorov >>> >: >>>> >>>>> Hi Илья, >>>>> >>>>> On Thu, Jan 16, 2020 at 10:19 AM Илья Шипицин >>>>> wrote: >>>>> >>>>>> >>>>>> Hello, Martin! >>>>>> >>>>>> btw, just curious, how is Apache Foundation (or you personally) >>>>>> interested in all that ? >>>>>> please do not blame me, I really like to know. >>>>>> >>>>> >>>> ok, so you work in some company that is interested in haproxy on ARM64. >>>> you do not want to tell it's name, at least is it legal ? is it related >>>> to some government ? >>>> if "no" and "no", I guess most people won't ask any more questions :) >>>> >>> >>> It is legal and I do not work for a government of any country! >>> >>> >>>> >>>> personally, I do not work at Haproxy Inc, I use haproxy, sometimes I >>>> contribute to it. >>>> Please do not consider me as an "official representative". >>>> >>>> >>>> I'm interested in testing haproxy on ARM64, I planned to do so. I can >>>> priorierize it according to your interest to it. >>>> and yes, I can accept hardware donation (even considering I'm not part >>>> of Haproxy Inc). >>>> >>>> also, from my point of view, what would be really useful in your case >>>> is testing not just official reg-tests, but also >>>> your configs. reg-tests cover only partially. If you enable clang asan >>>> in your own workload there's a chance to catch >>>> something interesting (or, to make sure your own workload is ok). I can >>>> help with that as well. >>>> >>> >>> Thanks for the offer! >>> I've discussed it with my managers. >>> Our offer is to donate a VM that could be used as an official CI agent >>> for the HAProxy project long term. >>> >> >> I'd split this into short term and long term approach. >> >> if you need to start any time soon, I'd focus on your own workload >> testing first. I would build stand which emulates >> your production workload, compile haproxy using clang address sanitizer >> and give it a try (functional testing, load testing, ...) >> >> I can help with that. >> >> As for long term solution, currently haproxy simply cannot attach any >> dedicated build agent to their CI. travis-ci does not allow >> attaching dedicated agents. And haproxy team is very conservative in >> adding new CI servers. >> >> I think, I will add arm64 (most prob
Re: ARM(64) builds
Hi all, Today's build consistently fails on my ARM64 VM: ## Starting vtest ## Testing with haproxy version: 2.2-dev0-70c5b0-123 #top TEST reg-tests/mcli/mcli_start_progs.vtc FAILED (3.004) exit=2 1 tests failed, 0 tests skipped, 34 tests passed ## Gathering results ## ## Test case: reg-tests/mcli/mcli_start_progs.vtc ## ## test results in: "/tmp/haregtests-2020-01-17_14-01-45.SGkYcJ/vtc.12807.6adfff44" h1 haproxy h1 PID file check failed: h1 Bad exit status: 0x0100 exit 0x1 signal 0 core 0 Makefile:964: recipe for target 'reg-tests' failed make: *** [reg-tests] Error 1 Regards, Martin On Fri, Jan 17, 2020 at 9:22 AM Илья Шипицин wrote: > привет! > > пт, 17 янв. 2020 г. в 11:42, Martin Grigorov : > >> Привет Илья, >> >> On Thu, Jan 16, 2020 at 10:37 AM Илья Шипицин >> wrote: >> >>> >>> >>> чт, 16 янв. 2020 г. в 13:26, Martin Grigorov >> >: >>> >>>> Hi Илья, >>>> >>>> On Thu, Jan 16, 2020 at 10:19 AM Илья Шипицин >>>> wrote: >>>> >>>>> >>>>> Hello, Martin! >>>>> >>>>> btw, just curious, how is Apache Foundation (or you personally) >>>>> interested in all that ? >>>>> please do not blame me, I really like to know. >>>>> >>>> >>> ok, so you work in some company that is interested in haproxy on ARM64. >>> you do not want to tell it's name, at least is it legal ? is it related >>> to some government ? >>> if "no" and "no", I guess most people won't ask any more questions :) >>> >> >> It is legal and I do not work for a government of any country! >> >> >>> >>> personally, I do not work at Haproxy Inc, I use haproxy, sometimes I >>> contribute to it. >>> Please do not consider me as an "official representative". >>> >>> >>> I'm interested in testing haproxy on ARM64, I planned to do so. I can >>> priorierize it according to your interest to it. >>> and yes, I can accept hardware donation (even considering I'm not part >>> of Haproxy Inc). >>> >>> also, from my point of view, what would be really useful in your case is >>> testing not just official reg-tests, but also >>> your configs. reg-tests cover only partially. If you enable clang asan >>> in your own workload there's a chance to catch >>> something interesting (or, to make sure your own workload is ok). I can >>> help with that as well. >>> >> >> Thanks for the offer! >> I've discussed it with my managers. >> Our offer is to donate a VM that could be used as an official CI agent >> for the HAProxy project long term. >> > > I'd split this into short term and long term approach. > > if you need to start any time soon, I'd focus on your own workload testing > first. I would build stand which emulates > your production workload, compile haproxy using clang address sanitizer > and give it a try (functional testing, load testing, ...) > > I can help with that. > > As for long term solution, currently haproxy simply cannot attach any > dedicated build agent to their CI. travis-ci does not allow > attaching dedicated agents. And haproxy team is very conservative in > adding new CI servers. > > I think, I will add arm64 (most probably openssl-1.1.1 only for now) soon. > Also, I'm going to investigate your libressl failures. > > so, dedicated vm definetly will help in troubleshooting issues, for manual > builds. It would save bunch of time. I do not mind if you will > add my ssh to that vm. > > > also, I requested access to Linaro. > > >> >> You can use Linaro for short term testing though. >> https://www.linaro.cloud/ >> Here you can request free VM for short periods: >> https://servicedesk.linaro.org/servicedesk/customer/portal/19/create/265 >> P.S. Linaro is not my employer! >> >> >> Regards, >> Martin >> >> >>> >>>> >>>>> >>>> @apache.org is just one of my several emails. And it is the default >>>> one in my email client. >>>> ASF is not related anyhow to my participation here. >>>> If I used my work email then it might look like some kind of >>>> advertisement. I'd like to avoid that! >>>> Next time I will use my @gmail.com one, as more neutral. Actually I've >>>> used the GMail one when registering to
Re: ARM(64) builds
Привет Илья, On Thu, Jan 16, 2020 at 10:37 AM Илья Шипицин wrote: > > > чт, 16 янв. 2020 г. в 13:26, Martin Grigorov : > >> Hi Илья, >> >> On Thu, Jan 16, 2020 at 10:19 AM Илья Шипицин >> wrote: >> >>> >>> Hello, Martin! >>> >>> btw, just curious, how is Apache Foundation (or you personally) >>> interested in all that ? >>> please do not blame me, I really like to know. >>> >> > ok, so you work in some company that is interested in haproxy on ARM64. > you do not want to tell it's name, at least is it legal ? is it related to > some government ? > if "no" and "no", I guess most people won't ask any more questions :) > It is legal and I do not work for a government of any country! > > personally, I do not work at Haproxy Inc, I use haproxy, sometimes I > contribute to it. > Please do not consider me as an "official representative". > > > I'm interested in testing haproxy on ARM64, I planned to do so. I can > priorierize it according to your interest to it. > and yes, I can accept hardware donation (even considering I'm not part of > Haproxy Inc). > > also, from my point of view, what would be really useful in your case is > testing not just official reg-tests, but also > your configs. reg-tests cover only partially. If you enable clang asan in > your own workload there's a chance to catch > something interesting (or, to make sure your own workload is ok). I can > help with that as well. > Thanks for the offer! I've discussed it with my managers. Our offer is to donate a VM that could be used as an official CI agent for the HAProxy project long term. You can use Linaro for short term testing though. https://www.linaro.cloud/ Here you can request free VM for short periods: https://servicedesk.linaro.org/servicedesk/customer/portal/19/create/265 P.S. Linaro is not my employer! Regards, Martin > >> >>> >> @apache.org is just one of my several emails. And it is the default one >> in my email client. >> ASF is not related anyhow to my participation here. >> If I used my work email then it might look like some kind of >> advertisement. I'd like to avoid that! >> Next time I will use my @gmail.com one, as more neutral. Actually I've >> used the GMail one when registering to this mailing list, so probably the >> post from @apache has been moderated. I'll be more careful next time! >> >> Thanks, Илья! >> >> Regards, >> Martin >> >> >>> >>> чт, 16 янв. 2020 г. в 12:32, Martin Grigorov : >>> >>>> Hello HAProxy developers, >>>> >>>> At work we are going to use more and more ARM64 based servers and >>>> HAProxy is one of the main products we rely on. >>>> So I went checking whether HAProxy project has a CI environment for >>>> testing on ARM architecture. >>>> >>> >>> we are looking towards >>> https://docs.travis-ci.com/user/multi-cpu-architectures >>> >>> >>>> I've found this recent discussion: >>>> https://www.mail-archive.com/haproxy@formilux.org/msg35302.html (I >>>> didn't find a way how to continue on the same mail thread, so I'm starting >>>> a new one. Apologies for that!). >>>> >>> >>> I played with arm64 for a while, the issue I encountered is travis-ci >>> cache key, i.e. we cache openssl builds between our builds. >>> so travis used the same cache key for both amd64 and arm64 builds (this >>> might have changed recently, I did not check yet) >>> >>> arm64 is in my queue (as well as recent s390x arch from travis), hope to >>> get back to it within month or so. >>> >>> >>>> From this discussion and from >>>> https://github.com/haproxy/haproxy/blob/master/.travis.yml I >>>> understand that there is no public CI in use (i.e. TravisCI or CirrusCI) >>>> but some of the developers run some tests locally regularly. >>>> >>> >>> it is not completely true. >>> there's public CI. we do not use github PR machinery, so sometimes tests >>> fail after push to master branch. it is considered as ok, failures are >>> fixed pretty fast. >>> for example, see >>> https://www.mail-archive.com/haproxy@formilux.org/msg35910.html >>> it was just perfect, failure detected using CI and fixed within few >>> days. no customers affected. >>> >>> >>>> I've forked the project and tested on TravisCI ( >>>> https://travis-ci
Re: ARM(64) builds
Hi Илья, On Thu, Jan 16, 2020 at 10:19 AM Илья Шипицин wrote: > > Hello, Martin! > > btw, just curious, how is Apache Foundation (or you personally) interested > in all that ? > please do not blame me, I really like to know. > > @apache.org is just one of my several emails. And it is the default one in my email client. ASF is not related anyhow to my participation here. If I used my work email then it might look like some kind of advertisement. I'd like to avoid that! Next time I will use my @gmail.com one, as more neutral. Actually I've used the GMail one when registering to this mailing list, so probably the post from @apache has been moderated. I'll be more careful next time! Thanks, Илья! Regards, Martin > > чт, 16 янв. 2020 г. в 12:32, Martin Grigorov : > >> Hello HAProxy developers, >> >> At work we are going to use more and more ARM64 based servers and HAProxy >> is one of the main products we rely on. >> So I went checking whether HAProxy project has a CI environment for >> testing on ARM architecture. >> > > we are looking towards > https://docs.travis-ci.com/user/multi-cpu-architectures > > >> I've found this recent discussion: >> https://www.mail-archive.com/haproxy@formilux.org/msg35302.html (I >> didn't find a way how to continue on the same mail thread, so I'm starting >> a new one. Apologies for that!). >> > > I played with arm64 for a while, the issue I encountered is travis-ci > cache key, i.e. we cache openssl builds between our builds. > so travis used the same cache key for both amd64 and arm64 builds (this > might have changed recently, I did not check yet) > > arm64 is in my queue (as well as recent s390x arch from travis), hope to > get back to it within month or so. > > >> From this discussion and from >> https://github.com/haproxy/haproxy/blob/master/.travis.yml I understand >> that there is no public CI in use (i.e. TravisCI or CirrusCI) but some of >> the developers run some tests locally regularly. >> > > it is not completely true. > there's public CI. we do not use github PR machinery, so sometimes tests > fail after push to master branch. it is considered as ok, failures are > fixed pretty fast. > for example, see > https://www.mail-archive.com/haproxy@formilux.org/msg35910.html > it was just perfect, failure detected using CI and fixed within few days. > no customers affected. > > >> I've forked the project and tested on TravisCI ( >> https://travis-ci.org/martin-g/haproxy/builds) but unfortunately the >> builds were not very stable: >> 1) some tests fail sometimes. I guess it is because of some timing issues >> For example: >> - https://travis-ci.org/martin-g/haproxy/jobs/636745241 >> - https://travis-ci.org/martin-g/haproxy/jobs/636750676 >> - https://travis-ci.org/martin-g/haproxy/jobs/636763346 >> > > that's very interesting. I'll have a look. > > > >> 2) There was some weird issue on testing with LibreSSL >> The output redirect at >> https://github.com/haproxy/haproxy/blob/bb9da0b8e23c46caab34fe6005b66fa8065fe3ac/.travis.yml#L96 >> for >> some reason got stuck the build. I've removed temporarily the output >> redirects and then it passed. So, it looks like some issue with TravisCI >> environment. >> > > arm64 is slower, I guess we should add "*travis*_*wait* *30" *to > build-ssl.sh script > thank for the hint > > >> >> In addition I've run the build and tests on one of our machines and all >> was OK! >> >> My question to you is: Are you happy with your current way of testing ARM >> architectures or you want to add more ? >> Here are some options: >> 1) enable TravisCI >> > > already done > > https://travis-ci.com/haproxy/haproxy > > >> 2) my company is willing to donate an ARM64 based VM, if you are >> interested. >> > > I do not work at Haproxy Inc :) > Willy ? > > >> You will have a SSH access and a user with sudo permissions to install >> anything that is needed. >> The spec is aarch64 8 cores CPU 2GHz (Kunpeng), 16GB RAM, 500G disk space >> and 5M network bandwidth. The OS could be any of CentOS 7.4/7.5/7.6, >> EulerOS 2.8, Fedora 29, OpenSuse 15.0 and Ubuntu 16.04/18.04. >> >> In both cases it will be ARM64. From the earlier mail discussion I >> understand you would prefer ARM32. >> > > as for myself, I prefer both arm64 and arm32. > however, both AMD64 and ARM64 are the same 64 bits. both of them are > little-endian. but you mentioned at least 3 builds with ARM64 failing (we > have those tests passing on AMD64)! > > > >> >> Kind regards, >> Martin >> >
ARM(64) builds
Hello HAProxy developers, At work we are going to use more and more ARM64 based servers and HAProxy is one of the main products we rely on. So I went checking whether HAProxy project has a CI environment for testing on ARM architecture. I've found this recent discussion: https://www.mail-archive.com/haproxy@formilux.org/msg35302.html (I didn't find a way how to continue on the same mail thread, so I'm starting a new one. Apologies for that!). >From this discussion and from https://github.com/haproxy/haproxy/blob/master/.travis.yml I understand that there is no public CI in use (i.e. TravisCI or CirrusCI) but some of the developers run some tests locally regularly. I've forked the project and tested on TravisCI ( https://travis-ci.org/martin-g/haproxy/builds) but unfortunately the builds were not very stable: 1) some tests fail sometimes. I guess it is because of some timing issues For example: - https://travis-ci.org/martin-g/haproxy/jobs/636745241 - https://travis-ci.org/martin-g/haproxy/jobs/636750676 - https://travis-ci.org/martin-g/haproxy/jobs/636763346 2) There was some weird issue on testing with LibreSSL The output redirect at https://github.com/haproxy/haproxy/blob/bb9da0b8e23c46caab34fe6005b66fa8065fe3ac/.travis.yml#L96 for some reason got stuck the build. I've removed temporarily the output redirects and then it passed. So, it looks like some issue with TravisCI environment. In addition I've run the build and tests on one of our machines and all was OK! My question to you is: Are you happy with your current way of testing ARM architectures or you want to add more ? Here are some options: 1) enable TravisCI 2) my company is willing to donate an ARM64 based VM, if you are interested. You will have a SSH access and a user with sudo permissions to install anything that is needed. The spec is aarch64 8 cores CPU 2GHz (Kunpeng), 16GB RAM, 500G disk space and 5M network bandwidth. The OS could be any of CentOS 7.4/7.5/7.6, EulerOS 2.8, Fedora 29, OpenSuse 15.0 and Ubuntu 16.04/18.04. In both cases it will be ARM64. From the earlier mail discussion I understand you would prefer ARM32. Kind regards, Martin
