On Fri, Jan 17, 2020 at 4:13 PM Martin Grigorov <[email protected]> wrote:
> Hi all, > > Today's build consistently fails on my ARM64 VM: > > ########################## Starting vtest ########################## > Testing with haproxy version: 2.2-dev0-70c5b0-123 > # top TEST reg-tests/mcli/mcli_start_progs.vtc FAILED (3.004) exit=2 > 1 tests failed, 0 tests skipped, 34 tests passed > ########################## Gathering results ########################## > ###### Test case: reg-tests/mcli/mcli_start_progs.vtc ###### > ## test results in: > "/tmp/haregtests-2020-01-17_14-01-45.SGkYcJ/vtc.12807.6adfff44" > ---- h1 haproxy h1 PID file check failed: > ---- h1 Bad exit status: 0x0100 exit 0x1 signal 0 core 0 > Makefile:964: recipe for target 'reg-tests' failed > make: *** [reg-tests] Error 1 > git bisect blaims this commit: 25b569302167e71b32e569a2366027e8e320e80a is the first bad commit commit 25b569302167e71b32e569a2366027e8e320e80a Author: William Lallemand <[email protected]> Date: Tue Jan 14 15:38:43 2020 +0100 REGTEST: mcli/mcli_start_progs: start 2 programs This regtest tests the issue #446 by starting 2 programs and checking if they exist in the "show proc" of the master CLI. Should be backported as far as 2.0. https://travis-ci.com/haproxy/haproxy is green https://cirrus-ci.com/github/haproxy/haproxy is green and what is even more interesting is that https://travis-ci.org/martin-g/haproxy/builds (my fork with enabled ARM64 testing on TravisCI) also just passed (after few failures due to timing issues (I guess) > Regards, > Martin > > On Fri, Jan 17, 2020 at 9:22 AM Илья Шипицин <[email protected]> wrote: > >> привет! >> >> пт, 17 янв. 2020 г. в 11:42, Martin Grigorov <[email protected]>: >> >>> Привет Илья, >>> >>> On Thu, Jan 16, 2020 at 10:37 AM Илья Шипицин <[email protected]> >>> wrote: >>> >>>> >>>> >>>> чт, 16 янв. 2020 г. в 13:26, Martin Grigorov <[email protected] >>>> >: >>>> >>>>> Hi Илья, >>>>> >>>>> On Thu, Jan 16, 2020 at 10:19 AM Илья Шипицин <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>> Hello, Martin! >>>>>> >>>>>> btw, just curious, how is Apache Foundation (or you personally) >>>>>> interested in all that ? >>>>>> please do not blame me, I really like to know. >>>>>> >>>>> >>>> ok, so you work in some company that is interested in haproxy on ARM64. >>>> you do not want to tell it's name, at least is it legal ? is it related >>>> to some government ? >>>> if "no" and "no", I guess most people won't ask any more questions :) >>>> >>> >>> It is legal and I do not work for a government of any country! >>> >>> >>>> >>>> personally, I do not work at Haproxy Inc, I use haproxy, sometimes I >>>> contribute to it. >>>> Please do not consider me as an "official representative". >>>> >>>> >>>> I'm interested in testing haproxy on ARM64, I planned to do so. I can >>>> priorierize it according to your interest to it. >>>> and yes, I can accept hardware donation (even considering I'm not part >>>> of Haproxy Inc). >>>> >>>> also, from my point of view, what would be really useful in your case >>>> is testing not just official reg-tests, but also >>>> your configs. reg-tests cover only partially. If you enable clang asan >>>> in your own workload there's a chance to catch >>>> something interesting (or, to make sure your own workload is ok). I can >>>> help with that as well. >>>> >>> >>> Thanks for the offer! >>> I've discussed it with my managers. >>> Our offer is to donate a VM that could be used as an official CI agent >>> for the HAProxy project long term. >>> >> >> I'd split this into short term and long term approach. >> >> if you need to start any time soon, I'd focus on your own workload >> testing first. I would build stand which emulates >> your production workload, compile haproxy using clang address sanitizer >> and give it a try (functional testing, load testing, ...) >> >> I can help with that. >> >> As for long term solution, currently haproxy simply cannot attach any >> dedicated build agent to their CI. travis-ci does not allow >> attaching dedicated agents. And haproxy team is very conservative in >> adding new CI servers. >> >> I think, I will add arm64 (most probably openssl-1.1.1 only for now) >> soon. Also, I'm going to investigate your libressl failures. >> >> so, dedicated vm definetly will help in troubleshooting issues, for >> manual builds. It would save bunch of time. I do not mind if you will >> add my ssh to that vm. >> >> >> also, I requested access to Linaro. >> >> >>> >>> You can use Linaro for short term testing though. >>> https://www.linaro.cloud/ >>> Here you can request free VM for short periods: >>> https://servicedesk.linaro.org/servicedesk/customer/portal/19/create/265 >>> P.S. Linaro is not my employer! >>> >>> >>> Regards, >>> Martin >>> >>> >>>> >>>>> >>>>>> >>>>> @apache.org is just one of my several emails. And it is the default >>>>> one in my email client. >>>>> ASF is not related anyhow to my participation here. >>>>> If I used my work email then it might look like some kind of >>>>> advertisement. I'd like to avoid that! >>>>> Next time I will use my @gmail.com one, as more neutral. Actually >>>>> I've used the GMail one when registering to this mailing list, so probably >>>>> the post from @apache has been moderated. I'll be more careful next time! >>>>> >>>>> Thanks, Илья! >>>>> >>>>> Regards, >>>>> Martin >>>>> >>>>> >>>>>> >>>>>> чт, 16 янв. 2020 г. в 12:32, Martin Grigorov <[email protected]>: >>>>>> >>>>>>> Hello HAProxy developers, >>>>>>> >>>>>>> At work we are going to use more and more ARM64 based servers and >>>>>>> HAProxy is one of the main products we rely on. >>>>>>> So I went checking whether HAProxy project has a CI environment for >>>>>>> testing on ARM architecture. >>>>>>> >>>>>> >>>>>> we are looking towards >>>>>> https://docs.travis-ci.com/user/multi-cpu-architectures >>>>>> >>>>>> >>>>>>> I've found this recent discussion: >>>>>>> https://www.mail-archive.com/[email protected]/msg35302.html (I >>>>>>> didn't find a way how to continue on the same mail thread, so I'm >>>>>>> starting >>>>>>> a new one. Apologies for that!). >>>>>>> >>>>>> >>>>>> I played with arm64 for a while, the issue I encountered is travis-ci >>>>>> cache key, i.e. we cache openssl builds between our builds. >>>>>> so travis used the same cache key for both amd64 and arm64 builds >>>>>> (this might have changed recently, I did not check yet) >>>>>> >>>>>> arm64 is in my queue (as well as recent s390x arch from travis), hope >>>>>> to get back to it within month or so. >>>>>> >>>>>> >>>>>>> From this discussion and from >>>>>>> https://github.com/haproxy/haproxy/blob/master/.travis.yml I >>>>>>> understand that there is no public CI in use (i.e. TravisCI or CirrusCI) >>>>>>> but some of the developers run some tests locally regularly. >>>>>>> >>>>>> >>>>>> it is not completely true. >>>>>> there's public CI. we do not use github PR machinery, so sometimes >>>>>> tests fail after push to master branch. it is considered as ok, failures >>>>>> are fixed pretty fast. >>>>>> for example, see >>>>>> https://www.mail-archive.com/[email protected]/msg35910.html >>>>>> it was just perfect, failure detected using CI and fixed within few >>>>>> days. no customers affected. >>>>>> >>>>>> >>>>>>> I've forked the project and tested on TravisCI ( >>>>>>> https://travis-ci.org/martin-g/haproxy/builds) but unfortunately >>>>>>> the builds were not very stable: >>>>>>> 1) some tests fail sometimes. I guess it is because of some timing >>>>>>> issues >>>>>>> For example: >>>>>>> - https://travis-ci.org/martin-g/haproxy/jobs/636745241 >>>>>>> - https://travis-ci.org/martin-g/haproxy/jobs/636750676 >>>>>>> - https://travis-ci.org/martin-g/haproxy/jobs/636763346 >>>>>>> >>>>>> >>>>>> that's very interesting. I'll have a look. >>>>>> >>>>>> >>>>>> >>>>>>> 2) There was some weird issue on testing with LibreSSL >>>>>>> The output redirect at >>>>>>> https://github.com/haproxy/haproxy/blob/bb9da0b8e23c46caab34fe6005b66fa8065fe3ac/.travis.yml#L96 >>>>>>> for >>>>>>> some reason got stuck the build. I've removed temporarily the output >>>>>>> redirects and then it passed. So, it looks like some issue with TravisCI >>>>>>> environment. >>>>>>> >>>>>> >>>>>> arm64 is slower, I guess we should add "*travis*_*wait* *30" *to >>>>>> build-ssl.sh script >>>>>> thank for the hint >>>>>> >>>>>> >>>>>>> >>>>>>> In addition I've run the build and tests on one of our machines and >>>>>>> all was OK! >>>>>>> >>>>>>> My question to you is: Are you happy with your current way of >>>>>>> testing ARM architectures or you want to add more ? >>>>>>> Here are some options: >>>>>>> 1) enable TravisCI >>>>>>> >>>>>> >>>>>> already done >>>>>> >>>>>> https://travis-ci.com/haproxy/haproxy >>>>>> >>>>>> >>>>>>> 2) my company is willing to donate an ARM64 based VM, if you are >>>>>>> interested. >>>>>>> >>>>>> >>>>>> I do not work at Haproxy Inc :) >>>>>> Willy ? >>>>>> >>>>>> >>>>>>> You will have a SSH access and a user with sudo permissions to >>>>>>> install anything that is needed. >>>>>>> The spec is aarch64 8 cores CPU 2GHz (Kunpeng), 16GB RAM, 500G disk >>>>>>> space and 5M network bandwidth. The OS could be any of CentOS >>>>>>> 7.4/7.5/7.6, >>>>>>> EulerOS 2.8, Fedora 29, OpenSuse 15.0 and Ubuntu 16.04/18.04. >>>>>>> >>>>>>> In both cases it will be ARM64. From the earlier mail discussion I >>>>>>> understand you would prefer ARM32. >>>>>>> >>>>>> >>>>>> as for myself, I prefer both arm64 and arm32. >>>>>> however, both AMD64 and ARM64 are the same 64 bits. both of them are >>>>>> little-endian. but you mentioned at least 3 builds with ARM64 failing (we >>>>>> have those tests passing on AMD64)! >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Kind regards, >>>>>>> Martin >>>>>>> >>>>>>
