Hi all, Today's build consistently fails on my ARM64 VM:
########################## Starting vtest ########################## Testing with haproxy version: 2.2-dev0-70c5b0-123 # top TEST reg-tests/mcli/mcli_start_progs.vtc FAILED (3.004) exit=2 1 tests failed, 0 tests skipped, 34 tests passed ########################## Gathering results ########################## ###### Test case: reg-tests/mcli/mcli_start_progs.vtc ###### ## test results in: "/tmp/haregtests-2020-01-17_14-01-45.SGkYcJ/vtc.12807.6adfff44" ---- h1 haproxy h1 PID file check failed: ---- h1 Bad exit status: 0x0100 exit 0x1 signal 0 core 0 Makefile:964: recipe for target 'reg-tests' failed make: *** [reg-tests] Error 1 Regards, Martin On Fri, Jan 17, 2020 at 9:22 AM Илья Шипицин <[email protected]> wrote: > привет! > > пт, 17 янв. 2020 г. в 11:42, Martin Grigorov <[email protected]>: > >> Привет Илья, >> >> On Thu, Jan 16, 2020 at 10:37 AM Илья Шипицин <[email protected]> >> wrote: >> >>> >>> >>> чт, 16 янв. 2020 г. в 13:26, Martin Grigorov <[email protected] >>> >: >>> >>>> Hi Илья, >>>> >>>> On Thu, Jan 16, 2020 at 10:19 AM Илья Шипицин <[email protected]> >>>> wrote: >>>> >>>>> >>>>> Hello, Martin! >>>>> >>>>> btw, just curious, how is Apache Foundation (or you personally) >>>>> interested in all that ? >>>>> please do not blame me, I really like to know. >>>>> >>>> >>> ok, so you work in some company that is interested in haproxy on ARM64. >>> you do not want to tell it's name, at least is it legal ? is it related >>> to some government ? >>> if "no" and "no", I guess most people won't ask any more questions :) >>> >> >> It is legal and I do not work for a government of any country! >> >> >>> >>> personally, I do not work at Haproxy Inc, I use haproxy, sometimes I >>> contribute to it. >>> Please do not consider me as an "official representative". >>> >>> >>> I'm interested in testing haproxy on ARM64, I planned to do so. I can >>> priorierize it according to your interest to it. >>> and yes, I can accept hardware donation (even considering I'm not part >>> of Haproxy Inc). >>> >>> also, from my point of view, what would be really useful in your case is >>> testing not just official reg-tests, but also >>> your configs. reg-tests cover only partially. If you enable clang asan >>> in your own workload there's a chance to catch >>> something interesting (or, to make sure your own workload is ok). I can >>> help with that as well. >>> >> >> Thanks for the offer! >> I've discussed it with my managers. >> Our offer is to donate a VM that could be used as an official CI agent >> for the HAProxy project long term. >> > > I'd split this into short term and long term approach. > > if you need to start any time soon, I'd focus on your own workload testing > first. I would build stand which emulates > your production workload, compile haproxy using clang address sanitizer > and give it a try (functional testing, load testing, ...) > > I can help with that. > > As for long term solution, currently haproxy simply cannot attach any > dedicated build agent to their CI. travis-ci does not allow > attaching dedicated agents. And haproxy team is very conservative in > adding new CI servers. > > I think, I will add arm64 (most probably openssl-1.1.1 only for now) soon. > Also, I'm going to investigate your libressl failures. > > so, dedicated vm definetly will help in troubleshooting issues, for manual > builds. It would save bunch of time. I do not mind if you will > add my ssh to that vm. > > > also, I requested access to Linaro. > > >> >> You can use Linaro for short term testing though. >> https://www.linaro.cloud/ >> Here you can request free VM for short periods: >> https://servicedesk.linaro.org/servicedesk/customer/portal/19/create/265 >> P.S. Linaro is not my employer! >> >> >> Regards, >> Martin >> >> >>> >>>> >>>>> >>>> @apache.org is just one of my several emails. And it is the default >>>> one in my email client. >>>> ASF is not related anyhow to my participation here. >>>> If I used my work email then it might look like some kind of >>>> advertisement. I'd like to avoid that! >>>> Next time I will use my @gmail.com one, as more neutral. Actually I've >>>> used the GMail one when registering to this mailing list, so probably the >>>> post from @apache has been moderated. I'll be more careful next time! >>>> >>>> Thanks, Илья! >>>> >>>> Regards, >>>> Martin >>>> >>>> >>>>> >>>>> чт, 16 янв. 2020 г. в 12:32, Martin Grigorov <[email protected]>: >>>>> >>>>>> Hello HAProxy developers, >>>>>> >>>>>> At work we are going to use more and more ARM64 based servers and >>>>>> HAProxy is one of the main products we rely on. >>>>>> So I went checking whether HAProxy project has a CI environment for >>>>>> testing on ARM architecture. >>>>>> >>>>> >>>>> we are looking towards >>>>> https://docs.travis-ci.com/user/multi-cpu-architectures >>>>> >>>>> >>>>>> I've found this recent discussion: >>>>>> https://www.mail-archive.com/[email protected]/msg35302.html (I >>>>>> didn't find a way how to continue on the same mail thread, so I'm >>>>>> starting >>>>>> a new one. Apologies for that!). >>>>>> >>>>> >>>>> I played with arm64 for a while, the issue I encountered is travis-ci >>>>> cache key, i.e. we cache openssl builds between our builds. >>>>> so travis used the same cache key for both amd64 and arm64 builds >>>>> (this might have changed recently, I did not check yet) >>>>> >>>>> arm64 is in my queue (as well as recent s390x arch from travis), hope >>>>> to get back to it within month or so. >>>>> >>>>> >>>>>> From this discussion and from >>>>>> https://github.com/haproxy/haproxy/blob/master/.travis.yml I >>>>>> understand that there is no public CI in use (i.e. TravisCI or CirrusCI) >>>>>> but some of the developers run some tests locally regularly. >>>>>> >>>>> >>>>> it is not completely true. >>>>> there's public CI. we do not use github PR machinery, so sometimes >>>>> tests fail after push to master branch. it is considered as ok, failures >>>>> are fixed pretty fast. >>>>> for example, see >>>>> https://www.mail-archive.com/[email protected]/msg35910.html >>>>> it was just perfect, failure detected using CI and fixed within few >>>>> days. no customers affected. >>>>> >>>>> >>>>>> I've forked the project and tested on TravisCI ( >>>>>> https://travis-ci.org/martin-g/haproxy/builds) but unfortunately the >>>>>> builds were not very stable: >>>>>> 1) some tests fail sometimes. I guess it is because of some timing >>>>>> issues >>>>>> For example: >>>>>> - https://travis-ci.org/martin-g/haproxy/jobs/636745241 >>>>>> - https://travis-ci.org/martin-g/haproxy/jobs/636750676 >>>>>> - https://travis-ci.org/martin-g/haproxy/jobs/636763346 >>>>>> >>>>> >>>>> that's very interesting. I'll have a look. >>>>> >>>>> >>>>> >>>>>> 2) There was some weird issue on testing with LibreSSL >>>>>> The output redirect at >>>>>> https://github.com/haproxy/haproxy/blob/bb9da0b8e23c46caab34fe6005b66fa8065fe3ac/.travis.yml#L96 >>>>>> for >>>>>> some reason got stuck the build. I've removed temporarily the output >>>>>> redirects and then it passed. So, it looks like some issue with TravisCI >>>>>> environment. >>>>>> >>>>> >>>>> arm64 is slower, I guess we should add "*travis*_*wait* *30" *to >>>>> build-ssl.sh script >>>>> thank for the hint >>>>> >>>>> >>>>>> >>>>>> In addition I've run the build and tests on one of our machines and >>>>>> all was OK! >>>>>> >>>>>> My question to you is: Are you happy with your current way of testing >>>>>> ARM architectures or you want to add more ? >>>>>> Here are some options: >>>>>> 1) enable TravisCI >>>>>> >>>>> >>>>> already done >>>>> >>>>> https://travis-ci.com/haproxy/haproxy >>>>> >>>>> >>>>>> 2) my company is willing to donate an ARM64 based VM, if you are >>>>>> interested. >>>>>> >>>>> >>>>> I do not work at Haproxy Inc :) >>>>> Willy ? >>>>> >>>>> >>>>>> You will have a SSH access and a user with sudo permissions to >>>>>> install anything that is needed. >>>>>> The spec is aarch64 8 cores CPU 2GHz (Kunpeng), 16GB RAM, 500G disk >>>>>> space and 5M network bandwidth. The OS could be any of CentOS >>>>>> 7.4/7.5/7.6, >>>>>> EulerOS 2.8, Fedora 29, OpenSuse 15.0 and Ubuntu 16.04/18.04. >>>>>> >>>>>> In both cases it will be ARM64. From the earlier mail discussion I >>>>>> understand you would prefer ARM32. >>>>>> >>>>> >>>>> as for myself, I prefer both arm64 and arm32. >>>>> however, both AMD64 and ARM64 are the same 64 bits. both of them are >>>>> little-endian. but you mentioned at least 3 builds with ARM64 failing (we >>>>> have those tests passing on AMD64)! >>>>> >>>>> >>>>> >>>>>> >>>>>> Kind regards, >>>>>> Martin >>>>>> >>>>>
