Re: [PATCH] CI: travis-ci: disable arm64 builds

2021-08-09 Thread Martin Grigorov
Hi Илья,

On Mon, Aug 9, 2021 at 12:45 PM Илья Шипицин  wrote:

> I'm using arm64 in Oracle Cloud Ampere A1 Compute | Oracle
> <https://www.oracle.com/cloud/compute/arm/>
>

Yes!
OCI has an ARM64 VM in their free tier!
They also provide easy integration with GitHub Actions -
https://blogs.oracle.com/cloud-infrastructure/post/announcing-github-actions-arm-runners-for-the-arm-compute-platform-on-oracle-cloud-infrastructure
But the issue here is that GitHub Actions self-hosted runners are not
recommended for public/OSS projects because bad people can execute malware
on your VM via (meaningless) Pull Requests.


>
>
> also, I've found promising approach (using ARM on Github Actions)  Bump
> Bootstrap version from 5.0.2 to 5.1.0 · phpmyadmin/phpmyadmin@c90affe
> (github.com) <https://github.com/phpmyadmin/phpmyadmin/runs/3274334375>
>

This setup uses QEMU -
https://github.com/phpmyadmin/phpmyadmin/blob/c90affe793b56fb6f5e1c7eed676ec9031fb1480/.github/workflows/tests.yml#L51
.
It works but it is much slower than using a real ARM64 machine/VM.

Martin


> пн, 9 авг. 2021 г. в 14:29, Willy Tarreau :
>
>> Hi Martin,
>>
>> On Mon, Aug 09, 2021 at 11:04:34AM +0300, Martin Grigorov wrote:
>> > TravisCI just announced some improvements related to 'arch: arm64'
>> (using
>> > Equnix Metal machines) -
>> https://blog.travis-ci.com/2021-08-06-oss-equinix.
>>
>> Thanks for the info!
>>
>> > But I also had some similar problems with them recently and replaced the
>> > config with 'arch: arm64-graviton2; group: edge; virt: vm;', i.e. AWS
>> > Graviton2 machines. In my experience they behave more stable!
>>
>> Yeah, these machines are really fantastic. The real problem anyway is
>> likely that nowadays everyone is interested in testing on arm and very
>> few have one available, let alone even a cross-compiler, so I suspect
>> that these days a lot of people enable arm builds in such CI environments
>> because it's the only way they have to make sure their code builds there
>> at all.
>>
>> Cheers,
>> Willy
>>
>


Re: [PATCH] CI: travis-ci: disable arm64 builds

2021-08-09 Thread Martin Grigorov
Hi Willy,

On Mon, Aug 9, 2021 at 12:29 PM Willy Tarreau  wrote:

> Hi Martin,
>
> On Mon, Aug 09, 2021 at 11:04:34AM +0300, Martin Grigorov wrote:
> > TravisCI just announced some improvements related to 'arch: arm64' (using
> > Equnix Metal machines) -
> https://blog.travis-ci.com/2021-08-06-oss-equinix.
>
> Thanks for the info!
>
> > But I also had some similar problems with them recently and replaced the
> > config with 'arch: arm64-graviton2; group: edge; virt: vm;', i.e. AWS
> > Graviton2 machines. In my experience they behave more stable!
>
> Yeah, these machines are really fantastic. The real problem anyway is
> likely that nowadays everyone is interested in testing on arm and very
> few have one available, let alone even a cross-compiler, so I suspect
> that these days a lot of people enable arm builds in such CI environments
> because it's the only way they have to make sure their code builds there
> at all.
>

I am not sure whether you understood me. Or maybe I didn't understand you.
Anyway, let me rephrase:
TravisCI provides two types of ARM64 VMs - arm64 (powered by Equinix Metal)
and arm64-graviton2 (by AWS Graviton2).
You, as a user, can use any or both of them in your .travis.yml.
I prefer the AWS Graviton2 instances because there are less issues with
them.
So, instead of disabling the ARM64 CI job I suggest to try with
arm64-graviton2. Here is a sample setup for it -
https://github.com/apache/wicket/blob/270a5a43970cd975539331b21a34bd83a59c9c39/.travis.yml#L18-L22
More info about it at https://blog.travis-ci.com/2020-09-11-arm-on-aws

Martin


>
> Cheers,
> Willy
>


Re: [PATCH] CI: travis-ci: disable arm64 builds

2021-08-09 Thread Martin Grigorov
Hi,

On Sat, Aug 7, 2021 at 8:31 AM Willy Tarreau  wrote:

> Hi Ilya,
>
> On Tue, Aug 03, 2021 at 02:58:40PM +0500,  ??? wrote:
> > Hello,
> >
> > it looks like "something on travis-ci side".
> >
> > CC  src/raw_sock.o
> > gcc: fatal error: Killed signal terminated program cc1
> > compilation terminated.
> >
> > let us disable arm64 for a while.
>

TravisCI just announced some improvements related to 'arch: arm64' (using
Equnix Metal machines) - https://blog.travis-ci.com/2021-08-06-oss-equinix.
But I also had some similar problems with them recently and replaced the
config with 'arch: arm64-graviton2; group: edge; virt: vm;', i.e. AWS
Graviton2 machines. In my experience they behave more stable!

Regards,
Martin


>
> Yes I noticed a few of these lately, and sometimes it even looked like
> impossible downloads. I suspect that github assigns a maximum runtime
> to these VMs and that they're simply overloaded and victims of their
> success. I could be wrong of course.
>
> Now applied, thank you!
> Willy
>
>


Re: Load testing HAProxy 2.2 on x86_64 and aarch64 VMs

2020-07-23 Thread Martin Grigorov
Hi Илья,

I didn't have much success with Google Perf Tools.
I've tried both with adding '-lprofiler' to LDFLAGS when building HAProxy
and with LD_PRELOAD.
In both cases on ARM64 it fails with: terminated by signal SIGSEGV (Address
boundary error)

On x86_64 there is no such issue but the produced result file is always
empty.

$ ldd haproxy
linux-vdso.so.1 (0x7ffcfb5cd000)
libprofiler.so.0 => /usr/local/lib/libprofiler.so.0
(0x7f2115507000)<<<<<<<<<<<<<<<<<<
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1
(0x7f21154cc000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x7f21154b)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f21154aa000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f211549f000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x7f211547c000)
libssl.so.1.1 => /home/ubuntu/opt/lib/libssl.so.1.1
(0x7f21151e6000)
libcrypto.so.1.1 => /home/ubuntu/opt/lib/libcrypto.so.1.1
(0x7f2114cf4000)
liblua5.3.so.0 => /usr/lib/x86_64-linux-gnu/liblua5.3.so.0
(0x7f2114cb9000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f2114b6a000)
libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0
(0x7f2114abd000)
libpcreposix.so.3 => /usr/lib/x86_64-linux-gnu/libpcreposix.so.3
(0x7f2114ab8000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3
(0x7f2114a43000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x7f2114a28000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f2114836000)
libunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8
(0x7f2114819000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(0x7f2114638000)
/lib64/ld-linux-x86-64.so.2 (0x7f211552e000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5
(0x7f211460f000)
liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1
(0x7f21145ee000)
libgcrypt.so.20 => /usr/lib/x86_64-linux-gnu/libgcrypt.so.20
(0x7f21144d)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0
(0x7f21144ad000)

CPUPROFILE=/tmp/haproxy-load-x64.prof haproxy -p pid.txt -f haproxy.cfg

The above command creates /tmp/haproxy-load-x64.prof but it is never
populated with any data. I've tried stopping HAProxy with signals INT, KILL
and TERM.
I even tried with env var CPUPROFILESIGNAL=12 to start/stop the profiler
manually but again no success.

I could share with you reports from Linux 'perf' command. Just let me know
which events you'd be interested in!

Regards,
Martin

On Sat, Jul 18, 2020 at 11:40 AM Илья Шипицин  wrote:

> Hello, Martin!
>
> Can you please compare load profiles using google perftools ?
>
> I never tried to use gperf on ARM64, also, my trial at Linaro is over, I
> do not have an access to any ARM64 anymore.
> in short, gperf can be found https://github.com/gperftools/gperftools
>
> please follow "CPU profiling part".
>
> it can collect cachegrind output, I attached example kcachegrind report
> (you can sort by "self" time).
> it would be interesting to compare amd64 <--> arm64
>
> [image: Screenshot from 2020-07-18 13-35-27.png]
>
>
> пт, 10 июл. 2020 г. в 19:00, Martin Grigorov :
>
>> Hello HAProxy community,
>>
>> I wanted to compare how the newly released HAProxy 2.2 (Congrats!)
>> behaves under heavy load so I've ran some tests on my x86_64 and aarch64
>> VMs:
>>
>>
>> https://medium.com/@martin.grigorov/compare-haproxy-performance-on-x86-64-and-arm64-cpu-architectures-bfd55d1d5566
>>
>> Without much surprise the x86_64 VM gave better results!
>>
>> It is *not* a real use case scenario: the backends serve on GET / and
>> return "Hello World", without any file/network operations.
>>
>> What is interesting though is that I can get 120-160K reqs/sec when
>> hitting directly one of the backend servers, and only 20-40K reqs/sec when
>> using HAProxy as a load balancer in front of them.
>>
>> I'd be happy to re-run the tests with any kind of improvements you may
>> have!
>>
>> Regards,
>> Martin
>>
>


Re: Load testing HAProxy 2.2 on x86_64 and aarch64 VMs

2020-07-20 Thread Martin Grigorov
Hi Илья,

I will do it sometimes this week!

Regards,
Martin

On Sat, Jul 18, 2020 at 11:40 AM Илья Шипицин  wrote:

> Hello, Martin!
>
> Can you please compare load profiles using google perftools ?
>
> I never tried to use gperf on ARM64, also, my trial at Linaro is over, I
> do not have an access to any ARM64 anymore.
> in short, gperf can be found https://github.com/gperftools/gperftools
>
> please follow "CPU profiling part".
>
> it can collect cachegrind output, I attached example kcachegrind report
> (you can sort by "self" time).
> it would be interesting to compare amd64 <--> arm64
>
> [image: Screenshot from 2020-07-18 13-35-27.png]
>
>
> пт, 10 июл. 2020 г. в 19:00, Martin Grigorov :
>
>> Hello HAProxy community,
>>
>> I wanted to compare how the newly released HAProxy 2.2 (Congrats!)
>> behaves under heavy load so I've ran some tests on my x86_64 and aarch64
>> VMs:
>>
>>
>> https://medium.com/@martin.grigorov/compare-haproxy-performance-on-x86-64-and-arm64-cpu-architectures-bfd55d1d5566
>>
>> Without much surprise the x86_64 VM gave better results!
>>
>> It is *not* a real use case scenario: the backends serve on GET / and
>> return "Hello World", without any file/network operations.
>>
>> What is interesting though is that I can get 120-160K reqs/sec when
>> hitting directly one of the backend servers, and only 20-40K reqs/sec when
>> using HAProxy as a load balancer in front of them.
>>
>> I'd be happy to re-run the tests with any kind of improvements you may
>> have!
>>
>> Regards,
>> Martin
>>
>


Re: Log levels when logging to stdout

2020-07-16 Thread Martin Grigorov
On Thu, Jul 16, 2020 at 10:22 AM Jerome Magnin  wrote:

> Hi Martin,
>
> On Thu, Jul 16, 2020 at 10:05:40AM +0300, Martin Grigorov wrote:
> >
> > I am using such logging configuration (HAProxy built from master branch):
> >
> > global
> >   log stdout format raw local0 err
> >   ...
> > defaults
> >   log global
> >   option dontlog-normal
> >   option httplog
> >   option dontlognull
> >   ...
> >
> > But HAProxy still logs entries like the following:
> >
> > 0001:test_fe.clireq[001c:]: GET /haproxy-load HTTP/1.1
> > 0002:test_fe.clihdr[001e:]: host: 192.168.0.206:8080
> > 0006:test_fe.accept(0004)=0020 from [:::192.168.0.72:46768]
> > ALPN=
> > 0001:test_fe.clihdr[001c:]: host: 192.168.0.206:8080
> > 0002:test_fe.clihdr[001e:]: content-type: application/json
> > 0004:test_fe.accept(0004)=0024 from [:::192.168.0.72:46754]
> > ALPN=
> > 0005:test_fe.accept(0004)=001f from [:::192.168.0.72:46766]
> > ALPN=
> > 0001:test_fe.clihdr[001c:]: content-type: application/json
> > 0007:test_fe.accept(0004)=0025 from [:::192.168.0.72:46776]
> > ALPN=
> > 0003:test_fe.clireq[0022:]: GET /haproxy-load HTTP/1.1
> > 0008:test_fe.accept(0004)=001d from [:::192.168.0.72:46756]
> > ALPN=
> > 0004:test_fe.clireq[0024:]: GET /haproxy-load HTTP/1.1
> > 0003:test_fe.clihdr[0022:]: host: 192.168.0.206:8080
> > 0004:test_fe.clihdr[0024:]: host: 192.168.0.206:8080
> >
> > Those do not look like errors but in any case I tried also with 'emerg'
> log
> > level instead of 'err'
> >  and this didn't change anything.
>
> This is haproxy debug output, available when you start with haproxy -d.
>

Thanks, Jerome!
This was the problem !

Martin


>
> >
> > Do I configure it in a wrong way ?
> > I want HAProxy to log only when there is a problem. Because now it logs
> few
> > GBs of those when I load it and this affects the performance.
> > https://medium.com/@martin.grigorov/hi-willy-476dee6439d3
>
> How do you start haproxy ?
>
> --
> Jérôme
>


Log levels when logging to stdout

2020-07-16 Thread Martin Grigorov
Hello HAProxy community,

I am using such logging configuration (HAProxy built from master branch):

global
  log stdout format raw local0 err
  ...
defaults
  log global
  option dontlog-normal
  option httplog
  option dontlognull
  ...

But HAProxy still logs entries like the following:

0001:test_fe.clireq[001c:]: GET /haproxy-load HTTP/1.1
0002:test_fe.clihdr[001e:]: host: 192.168.0.206:8080
0006:test_fe.accept(0004)=0020 from [:::192.168.0.72:46768]
ALPN=
0001:test_fe.clihdr[001c:]: host: 192.168.0.206:8080
0002:test_fe.clihdr[001e:]: content-type: application/json
0004:test_fe.accept(0004)=0024 from [:::192.168.0.72:46754]
ALPN=
0005:test_fe.accept(0004)=001f from [:::192.168.0.72:46766]
ALPN=
0001:test_fe.clihdr[001c:]: content-type: application/json
0007:test_fe.accept(0004)=0025 from [:::192.168.0.72:46776]
ALPN=
0003:test_fe.clireq[0022:]: GET /haproxy-load HTTP/1.1
0008:test_fe.accept(0004)=001d from [:::192.168.0.72:46756]
ALPN=
0004:test_fe.clireq[0024:]: GET /haproxy-load HTTP/1.1
0003:test_fe.clihdr[0022:]: host: 192.168.0.206:8080
0004:test_fe.clihdr[0024:]: host: 192.168.0.206:8080

Those do not look like errors but in any case I tried also with 'emerg' log
level instead of 'err'
 and this didn't change anything.

Do I configure it in a wrong way ?
I want HAProxy to log only when there is a problem. Because now it logs few
GBs of those when I load it and this affects the performance.
https://medium.com/@martin.grigorov/hi-willy-476dee6439d3

Regards,
Martin


Load testing HAProxy 2.2 on x86_64 and aarch64 VMs

2020-07-10 Thread Martin Grigorov
Hello HAProxy community,

I wanted to compare how the newly released HAProxy 2.2 (Congrats!) behaves
under heavy load so I've ran some tests on my x86_64 and aarch64 VMs:

https://medium.com/@martin.grigorov/compare-haproxy-performance-on-x86-64-and-arm64-cpu-architectures-bfd55d1d5566

Without much surprise the x86_64 VM gave better results!

It is *not* a real use case scenario: the backends serve on GET / and
return "Hello World", without any file/network operations.

What is interesting though is that I can get 120-160K reqs/sec when hitting
directly one of the backend servers, and only 20-40K reqs/sec when using
HAProxy as a load balancer in front of them.

I'd be happy to re-run the tests with any kind of improvements you may have!

Regards,
Martin


Broken builds after "REORG: dgram: rename proto_udp to dgram"

2020-06-11 Thread Martin Grigorov
Hello,

Just FYI: the build is broken since this commit:
https://github.com/haproxy/haproxy/commit/7c18b54106ec21273aea3fe59ba23280e86821f5

https://travis-ci.com/github/haproxy/haproxy/builds

Regards,
Martin


Re: [PATCH] enable arm64 builds in travis-ci

2020-05-17 Thread Martin Grigorov
Hi Willy,

On Fri, May 15, 2020 at 6:07 PM Willy Tarreau  wrote:

> Ilya,
>
> > also, I'd suggest to purge travis-ci cache (if you are build in your own
> > fork).
> > some travis related issue might be related when something is took from
> > cache (which was not supposed to happen)
>
> Could you please handle Martin's patch, possibly cut it into several
> pieces if relevant and add a commit message indicating what it does
> (and why) ? Martin is not at ease with Git (which is not a problem),
> and it seems only him and you understand how the reasons of the changes
> in his patch. At least it's totally unclear to me why there's a new
> install target for arm64 and why there's a special "make" invocation
> there.
>

Let me explain the change.
At
https://github.com/haproxy/haproxy/blob/a8dbdf3c4b463a3f3e018f0cd02fa0d8d179bc07/.travis.yml#L113-L117
you
may see the default 'install' phase.
At
https://github.com/haproxy/haproxy/blob/a8dbdf3c4b463a3f3e018f0cd02fa0d8d179bc07/.travis.yml#L12-L19
is
the default environment.
They are used by every job from the matrix (
https://github.com/haproxy/haproxy/blob/a8dbdf3c4b463a3f3e018f0cd02fa0d8d179bc07/.travis.yml#L35
).
But each job can override the default environment and any of the phases
(before_install, install, after_install, script).
For the ARM64 build I overwrote the 'install' phase by copying the default
one and removing the execution of the build_ssl() function (the one that
builds OpenSSL from source) and I also overwrote the environment to update
the values of SSL_INC and SSL_LIB variables.
'openssl' and 'libssl-dev' packages are already installed in the Ubuntu
image used by TravisCI so there is nothing to install manually.. I've added
a comment (
https://github.com/haproxy/haproxy/blob/a8dbdf3c4b463a3f3e018f0cd02fa0d8d179bc07/.travis.yml#L47)
to remind us how it works.


> Feel free to add your "purge cache" change as an extra patch if needed.
> But in any case, please make sure it's still possible to follow the
> impact of each change, because we've touched many things blindly for
> a while on this arm64 issue and most of the changes were basically
> "let's see if this helps", which is a real mess :-/
>
> Thanks!
> Willy
>


Re: [PATCH] enable arm64 builds in travis-ci

2020-05-17 Thread Martin Grigorov
Thank you for applying the patch!

The ARM64 build passed at
https://travis-ci.com/github/haproxy/haproxy/jobs/335338296 !
But it passes only for the builds which have 'haproxy-mirror' :
https://travis-ci.com/github/haproxy/haproxy/builds
I am not sure what exactly "haproxy-mirror" is in the TravisCI config.
The builds which do not have "haproxy-mirror" next to the branch name also
do not have ARM64 build at all, e.g.
https://travis-ci.com/github/haproxy/haproxy/builds/166573019.

One thing that I notice is that the successful link above has *jobs *and
the failing one: *builds *before the job/build id.

On Fri, May 15, 2020 at 9:53 PM Willy Tarreau  wrote:

> On Fri, May 15, 2020 at 11:44:48PM +0500,  ??? wrote:
> > commit message adjusted
>
> Many thanks for this, Ilya, now pushed!
>
> Willy
>


Re: [PATCH] enable arm64 builds in travis-ci

2020-05-15 Thread Martin Grigorov
Those are set to new values at
https://github.com/haproxy/haproxy/pull/630/files#diff-354f30a63fb0907d4ad57269548329e3R51

On Fri, May 15, 2020 at 1:11 PM Илья Шипицин  wrote:

> or we'd better move SSL_LIB, SSL_INC to build-ssl.sh script
>
> пт, 15 мая 2020 г. в 15:09, Илья Шипицин :
>
>> probably, you also need to unset SSL_LIB and SSL_INC
>>
>>
>>
>> btw, I got an answer how to grant travis-ci rights (for triggering build
>> manually)
>>
>> https://travis-ci.community/t/undocumented-require-admin-permissions/8530
>>
>>
>> пт, 15 мая 2020 г. в 14:57, Martin Grigorov :
>>
>>> Hi,
>>>
>>> I've created https://github.com/haproxy/haproxy/pull/630
>>> With this change the build passed successfully for 5 mins and 7 secs for
>>> ARM64.
>>>
>>> Please let me know if you prefer me to send it as an attached .patch
>>> file here. (I haven't used `git format-patch` before :-/).
>>>
>>> Martin
>>>
>>> On Mon, May 11, 2020 at 12:38 PM Илья Шипицин 
>>> wrote:
>>>
>>>>
>>>>
>>>> сб, 9 мая 2020 г. в 11:45, Willy Tarreau :
>>>>
>>>>> On Sat, May 09, 2020 at 08:11:27AM +0200, Vincent Bernat wrote:
>>>>> >  ?  8 mai 2020 14:25 +02, Willy Tarreau:
>>>>> >
>>>>> > >> > Let's increase the timeout to see if it has a chance to finish,
>>>>> no ?
>>>>> > >> >
>>>>> > >>
>>>>> > >> yes
>>>>> > >
>>>>> > > OK now pushed. It's really annoying to work blindly like this. The
>>>>> > > build model Travis uses is broken by design. Requiring to commit
>>>>> > > something for testing is utterly wrong. And doing so within the
>>>>> > > project that's supposed to being test is further wrong. We already
>>>>> > > have 44 patches only on .travis.yml! If this continues like this,
>>>>> > > I predict that a "pre-CI" solution will appear to test if your
>>>>> > > change is likely to trigger a travis error before it gets merged...
>>>>> >
>>>>> > You can push changes to a (throwable) branch instead.
>>>>>
>>>>> Good point, that can also be a solution. But it remains completely
>>>>> hackish. It's basically abusing a versioning system to use it as a
>>>>> messaging system to indicate "please build with this".
>>>>>
>>>>> Willy
>>>>>
>>>>
>>>>
>>>> I created several topics (no answer yet).
>>>>
>>>> as for travis-ci rights, it's totally undocumented. but I suspect
>>>> travis grants
>>>> rights based on github rights. i.e. github admin becomes travis admin
>>>> as well.
>>>>
>>>> https://travis-ci.community/t/arm64-fails-with-non-clear-reason/8529
>>>>
>>>>
>>>> https://travis-ci.community/t/undocumented-require-admin-permissions/8530
>>>>
>>>>
>>>> https://travis-ci.community/t/undocumented-operation-requires-create-request-access-to-repository/8528
>>>>
>>>>
>>>


Re: [PATCH] enable arm64 builds in travis-ci

2020-05-15 Thread Martin Grigorov
Hi,

I've created https://github.com/haproxy/haproxy/pull/630
With this change the build passed successfully for 5 mins and 7 secs for
ARM64.

Please let me know if you prefer me to send it as an attached .patch file
here. (I haven't used `git format-patch` before :-/).

Martin

On Mon, May 11, 2020 at 12:38 PM Илья Шипицин  wrote:

>
>
> сб, 9 мая 2020 г. в 11:45, Willy Tarreau :
>
>> On Sat, May 09, 2020 at 08:11:27AM +0200, Vincent Bernat wrote:
>> >  ?  8 mai 2020 14:25 +02, Willy Tarreau:
>> >
>> > >> > Let's increase the timeout to see if it has a chance to finish, no
>> ?
>> > >> >
>> > >>
>> > >> yes
>> > >
>> > > OK now pushed. It's really annoying to work blindly like this. The
>> > > build model Travis uses is broken by design. Requiring to commit
>> > > something for testing is utterly wrong. And doing so within the
>> > > project that's supposed to being test is further wrong. We already
>> > > have 44 patches only on .travis.yml! If this continues like this,
>> > > I predict that a "pre-CI" solution will appear to test if your
>> > > change is likely to trigger a travis error before it gets merged...
>> >
>> > You can push changes to a (throwable) branch instead.
>>
>> Good point, that can also be a solution. But it remains completely
>> hackish. It's basically abusing a versioning system to use it as a
>> messaging system to indicate "please build with this".
>>
>> Willy
>>
>
>
> I created several topics (no answer yet).
>
> as for travis-ci rights, it's totally undocumented. but I suspect travis
> grants
> rights based on github rights. i.e. github admin becomes travis admin as
> well.
>
> https://travis-ci.community/t/arm64-fails-with-non-clear-reason/8529
>
> https://travis-ci.community/t/undocumented-require-admin-permissions/8530
>
>
> https://travis-ci.community/t/undocumented-operation-requires-create-request-access-to-repository/8528
>
>


Re: [PATCH] enable arm64 builds in travis-ci

2020-05-08 Thread Martin Grigorov
On Fri, May 8, 2020 at 3:25 PM Willy Tarreau  wrote:

> On Fri, May 08, 2020 at 05:04:43PM +0500,  ??? wrote:
> > > Let's increase the timeout to see if it has a chance to finish, no ?
> > >
> >
> > yes
>
> OK now pushed. It's really annoying to work blindly like this. The
> build model Travis uses is broken by design. Requiring to commit
> something for testing is utterly wrong. And doing so within the
> project that's supposed to being test is further wrong. We already
> have 44 patches only on .travis.yml! If this continues like this,
> I predict that a "pre-CI" solution will appear to test if your
> change is likely to trigger a travis error before it gets merged...
>

You can avoid this by making changes to .travis.yml directly in Travis UI.
At the right-top part of the page there is "More options" menu. Select
"Trigger build" and in the dialog you can paste "Custom config". Once you
are happy with the outcome you can push it in your Git repo.

Back to the main topic:
Now it fails with totally new reason. It doesn't even start installing the
dependencies.
It started failing this way since:
https://github.com/haproxy/haproxy/commit/8d76af55e10f018ab3d482125433be4992884372
You can try the "manual approach" by reverting this change! ;-)


>
> Willy
>


Re: [PATCH] enable arm64 builds in travis-ci

2020-05-08 Thread Martin Grigorov
On Fri, May 8, 2020 at 10:54 AM Илья Шипицин  wrote:

>
>
> пт, 8 мая 2020 г. в 12:26, Willy Tarreau :
>
>> On Fri, May 08, 2020 at 09:34:32AM +0300, Martin Grigorov wrote:
>> > It must have started failing when you updated the version of OpenSSL.
>> > .travis.yml caches ~/opt folder between builds. After the update to
>> 1.1.1f
>> > the build doesn't see the OpenSSL binaries in the cache anymore and
>> tries
>> > to download it and build it.
>> > But as I've noticed in my attempt to build HAProxy with Docker+QEMU the
>> > build of OpenSSL is taking too long.
>> > The build of OpenSSL is wrapped with travis_wait to reduce the writes to
>> > stdout but the default time for travis_wait is 20 mins and this is not
>> > enough to build OpenSSL.
>>
>> That's very likely indeed.
>>
>
>
> it is not :)
> I provided link to my fork, openssl build takes 640 sec
>
>
>>
>> > Due to
>> >
>> https://travis-ci.community/t/output-is-truncated-heavily-in-arm64-when-a-command-hangs/7630
>> > TravisCI
>> > does not properly report that the problem is at build_ssl() step but
>> shows
>> > the last chunk of the buffered response and this confuses us all.
>>
>> Ah, excellent, precisely what I was looking for. And some indicate
>> that the buffering further causes issues in the build system itself!
>>
>> > I think the build will become green if we extend travis_wait to a higher
>> > value (
>> >
>> https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received
>> ).
>> > I don't remember where I have read it but I think the upper limit is 120
>> > mins.
>> > @Willy: could you please change
>> > https://github.com/haproxy/haproxy/blob/master/.travis.yml#L112 to:
>> >
>> > travis_wait 120 bash -c 'scripts/build-ssl.sh >build-ssl.log 2>&1' ||
>> (cat
>> > build-ssl.log && exit 1)
>> >
>> > i.e. add '120' after travis_wait
>>
>> We could, but 120 (2 hours) seems a bit extreme. It also means we can
>> steal
>> 2 hours of CPU there in case something goes wrong.
>>
>
> travis-ci limits build at 60 min.
>
>
>>
>> > This should give it the time to download and install OpenSSL 1.1.1f and
>> to
>> > cache it. If the build passes once then the next builds should be much
>> > faster because OpenSSL will be used from the cache.
>>
>> I'm wondering why instead we don't fallback on an already packaged version
>> of OpenSSL for this platform. I mean, sure it's convenient to test the
>> latest version but it's already tested on x86 and we could very well use
>> any other version already packaged on the distro present there. This would
>> solve the problem and even increase versions coverage.
>>
>> Ilya, what do you think ?
>>
>
> yes, there are few options to think about.
> I also provided options (numbered 1..4), Hopefully, I think on your
> suggestions, Martin will think on my suggestion ... and we will decide what
> next would be.
>

I still believe the problem is in the time needed to build OpenSSL, not in
apt[-get].

We can increase temporarily the travis_wait to 60 just to cache the build
of 1.1.1f and then remove '60' again. Although if this is the problem then
'travis_wait 60 ...' won't do any harm for the next builds because it will
use the cache and return in few secs.

I like the idea to install openssl with apt but I am not sure whether we
can do this *only* for ARM64.


>
>
>>
>> Thanks,
>> Willy
>>
>


Re: [PATCH] enable arm64 builds in travis-ci

2020-05-08 Thread Martin Grigorov
Hi,

I think I understand why it started failing.
It must have started failing when you updated the version of OpenSSL.
.travis.yml caches ~/opt folder between builds. After the update to 1.1.1f
the build doesn't see the OpenSSL binaries in the cache anymore and tries
to download it and build it.
But as I've noticed in my attempt to build HAProxy with Docker+QEMU the
build of OpenSSL is taking too long.
The build of OpenSSL is wrapped with travis_wait to reduce the writes to
stdout but the default time for travis_wait is 20 mins and this is not
enough to build OpenSSL.
Due to
https://travis-ci.community/t/output-is-truncated-heavily-in-arm64-when-a-command-hangs/7630
TravisCI
does not properly report that the problem is at build_ssl() step but shows
the last chunk of the buffered response and this confuses us all.
I think the build will become green if we extend travis_wait to a higher
value (
https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received).
I don't remember where I have read it but I think the upper limit is 120
mins.
@Willy: could you please change
https://github.com/haproxy/haproxy/blob/master/.travis.yml#L112 to:

travis_wait 120 bash -c 'scripts/build-ssl.sh >build-ssl.log 2>&1' || (cat
build-ssl.log && exit 1)

i.e. add '120' after travis_wait

This should give it the time to download and install OpenSSL 1.1.1f and to
cache it. If the build passes once then the next builds should be much
faster because OpenSSL will be used from the cache.

Regards,
Martin

On Fri, May 8, 2020 at 9:18 AM Willy Tarreau  wrote:

> Hi Martin,
>
> On Fri, May 08, 2020 at 08:56:07AM +0300, Martin Grigorov wrote:
> > Unfortunately it is not good:
> > https://travis-ci.com/github/haproxy/haproxy/jobs/329657180
>
> Indeed it's still not fixed on Travis' side. However what Ilya did
> actually worked, in that the status is not reported as a global
> build failure anymore. This allows us to continue to monitor if
> and when this issue finally resolves on the build infrastructure.
> It's also possible that they're not aware of the problem due to
> too few people using arm64. If someone has contacts there it might
> be worth checking with them. All we know for now is that it seems
> to stop moving while setting up libpcre2. Maybe there's a bug in
> a script in this package, resulting in a prompt for a question
> which never gets a response :-/  But that's something we can't
> check since we don't have access to an interactive shell there
> to diagnose.
>
> Willy
>


Re: [PATCH] enable arm64 builds in travis-ci

2020-05-07 Thread Martin Grigorov
Hi all,

On Thu, May 7, 2020 at 11:56 PM Willy Tarreau  wrote:

> Hi Ilya,
>
> On Thu, May 07, 2020 at 09:19:48PM +0500,  ??? wrote:
> > Hello,
> >
> > let us enable arm64 builds back.
>
> Good idea, just merged now. Let's see how that ends up now.
>

Unfortunately it is not good:
https://travis-ci.com/github/haproxy/haproxy/jobs/329657180

Martin


>
> Thanks,
> Willy
>
>


Re: [PATCH] fix errored ARM64 builds in travis-ci

2020-05-06 Thread Martin Grigorov
Hi,

I've just created a PR (https://github.com/haproxy/haproxy/pull/617/files)
that introduces testing on ARM64/AARCH64 at GitHub Actions.
It almost works! There are few tests that fail. Any help finding the reason
is very welcome!

Martin

On Mon, Mar 23, 2020 at 11:12 AM Martin Grigorov 
wrote:

> Hi Илья,
>
> On Sun, Mar 22, 2020 at 2:46 PM Илья Шипицин  wrote:
>
>> Martin,
>>
>> as the one of the most interested in ARM64 builds, I've got news for you
>>
>>
>> can you try
>>
>> travis_wait 30 bash -c 'scripts/build-ssl.sh >build-ssl.log 2>&1' || (cat
>> build-ssl.log && exit 1)
>>
>> in travis ? (please not "travis_wait 30" instead of "travis_wait")
>>
>
> it is running at the moment here:
> https://travis-ci.org/github/martin-g/haproxy/builds/665770469
>
>
>>
>>
>> also, it might be important to clear travis cache from time to time.
>> as for myself, "travis_wait 30" helped me to resolve similar issue on
>> another project (in my own fork haproxy on arm64 builds just fine)
>>
>> ср, 18 мар. 2020 г. в 23:35, Илья Шипицин :
>>
>>> well, there are several topics on travis-ci forum related to "output on
>>> ARM64 got truncated in the mid of ..."
>>> Let us disable ARM64 travis-ci builds for few months.
>>>
>>> Martin, I'll play with hosted github runner in order to find a way how
>>> we can limit its builds to allowed only.
>>>
>>> ср, 18 мар. 2020 г. в 18:57, Martin Grigorov :
>>>
>>>>
>>>> Current master's build passed the problematic point in my TravisCI
>>>> project: https://travis-ci.org/github/martin-g/haproxy/jobs/663953359
>>>> Note: I use TravisCI .org while HAProxy's official project is at .com:
>>>> https://travis-ci.com/github/haproxy/haproxy
>>>> I also think this is a problem on TravisCI's end.
>>>>
>>>> Martin
>>>>
>>>> On Wed, Mar 18, 2020 at 3:43 PM Илья Шипицин 
>>>> wrote:
>>>>
>>>>> I will disable PR builds.
>>>>>
>>>>> On Wed, Mar 18, 2020, 6:27 PM Willy Tarreau  wrote:
>>>>>
>>>>>> On Wed, Mar 18, 2020 at 06:21:15PM +0500,  ??? wrote:
>>>>>> > let us calm down a bit :)
>>>>>>
>>>>>> Agreed, especially since the build on PRs already happens and already
>>>>>> adds noise.
>>>>>>
>>>>>> > yes, I still believe it is because of buffering. I might have missed
>>>>>> > something.
>>>>>> > unless I will repair it, I'll drop arm64 support on travis (and we
>>>>>> will
>>>>>> > switch to self hosted github action runner)
>>>>>>
>>>>>> OK.
>>>>>>
>>>>>> Willy
>>>>>>
>>>>>


Re: Failing tests if USE_OPENSSL=1 is omitted in the FLAGS

2020-05-06 Thread Martin Grigorov
Hi Илья,

On Wed, May 6, 2020 at 11:59 AM Илья Шипицин  wrote:

> do you run tests on GH arm64 agents ? is it dedicated (your own) agents
> attached to your repo ? can you give a link ?
>

I use Docker + QEMU with GH hosted runner.
You can see the current diff at
https://github.com/haproxy/haproxy/compare/master...martin-g:feature/test-aarch64-on-github-actions
'windows-latest.yml` is temporary disabled until I'm ready with my
experiments.

As explained at
https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories
using
self-hosted runner is not secure if they are used for Pull Requests.
They can be used for pushes to `master` branch though since this is
supposed to be a reviewed code!

But the tests failures I reported below fail on my local machine (x86_64)
and my ARM64 VM. I do not use Docker and/or QEMU here.

Martin


> ср, 6 мая 2020 г. в 13:22, Martin Grigorov :
>
>> Hello HAProxy team,
>>
>> While working on a PR to build & test HAProxy on AARCH64 at GitHub
>> Actions I've noticed a strange behavior for some of the tests.
>>
>> To reduce the time of the build I've removed USE_OPENSSL=1 from the FLAGS
>> [1] passed to "make".
>> The build passed successfully, some of the tests are skipped because they
>> depend on SSL library, e.g.:
>>
>> ...
>> Add test: reg-tests/lua/txn_get_priv.vtc
>>   Add test: reg-tests/lua/h_txn_get_priv.vtc
>>   Skip reg-tests/ssl/wrong_ctx_storage.vtc because haproxy is not
>> compiled with the required option OPENSSL
>>   Skip reg-tests/ssl/ssl_client_auth.vtc because haproxy is not compiled
>> with the required option OPENSSL
>>   Skip reg-tests/ssl/add_ssl_crt-list.vtc because haproxy is not compiled
>> with the required option OPENSSL
>>   Skip reg-tests/ssl/set_ssl_cert.vtc because haproxy is not compiled
>> with the required option OPENSSL
>> ...
>>
>> but few tests just fail:
>>
>> Testing with haproxy version: 2.2-dev7
>> #top  TEST reg-tests/lua/txn_get_priv.vtc FAILED (5.008) exit=2
>> #top  TEST reg-tests/compression/lua_validation.vtc TIMED OUT (kill
>> -9)
>> #top  TEST reg-tests/compression/lua_validation.vtc FAILED (10.009)
>> signal=9
>> 2 tests failed, 0 tests skipped, 64 tests passed
>> ## Gathering results ##
>> ## Test case: reg-tests/compression/lua_validation.vtc ##
>> ## test results in:
>> "/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.7fffd08a"
>> ## Test case: reg-tests/lua/txn_get_priv.vtc ##
>> ## test results in:
>> "/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.065f2677"
>>  c0   HTTP rx timeout (fd:6 5000 ms)
>>  h1   Bad exit status: 0x0100 exit 0x1 signal 0 core 0
>> Makefile:995: recipe for target 'reg-tests' failed
>> make: *** [reg-tests] Error 1
>>
>> These tests fail both on AARCH64 and x86_64 consistently until I add back
>> USE_OPENSSL=1
>> to the flags.
>>
>> Is there an issue with these tests ?
>>
>> 1.
>> https://github.com/haproxy/haproxy/blob/fafa13dd6549ee431f41dc3c1857855974d79bea/.travis.yml#L14
>>
>> Regards,
>> Martin
>>
>


Failing tests if USE_OPENSSL=1 is omitted in the FLAGS

2020-05-06 Thread Martin Grigorov
Hello HAProxy team,

While working on a PR to build & test HAProxy on AARCH64 at GitHub Actions
I've noticed a strange behavior for some of the tests.

To reduce the time of the build I've removed USE_OPENSSL=1 from the FLAGS
[1] passed to "make".
The build passed successfully, some of the tests are skipped because they
depend on SSL library, e.g.:

...
Add test: reg-tests/lua/txn_get_priv.vtc
  Add test: reg-tests/lua/h_txn_get_priv.vtc
  Skip reg-tests/ssl/wrong_ctx_storage.vtc because haproxy is not compiled
with the required option OPENSSL
  Skip reg-tests/ssl/ssl_client_auth.vtc because haproxy is not compiled
with the required option OPENSSL
  Skip reg-tests/ssl/add_ssl_crt-list.vtc because haproxy is not compiled
with the required option OPENSSL
  Skip reg-tests/ssl/set_ssl_cert.vtc because haproxy is not compiled with
the required option OPENSSL
...

but few tests just fail:

Testing with haproxy version: 2.2-dev7
#top  TEST reg-tests/lua/txn_get_priv.vtc FAILED (5.008) exit=2
#top  TEST reg-tests/compression/lua_validation.vtc TIMED OUT (kill -9)
#top  TEST reg-tests/compression/lua_validation.vtc FAILED (10.009)
signal=9
2 tests failed, 0 tests skipped, 64 tests passed
## Gathering results ##
## Test case: reg-tests/compression/lua_validation.vtc ##
## test results in:
"/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.7fffd08a"
## Test case: reg-tests/lua/txn_get_priv.vtc ##
## test results in:
"/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.065f2677"
 c0   HTTP rx timeout (fd:6 5000 ms)
 h1   Bad exit status: 0x0100 exit 0x1 signal 0 core 0
Makefile:995: recipe for target 'reg-tests' failed
make: *** [reg-tests] Error 1

These tests fail both on AARCH64 and x86_64 consistently until I add back
USE_OPENSSL=1
to the flags.

Is there an issue with these tests ?

1.
https://github.com/haproxy/haproxy/blob/fafa13dd6549ee431f41dc3c1857855974d79bea/.travis.yml#L14

Regards,
Martin


Re: regtest: abns should work now :-)

2020-04-03 Thread Martin Grigorov
Hi everyone,

On Mon, Mar 23, 2020 at 11:11 AM Martin Grigorov 
wrote:

> Hi Илья,
>
> On Mon, Mar 23, 2020 at 10:52 AM Илья Шипицин 
> wrote:
>
>> well, I tried to repro abns failures on x86_64
>> I chose MS Azure VM of completely different size, both number of CPU and
>> RAM.
>> it was never reproduced, say on 1000 execution in loop.
>>
>> so, I decided "it looks like something with memory aligning".
>> also, I tried to run arm64 emulation on virtualbox. no luck yet.
>>
>
>



> Have you tried with multiarch Docker ?
>
> 1) execute
> docker run --rm --privileged multiarch/qemu-user-static:register --reset
> to register QEMU
>
> 2) create Dockerfile
> for Centos use: FROM multiarch/centos:7-aarch64-clean
> for Ubuntu use: FROM multiarch/ubuntu-core:arm64-bionic
>
> 3) enjoy :-)
>

Here is a PR for Varnish Cache project where I use Docker + QEMU to build
and package for several Linux distros and two architectures:
https://github.com/varnishcache/varnish-cache/pull/3263
They use CircleCI but I guess the same approach can be applied on GitHub
Actions.
If you are interested in this approach I could give it a try.


Regards,
Martin


>
>
>>
>> пн, 23 мар. 2020 г. в 13:43, Willy Tarreau :
>>
>>> Hi Ilya,
>>>
>>> I think this time I managed to fix the ABNS test. To make a long story
>>> short, it was by design extremely sensitive to the new process's startup
>>> time, which is increased with larger FD counts and/or less powerful VMs
>>> and/or noisy neighbors. This explains why it started to misbehave with
>>> the commit which relaxed the maxconn limitations. A starting process
>>> stealing a few ms of CPU from the old one could make its keep-alive
>>> timeout expire before it got a new request on a reused connection,
>>> resulting in an empty response as reported by the client.
>>>
>>> I'm going to issue dev5 now. s390x is currently down but all x86 ones
>>> build and run fine for now.
>>>
>>> Cheers,
>>> Willy
>>>
>>


Re: [PATCH] fix errored ARM64 builds in travis-ci

2020-03-23 Thread Martin Grigorov
Hi Илья,

On Sun, Mar 22, 2020 at 2:46 PM Илья Шипицин  wrote:

> Martin,
>
> as the one of the most interested in ARM64 builds, I've got news for you
>
>
> can you try
>
> travis_wait 30 bash -c 'scripts/build-ssl.sh >build-ssl.log 2>&1' || (cat
> build-ssl.log && exit 1)
>
> in travis ? (please not "travis_wait 30" instead of "travis_wait")
>

it is running at the moment here:
https://travis-ci.org/github/martin-g/haproxy/builds/665770469


>
>
> also, it might be important to clear travis cache from time to time.
> as for myself, "travis_wait 30" helped me to resolve similar issue on
> another project (in my own fork haproxy on arm64 builds just fine)
>
> ср, 18 мар. 2020 г. в 23:35, Илья Шипицин :
>
>> well, there are several topics on travis-ci forum related to "output on
>> ARM64 got truncated in the mid of ..."
>> Let us disable ARM64 travis-ci builds for few months.
>>
>> Martin, I'll play with hosted github runner in order to find a way how we
>> can limit its builds to allowed only.
>>
>> ср, 18 мар. 2020 г. в 18:57, Martin Grigorov :
>>
>>>
>>> Current master's build passed the problematic point in my TravisCI
>>> project: https://travis-ci.org/github/martin-g/haproxy/jobs/663953359
>>> Note: I use TravisCI .org while HAProxy's official project is at .com:
>>> https://travis-ci.com/github/haproxy/haproxy
>>> I also think this is a problem on TravisCI's end.
>>>
>>> Martin
>>>
>>> On Wed, Mar 18, 2020 at 3:43 PM Илья Шипицин 
>>> wrote:
>>>
>>>> I will disable PR builds.
>>>>
>>>> On Wed, Mar 18, 2020, 6:27 PM Willy Tarreau  wrote:
>>>>
>>>>> On Wed, Mar 18, 2020 at 06:21:15PM +0500,  ??? wrote:
>>>>> > let us calm down a bit :)
>>>>>
>>>>> Agreed, especially since the build on PRs already happens and already
>>>>> adds noise.
>>>>>
>>>>> > yes, I still believe it is because of buffering. I might have missed
>>>>> > something.
>>>>> > unless I will repair it, I'll drop arm64 support on travis (and we
>>>>> will
>>>>> > switch to self hosted github action runner)
>>>>>
>>>>> OK.
>>>>>
>>>>> Willy
>>>>>
>>>>


Re: regtest: abns should work now :-)

2020-03-23 Thread Martin Grigorov
Hi Илья,

On Mon, Mar 23, 2020 at 10:52 AM Илья Шипицин  wrote:

> well, I tried to repro abns failures on x86_64
> I chose MS Azure VM of completely different size, both number of CPU and
> RAM.
> it was never reproduced, say on 1000 execution in loop.
>
> so, I decided "it looks like something with memory aligning".
> also, I tried to run arm64 emulation on virtualbox. no luck yet.
>

Have you tried with multiarch Docker ?

1) execute
docker run --rm --privileged multiarch/qemu-user-static:register --reset
to register QEMU

2) create Dockerfile
for Centos use: FROM multiarch/centos:7-aarch64-clean
for Ubuntu use: FROM multiarch/ubuntu-core:arm64-bionic

3) enjoy :-)


>
> пн, 23 мар. 2020 г. в 13:43, Willy Tarreau :
>
>> Hi Ilya,
>>
>> I think this time I managed to fix the ABNS test. To make a long story
>> short, it was by design extremely sensitive to the new process's startup
>> time, which is increased with larger FD counts and/or less powerful VMs
>> and/or noisy neighbors. This explains why it started to misbehave with
>> the commit which relaxed the maxconn limitations. A starting process
>> stealing a few ms of CPU from the old one could make its keep-alive
>> timeout expire before it got a new request on a reused connection,
>> resulting in an empty response as reported by the client.
>>
>> I'm going to issue dev5 now. s390x is currently down but all x86 ones
>> build and run fine for now.
>>
>> Cheers,
>> Willy
>>
>


Re: [PATCH] fix errored ARM64 builds in travis-ci

2020-03-18 Thread Martin Grigorov
Hi,

On Wed, Mar 18, 2020 at 3:29 PM Willy Tarreau  wrote:

> On Wed, Mar 18, 2020 at 06:21:15PM +0500,  ??? wrote:
> > let us calm down a bit :)
>
> Agreed, especially since the build on PRs already happens and already
> adds noise.
>
> > yes, I still believe it is because of buffering. I might have missed
> > something.
> > unless I will repair it, I'll drop arm64 support on travis (and we will
> > switch to self hosted github action runner)
>

There is one major problem with GitHub Actions self hosted runners at the
moment - they are not really private.
I.e. if someone forks HAProxy and pushes something into their fork it will
trigger builds on your private node, i.e. it will consume its resources.
There is no way to say "this is my private node and I want it to build only
after commits in https://github.com/haproxy/haproxy;

If you find a way around this issue and you need an ARM64 VM then just let
me know!

Regards,
Martin


>
> OK.
>
> Willy
>
>


Re: [PATCH] switch to clang-9 in Linux/travis-ci builds

2020-03-15 Thread Martin Grigorov
Hi,

On Sat, Mar 14, 2020 at 11:26 AM Willy Tarreau  wrote:

> Hi Ilya,
>
> On Fri, Jan 24, 2020 at 11:46:45AM +0500,  ??? wrote:
> > Hello,
> >
> > let us use clang-9 instead of default clang-7 for linux builds.
>
> It seems I missed this one. Now applied carefully, we'll see. If it
> causes new failures, we'll adjust accordingly.
>

All looks good here on aarch64 with this change!

Martin


>
> Willy
>
>


Re: [PATCH]: BUILD link to lib atomic on ARM

2020-03-15 Thread Martin Grigorov
Hi David,

On my ARM64 VM `uname -m` returns:
$ uname -m
aarch64

Should your change take 'aarch64' into account as well ?

Martin

On Sun, Mar 15, 2020 at 3:34 PM David CARLIER  wrote:

> Oups sorry I really forgot :-)
>
> On Sun, 15 Mar 2020 at 13:32, Martin Grigorov 
> wrote:
>
>> Hi
>>
>> On Sun, Mar 15, 2020 at 3:03 PM Aleksandar Lazic 
>> wrote:
>>
>>> On 15.03.20 11:33, David CARLIER wrote:
>>> > Hi
>>> >
>>> > Here a little patch proposal to fix build on ARM.
>>> >
>>> > Regards.
>>>
>>> Ähm, maybe my mail client hide the Patch because I can't see it ;-)?
>>>
>>
>> It seems David forgot to attach it or the attachment didn't make it for
>> other reason. I also don't see it.
>>
>> Martin
>>
>>
>>>
>>> Regards
>>> Aleks
>>>
>>>


Re: [PATCH] enable DEBUG_STRICT=1 for all kind of CI builds

2020-03-15 Thread Martin Grigorov
Hello,

I've just tested the change on my ARM64 VM and the build is successful!

Regards,
Martin

On Sun, Mar 15, 2020 at 9:12 AM Илья Шипицин  wrote:

> Hello,
>
> I added DEBUG_STRICT=1 to all builds.
>
> Ilya Shipitcin
>


Re: [PATCH]: BUILD link to lib atomic on ARM

2020-03-15 Thread Martin Grigorov
Hi

On Sun, Mar 15, 2020 at 3:03 PM Aleksandar Lazic  wrote:

> On 15.03.20 11:33, David CARLIER wrote:
> > Hi
> >
> > Here a little patch proposal to fix build on ARM.
> >
> > Regards.
>
> Ähm, maybe my mail client hide the Patch because I can't see it ;-)?
>

It seems David forgot to attach it or the attachment didn't make it for
other reason. I also don't see it.

Martin


>
> Regards
> Aleks
>
>


Re: Tests timeout on my ARM64 test VM

2020-03-15 Thread Martin Grigorov
Hi Willy,

On Fri, Mar 13, 2020 at 10:03 PM Willy Tarreau  wrote:

> Hi Martin,
>
> On Fri, Mar 13, 2020 at 12:35:12PM +0200, Martin Grigorov wrote:
> > Hi ,
> >
> > Suddenly today the build is again green here!
> > I didn't make any changes to my testing setup.
> > It must be something on the OS level but I wasn't able to figure out what
> > makes the HAProxy tests timeout in the last several days.
>
> We've had issues with the abns test on other platforms in the past,
> namely s390x and ppc64le. It used to occasionally break on x86_64 as
> well but far less frequently. It was affected by two bugs that were
> solved yesterday after a few days of investigation and testing. We've
> seen yet another failure again on ppc64 while it was expected not to
> fail, so I'd be careful before claiming victory. However the abns
> test is extremely time sensitive and uses short delays around 15ms to
> try to trigger the issue, and in a VM it is possible to see this
> happen from time to time due to noisy neighbors. That's why I'm
> staying extremely prudent on the verdict. The PPC64 machine I tested
> on is provided by Minicloud and is a VM running on a real CPU, so it's
> much less affected by timing issues. I've run the test several hundreds
> of times in a row and couldn't make it fail anymore.
>
> So don't worry too much if it appeared and disappeared. The change
> that emphasized it was the increase in default maxconn (304e17eb8),
> apparently just due to a slightly longer startup time! And the ones
> expected to have fixed it are between bdb00c5d and 4b3f27b included.
>
> Note that I didn't manage to make it fail on arm64 (real machine,
> SolidRun's Macchiatobin).
>
> Hoping this clarifies the situation.
>

Thank you for this explanation!

The problem here was that the tests were failing even with older commits.
I've tried to git bisect the problem but no matter how far back in Git
history I went the problem was still there. But the logs from the tests
runs from just few days back were OK, no errors & timeouts.
Also the ARM64 tests on TravisCI were OK. And Travis's ARM64 nodes are less
powerful than mine VM.
Those are the reasons I believe that the problem was at my VM.
I just needed help with reading HAProxy's tests error logs. I didn't know
how to approach debugging them.

Regards,
Martin


>
> Regards,
> Willy
>


Re: Tests timeout on my ARM64 test VM

2020-03-13 Thread Martin Grigorov
Hi Илья,

Suddenly today the build is again green here!
I didn't make any changes to my testing setup.
It must be something on the OS level but I wasn't able to figure out what
makes the HAProxy tests timeout in the last several days.

Regards,
Martin

On Wed, Mar 11, 2020 at 4:13 PM Martin Grigorov 
wrote:

>
>
> On Wed, Mar 11, 2020 at 3:06 PM Илья Шипицин  wrote:
>
>> I will a look during next weekend
>>
>
> Thank you, Илья!
>
>
>>
>> BTW, I've managed to get Linaro VM :)
>>
>
> Congrats! :-)
>
>
>>
>> On Wed, Mar 11, 2020, 5:40 PM Martin Grigorov 
>> wrote:
>>
>>> Hi,
>>>
>>> On Mon, Mar 9, 2020 at 10:22 PM Martin Grigorov <
>>> martin.grigo...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am not sure what have changed on my test ARM64 VM but the reg tests
>>>> started timing out.
>>>> Everything is fine on my dev machine (x86_64) and at Travis (
>>>> https://travis-ci.com/haproxy/haproxy).
>>>> I don't think it is ARM64 related. Most probably some OS setting or
>>>> something.
>>>> I've rebooted the system just to make sure it is not some busy port or
>>>> opened file descriptor but
>>>> it still fails the same way.
>>>>
>>>> Does someone see in the attached logs what could be the problem?
>>>>
>>>
>>> Anyone can help me here ?
>>>
>>> Martin
>>>
>>>
>>>> Thank you!
>>>>
>>>> Martin
>>>>
>>>


Re: Tests timeout on my ARM64 test VM

2020-03-11 Thread Martin Grigorov
On Wed, Mar 11, 2020 at 3:06 PM Илья Шипицин  wrote:

> I will a look during next weekend
>

Thank you, Илья!


>
> BTW, I've managed to get Linaro VM :)
>

Congrats! :-)


>
> On Wed, Mar 11, 2020, 5:40 PM Martin Grigorov 
> wrote:
>
>> Hi,
>>
>> On Mon, Mar 9, 2020 at 10:22 PM Martin Grigorov <
>> martin.grigo...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am not sure what have changed on my test ARM64 VM but the reg tests
>>> started timing out.
>>> Everything is fine on my dev machine (x86_64) and at Travis (
>>> https://travis-ci.com/haproxy/haproxy).
>>> I don't think it is ARM64 related. Most probably some OS setting or
>>> something.
>>> I've rebooted the system just to make sure it is not some busy port or
>>> opened file descriptor but
>>> it still fails the same way.
>>>
>>> Does someone see in the attached logs what could be the problem?
>>>
>>
>> Anyone can help me here ?
>>
>> Martin
>>
>>
>>> Thank you!
>>>
>>> Martin
>>>
>>


Re: Tests timeout on my ARM64 test VM

2020-03-11 Thread Martin Grigorov
Hi,

On Mon, Mar 9, 2020 at 10:22 PM Martin Grigorov 
wrote:

> Hi,
>
> I am not sure what have changed on my test ARM64 VM but the reg tests
> started timing out.
> Everything is fine on my dev machine (x86_64) and at Travis (
> https://travis-ci.com/haproxy/haproxy).
> I don't think it is ARM64 related. Most probably some OS setting or
> something.
> I've rebooted the system just to make sure it is not some busy port or
> opened file descriptor but
> it still fails the same way.
>
> Does someone see in the attached logs what could be the problem?
>

Anyone can help me here ?

Martin


> Thank you!
>
> Martin
>


Re: Lua detection on aarch64

2020-01-30 Thread Martin Grigorov
Hi Илья,

On Wed, Jan 29, 2020 at 11:34 AM Илья Шипицин  wrote:

> Hello,
>
> I started to work on rpm packages.
> I also applied at Linaro for arm64 vm (request is not completed yet).
>
>
> interesting that Lua is not detected:
>
>
> https://copr-be.cloud.fedoraproject.org/results/chipitsine/haproxy-rpm/fedora-31-aarch64/01207488-haproxy/builder-live.log.gz
>
> Martin, do you have arm64, can you check that ?
>

I've had the same problem on my VM.
I've tried to debug it and I think the problem was somewhere here:
https://github.com/haproxy/haproxy/blob/160287b6760584eb1dc20d9032d6d49ed051ca0b/Makefile#L547
Maybe
the string concatenation with the quotes cause it.
At Travis it seems to work fine.
I've worked it around by specifying manually LUA_INC=/usr/include/lua5.3/:

make -j3 CC=$CC V=0 TARGET=$TARGET $FLAGS DEBUG_CFLAGS="$DEBUG_CFLAGS"
LDFLAGS="$LDFLAGS -L$SSL_LIB -Wl,-rpath,$SSL_LIB"
51DEGREES_SRC="$FIFTYONEDEGREES_SRC" EXTRA_OBJS="$EXTRA_OBJS"
LUA_INC=/usr/include/lua5.3/


>
> cheers,
> Ilya Shipitcin
>


Re: Disabling regtests in Travis ?

2020-01-27 Thread Martin Grigorov
On Fri, Jan 24, 2020 at 6:43 PM Willy Tarreau  wrote:

> On Fri, Jan 24, 2020 at 09:12:58PM +0500,  ??? wrote:
> > >> +  - make reg-tests VTEST_PROGRAM=../vtest/vtest
> > >> REGTESTS_TYPES=default,bug,devel
> > >>
> > >
> > > let us try that.
>
> OK, now pushed.
>
> > > I will have a look at "racy" tests.
> > > Maybe we'll enable them on Github Actions.
> > >
> > >
> > the good thing about Github Actions, it is possible to attach own build
> > agents. So, if we
> > have dedicated hardware and we not want to depend on travis-ci
> neighbours,
> > it might be an option.
>
> That's good to know, even if I doubt we'd need it, at least it
> opens possibilities.
>

The regtests run fine on my ARM64 VM. I run them daily.
If HAProxy team decides to move to GitHub Actions and to use an external
build agent for ARM64 then just ping me!

Regards,
Martin


>
> Willy
>
>


Re: [PATCH] introduce ARM64 travis-ci builds

2020-01-20 Thread Martin Grigorov
Thank you, Илья!

On Sun, Jan 19, 2020 at 9:20 AM Илья Шипицин  wrote:

> hello,
>
> sometimes arm64 builds fails, I think it is good chance to introduce
> regular builds
> and fix them.
>
> also, few small improvements.
>
> cheers,
> Ilya Shipicin
>


Re: ARM(64) builds

2020-01-19 Thread Martin Grigorov
Hi,

On Sat, Jan 18, 2020, 22:10 Илья Шипицин  wrote:

> tests on ARM64 randomly fail
> https://travis-ci.com/chipitsine/haproxy/jobs/277236120
>
> (after restart there's a good chance to success)
>

I have the same observation on TravisCI. I think the reason is that their
arm64 instances are less powerful than the amd64 ones.
At https://docs.travis-ci.com/user/multi-cpu-architectures it is said:
While available to all Open Source repositories, the concurrency available
for multiple CPU arch-based jobs is limited during the alpha period.
Not sure how much limited it is though.

Today this test failed once on my VM:

#top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (1.111)
exit=2
1 tests failed, 0 tests skipped, 34 tests passed
## Gathering results ##
## Test case: reg-tests/seamless-reload/abns_socket.vtc ##
## test results in:
"/tmp/haregtests-2020-01-19_08-06-39.45Hchw/vtc.8496.328d7f95"
 c1   HTTP rx failed (fd:6 read: Connection reset by peer)
Makefile:964: recipe for target 'reg-tests' failed
make: *** [reg-tests] Error 1

but the next 4 runs were successul.


> сб, 18 янв. 2020 г. в 09:52, Martin Grigorov :
>
>>
>>
>> On Fri, Jan 17, 2020 at 11:17 PM Martin Grigorov <
>> martin.grigo...@gmail.com> wrote:
>>
>>>
>>>
>>> On Fri, Jan 17, 2020, 23:12 William Lallemand 
>>> wrote:
>>>
>>>> On Fri, Jan 17, 2020 at 08:50:27PM +0200, Martin Grigorov wrote:
>>>> > Testing with haproxy version: 2.2-dev0-70c5b0-123
>>>>
>>>> This binary was built with code from 1 week ago, it's normal that the
>>>> test does
>>>> not work since the fix was made this week.
>>>>
>>>
>>> I'm using the same steps to build HAProxy as from .travis-ci.yml
>>> I guess I have to add "make clean" in the beginning.
>>> I'll try it tomorrow! Thanks!
>>>
>>
>> That was it!
>> Everything is back to normal now!
>> Thank you, William!
>>
>>
>>>
>>>
>>>> --
>>>> William Lallemand
>>>>
>>>


Re: ARM(64) builds

2020-01-17 Thread Martin Grigorov
On Fri, Jan 17, 2020 at 11:17 PM Martin Grigorov 
wrote:

>
>
> On Fri, Jan 17, 2020, 23:12 William Lallemand 
> wrote:
>
>> On Fri, Jan 17, 2020 at 08:50:27PM +0200, Martin Grigorov wrote:
>> > Testing with haproxy version: 2.2-dev0-70c5b0-123
>>
>> This binary was built with code from 1 week ago, it's normal that the
>> test does
>> not work since the fix was made this week.
>>
>
> I'm using the same steps to build HAProxy as from .travis-ci.yml
> I guess I have to add "make clean" in the beginning.
> I'll try it tomorrow! Thanks!
>

That was it!
Everything is back to normal now!
Thank you, William!


>
>
>> --
>> William Lallemand
>>
>


Re: ARM(64) builds

2020-01-17 Thread Martin Grigorov
On Fri, Jan 17, 2020, 23:12 William Lallemand 
wrote:

> On Fri, Jan 17, 2020 at 08:50:27PM +0200, Martin Grigorov wrote:
> > Testing with haproxy version: 2.2-dev0-70c5b0-123
>
> This binary was built with code from 1 week ago, it's normal that the test
> does
> not work since the fix was made this week.
>

I'm using the same steps to build HAProxy as from .travis-ci.yml
I guess I have to add "make clean" in the beginning.
I'll try it tomorrow! Thanks!


> --
> William Lallemand
>


Re: ARM(64) builds

2020-01-17 Thread Martin Grigorov
Hi Илья,

On Fri, Jan 17, 2020 at 5:43 PM Илья Шипицин  wrote:

>
>
> пт, 17 янв. 2020 г. в 19:33, Martin Grigorov :
>
>>
>>
>> On Fri, Jan 17, 2020 at 4:13 PM Martin Grigorov <
>> martin.grigo...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> Today's build consistently fails on my ARM64 VM:
>>>
>>> ## Starting vtest ##
>>> Testing with haproxy version: 2.2-dev0-70c5b0-123
>>> #top  TEST reg-tests/mcli/mcli_start_progs.vtc FAILED (3.004) exit=2
>>> 1 tests failed, 0 tests skipped, 34 tests passed
>>> ## Gathering results ##
>>> ## Test case: reg-tests/mcli/mcli_start_progs.vtc ##
>>> ## test results in:
>>> "/tmp/haregtests-2020-01-17_14-01-45.SGkYcJ/vtc.12807.6adfff44"
>>>  h1   haproxy h1 PID file check failed:
>>>  h1   Bad exit status: 0x0100 exit 0x1 signal 0 core 0
>>> Makefile:964: recipe for target 'reg-tests' failed
>>> make: *** [reg-tests] Error 1
>>>
>>
>> git bisect blaims this commit:
>>
>> 25b569302167e71b32e569a2366027e8e320e80a is the first bad commit
>> commit 25b569302167e71b32e569a2366027e8e320e80a
>> Author: William Lallemand 
>> Date:   Tue Jan 14 15:38:43 2020 +0100
>>
>> REGTEST: mcli/mcli_start_progs: start 2 programs
>>
>> This regtest tests the issue #446 by starting 2 programs and checking
>> if
>> they exist in the "show proc" of the master CLI.
>>
>> Should be backported as far as 2.0.
>>
>>
>> https://travis-ci.com/haproxy/haproxy is green
>> https://cirrus-ci.com/github/haproxy/haproxy is green
>> and what is even more interesting is that
>> https://travis-ci.org/martin-g/haproxy/builds (my fork with enabled
>> ARM64 testing on TravisCI) also just passed (after few failures due to
>> timing issues (I guess)
>>
>
>
> timing out might happen because of
> https://github.com/haproxy/haproxy/commit/ac8147446c7a3d1aa607042bc782095b03bc8dc4
> your fork is 16 commits behind current master. try to rebase to master
>

Thanks!
I've updated HAProxy on my ARM64 VM but didn't update my fork at GitHub so
it was behind.
I've just did it and the build again passed successfully:
https://travis-ci.org/martin-g/haproxy/builds/638568217


>
>
>>
>>
>>> Regards,
>>> Martin
>>>
>>> On Fri, Jan 17, 2020 at 9:22 AM Илья Шипицин 
>>> wrote:
>>>
>>>> привет!
>>>>
>>>> пт, 17 янв. 2020 г. в 11:42, Martin Grigorov >>> >:
>>>>
>>>>> Привет Илья,
>>>>>
>>>>> On Thu, Jan 16, 2020 at 10:37 AM Илья Шипицин 
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> чт, 16 янв. 2020 г. в 13:26, Martin Grigorov <
>>>>>> martin.grigo...@gmail.com>:
>>>>>>
>>>>>>> Hi Илья,
>>>>>>>
>>>>>>> On Thu, Jan 16, 2020 at 10:19 AM Илья Шипицин 
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hello, Martin!
>>>>>>>>
>>>>>>>> btw, just curious, how is Apache Foundation (or you personally)
>>>>>>>> interested in all that ?
>>>>>>>> please do not blame me, I really like to know.
>>>>>>>>
>>>>>>>
>>>>>> ok, so you work in some company that is interested in haproxy on
>>>>>> ARM64.
>>>>>> you do not want to tell it's name, at least is it legal ? is it
>>>>>> related to some government ?
>>>>>> if "no" and "no", I guess most people won't ask any more questions :)
>>>>>>
>>>>>
>>>>> It is legal and I do not work for a government of any country!
>>>>>
>>>>>
>>>>>>
>>>>>> personally, I do not work at Haproxy Inc, I use haproxy, sometimes I
>>>>>> contribute to it.
>>>>>> Please do not consider me as an "official representative".
>>>>>>
>>>>>>
>>>>>> I'm interested in testing haproxy on ARM64, I planned to do so. I can
>>>>>> priorierize it according to your interest to it.
>>>>>> and yes, I can accept hardware donati

Re: ARM(64) builds

2020-01-17 Thread Martin Grigorov
Hi William,

On Fri, Jan 17, 2020 at 4:46 PM William Lallemand 
wrote:

> On Fri, Jan 17, 2020 at 04:33:22PM +0200, Martin Grigorov wrote:
> >
> > git bisect blaims this commit:
> >
> > 25b569302167e71b32e569a2366027e8e320e80a is the first bad commit
> > commit 25b569302167e71b32e569a2366027e8e320e80a
> > Author: William Lallemand 
> > Date:   Tue Jan 14 15:38:43 2020 +0100
> >
> > REGTEST: mcli/mcli_start_progs: start 2 programs
>
> Well that's the commit which introduces the vtc file so that's normal.
>
> >
> > https://travis-ci.com/haproxy/haproxy is green
> > https://cirrus-ci.com/github/haproxy/haproxy is green
> > and what is even more interesting is that
> > https://travis-ci.org/martin-g/haproxy/builds (my fork with enabled
> ARM64
> > testing on TravisCI) also just passed (after few failures due to timing
> > issues (I guess)
>
> I don't see anything regarding mcli_start_progs.vtc in this links, can you
> provide the output of
> `make reg-tests -- --debug reg-tests/mcli/mcli_start_progs.vtc` ?
>

Please find attached the output.

this two lines look bad in it:

***  h1   debug|[ALERT] 016/184150 (7188) : parsing [cur--1:0] : proxy
'MASTER', another server named 'cur--1' was already defined at line 0,
please use distinct names.
***  h1   debug|[ALERT] 016/184150 (7188) : Fatal errors found in
configuration.

$ grep -rnH 'cur'
/tmp/haregtests-2020-01-17_18-41-50.A6zVLb/vtc.7182.700b0bcd/
/tmp/haregtests-2020-01-17_18-41-50.A6zVLb/vtc.7182.700b0bcd/LOG:71:***  h1
  debug|[ALERT] 016/184150 (7188) : parsing [cur--1:0] : proxy 'MASTER',
another server named 'cur--1' was already defined at line 0, please use
distinct names.


   │ File:
/tmp/haregtests-2020-01-17_18-41-50.A6zVLb/vtc.7182.700b0bcd/h1/cfg
───┼─
   1   │ global
   2   │ stats socket
"/tmp/haregtests-2020-01-17_18-41-50.A6zVLb/vtc.7182.700b0bcd/h1/stats.sock"
level admin mode 600
   3   │ stats socket "fd@${cli}" level admin
   4   │
   5   │ global
   6   │ nbproc 1
   7   │ defaults
   8   │ mode http
   9   │  option http-use-htx
  10   │ timeout connect 1s
  11   │ timeout client  1s
  12   │ timeout server  1s
  13   │
  14   │ frontend myfrontend
  15   │ bind "fd@${my_fe}"
  16   │ default_backend test
  17   │
  18   │ backend test
  19   │ server www1 127.0.0.1:39247
  20   │
  21   │ program foo
  22   │ command sleep 10
  23   │
  24   │ program bar
  25   │ command sleep 10

Please let me know if you need more information!


Thanks
>
> --
> William Lallemand
>
env VTEST_PROGRAM=../vtest/vtest make reg-tests -- --debug 
reg-tests/mcli/mcli_start_progs.vtc

## Preparing to run tests ##
Testing with haproxy version: 2.2-dev0-70c5b0-123
Target : linux-glibc
Options : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT -PCRE2 
-PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE 
-STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL 
+GETADDRINFO -OPENSSL -LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY 
+TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER 
+PRCTL +THREAD_DUMP -EVPORTS
## Gathering tests to run ##
  Add test: reg-tests/mcli/mcli_start_progs.vtc
## Starting vtest ##
Testing with haproxy version: 2.2-dev0-70c5b0-123
 dT   0.000
*top  TEST reg-tests/mcli/mcli_start_progs.vtc starting
 top  extmacro def pwd=/home/ubuntu/git/haproxy/haproxy
 top  extmacro def no-htx=
 top  extmacro def localhost=127.0.0.1
 top  extmacro def bad_backend=127.0.0.1 45249
 top  extmacro def bad_ip=192.0.2.255
 top  macro def testdir=/home/ubuntu/git/haproxy/haproxy/reg-tests/mcli
 top  macro def 
tmpdir=/tmp/haregtests-2020-01-17_18-41-50.A6zVLb/vtc.7182.700b0bcd
**   top  === varnishtest "Try to start a master CLI with 2 programs"
*top  VTEST Try to start a master CLI with 2 programs
**   top  === feature ignore_unknown_macro
**   top  === server s1 {
**   s1   Starting server
 s1   macro def s1_addr=127.0.0.1
 s1   macro def s1_port=39247
 s1   macro def s1_sock=127.0.0.1 39247
*s1   Listen on 127.0.0.1 39247
**   top  === haproxy h1 -W -S -conf {
**   s1   Started on 127.0.0.1 39247 (1 iterations)
 h1   macro def h1_closed_sock=127.0.0.1 33279
 h1   macro def h1_closed_addr=127.0.0.1
 h1   macro def h1_closed_port=33279
 dT   

Re: ARM(64) builds

2020-01-17 Thread Martin Grigorov
On Fri, Jan 17, 2020 at 4:13 PM Martin Grigorov 
wrote:

> Hi all,
>
> Today's build consistently fails on my ARM64 VM:
>
> ## Starting vtest ##
> Testing with haproxy version: 2.2-dev0-70c5b0-123
> #top  TEST reg-tests/mcli/mcli_start_progs.vtc FAILED (3.004) exit=2
> 1 tests failed, 0 tests skipped, 34 tests passed
> ## Gathering results ##
> ## Test case: reg-tests/mcli/mcli_start_progs.vtc ##
> ## test results in:
> "/tmp/haregtests-2020-01-17_14-01-45.SGkYcJ/vtc.12807.6adfff44"
>  h1   haproxy h1 PID file check failed:
>  h1   Bad exit status: 0x0100 exit 0x1 signal 0 core 0
> Makefile:964: recipe for target 'reg-tests' failed
> make: *** [reg-tests] Error 1
>

git bisect blaims this commit:

25b569302167e71b32e569a2366027e8e320e80a is the first bad commit
commit 25b569302167e71b32e569a2366027e8e320e80a
Author: William Lallemand 
Date:   Tue Jan 14 15:38:43 2020 +0100

REGTEST: mcli/mcli_start_progs: start 2 programs

This regtest tests the issue #446 by starting 2 programs and checking if
they exist in the "show proc" of the master CLI.

Should be backported as far as 2.0.


https://travis-ci.com/haproxy/haproxy is green
https://cirrus-ci.com/github/haproxy/haproxy is green
and what is even more interesting is that
https://travis-ci.org/martin-g/haproxy/builds (my fork with enabled ARM64
testing on TravisCI) also just passed (after few failures due to timing
issues (I guess)


> Regards,
> Martin
>
> On Fri, Jan 17, 2020 at 9:22 AM Илья Шипицин  wrote:
>
>> привет!
>>
>> пт, 17 янв. 2020 г. в 11:42, Martin Grigorov :
>>
>>> Привет Илья,
>>>
>>> On Thu, Jan 16, 2020 at 10:37 AM Илья Шипицин 
>>> wrote:
>>>
>>>>
>>>>
>>>> чт, 16 янв. 2020 г. в 13:26, Martin Grigorov >>> >:
>>>>
>>>>> Hi Илья,
>>>>>
>>>>> On Thu, Jan 16, 2020 at 10:19 AM Илья Шипицин 
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Hello, Martin!
>>>>>>
>>>>>> btw, just curious, how is Apache Foundation (or you personally)
>>>>>> interested in all that ?
>>>>>> please do not blame me, I really like to know.
>>>>>>
>>>>>
>>>> ok, so you work in some company that is interested in haproxy on ARM64.
>>>> you do not want to tell it's name, at least is it legal ? is it related
>>>> to some government ?
>>>> if "no" and "no", I guess most people won't ask any more questions :)
>>>>
>>>
>>> It is legal and I do not work for a government of any country!
>>>
>>>
>>>>
>>>> personally, I do not work at Haproxy Inc, I use haproxy, sometimes I
>>>> contribute to it.
>>>> Please do not consider me as an "official representative".
>>>>
>>>>
>>>> I'm interested in testing haproxy on ARM64, I planned to do so. I can
>>>> priorierize it according to your interest to it.
>>>> and yes, I can accept hardware donation (even considering I'm not part
>>>> of Haproxy Inc).
>>>>
>>>> also, from my point of view, what would be really useful in your case
>>>> is testing not just official reg-tests, but also
>>>> your configs. reg-tests cover only partially. If you enable clang asan
>>>> in your own workload there's a chance to catch
>>>> something interesting (or, to make sure your own workload is ok). I can
>>>> help with that as well.
>>>>
>>>
>>> Thanks for the offer!
>>> I've discussed it with my managers.
>>> Our offer is to donate a VM that could be used as an official CI agent
>>> for the HAProxy project long term.
>>>
>>
>> I'd split this into short term and long term approach.
>>
>> if you need to start any time soon, I'd focus on your own workload
>> testing first. I would build stand which emulates
>> your production workload, compile haproxy using clang address sanitizer
>> and give it a try (functional testing, load testing, ...)
>>
>> I can help with that.
>>
>> As for long term solution, currently haproxy simply cannot attach any
>> dedicated build agent to their CI. travis-ci does not allow
>> attaching dedicated agents. And haproxy team is very conservative in
>> adding new CI servers.
>>
>> I think, I will add arm64 (most prob

Re: ARM(64) builds

2020-01-17 Thread Martin Grigorov
Hi all,

Today's build consistently fails on my ARM64 VM:

## Starting vtest ##
Testing with haproxy version: 2.2-dev0-70c5b0-123
#top  TEST reg-tests/mcli/mcli_start_progs.vtc FAILED (3.004) exit=2
1 tests failed, 0 tests skipped, 34 tests passed
## Gathering results ##
## Test case: reg-tests/mcli/mcli_start_progs.vtc ##
## test results in:
"/tmp/haregtests-2020-01-17_14-01-45.SGkYcJ/vtc.12807.6adfff44"
 h1   haproxy h1 PID file check failed:
 h1   Bad exit status: 0x0100 exit 0x1 signal 0 core 0
Makefile:964: recipe for target 'reg-tests' failed
make: *** [reg-tests] Error 1

Regards,
Martin

On Fri, Jan 17, 2020 at 9:22 AM Илья Шипицин  wrote:

> привет!
>
> пт, 17 янв. 2020 г. в 11:42, Martin Grigorov :
>
>> Привет Илья,
>>
>> On Thu, Jan 16, 2020 at 10:37 AM Илья Шипицин 
>> wrote:
>>
>>>
>>>
>>> чт, 16 янв. 2020 г. в 13:26, Martin Grigorov >> >:
>>>
>>>> Hi Илья,
>>>>
>>>> On Thu, Jan 16, 2020 at 10:19 AM Илья Шипицин 
>>>> wrote:
>>>>
>>>>>
>>>>> Hello, Martin!
>>>>>
>>>>> btw, just curious, how is Apache Foundation (or you personally)
>>>>> interested in all that ?
>>>>> please do not blame me, I really like to know.
>>>>>
>>>>
>>> ok, so you work in some company that is interested in haproxy on ARM64.
>>> you do not want to tell it's name, at least is it legal ? is it related
>>> to some government ?
>>> if "no" and "no", I guess most people won't ask any more questions :)
>>>
>>
>> It is legal and I do not work for a government of any country!
>>
>>
>>>
>>> personally, I do not work at Haproxy Inc, I use haproxy, sometimes I
>>> contribute to it.
>>> Please do not consider me as an "official representative".
>>>
>>>
>>> I'm interested in testing haproxy on ARM64, I planned to do so. I can
>>> priorierize it according to your interest to it.
>>> and yes, I can accept hardware donation (even considering I'm not part
>>> of Haproxy Inc).
>>>
>>> also, from my point of view, what would be really useful in your case is
>>> testing not just official reg-tests, but also
>>> your configs. reg-tests cover only partially. If you enable clang asan
>>> in your own workload there's a chance to catch
>>> something interesting (or, to make sure your own workload is ok). I can
>>> help with that as well.
>>>
>>
>> Thanks for the offer!
>> I've discussed it with my managers.
>> Our offer is to donate a VM that could be used as an official CI agent
>> for the HAProxy project long term.
>>
>
> I'd split this into short term and long term approach.
>
> if you need to start any time soon, I'd focus on your own workload testing
> first. I would build stand which emulates
> your production workload, compile haproxy using clang address sanitizer
> and give it a try (functional testing, load testing, ...)
>
> I can help with that.
>
> As for long term solution, currently haproxy simply cannot attach any
> dedicated build agent to their CI. travis-ci does not allow
> attaching dedicated agents. And haproxy team is very conservative in
> adding new CI servers.
>
> I think, I will add arm64 (most probably openssl-1.1.1 only for now) soon.
> Also, I'm going to investigate your libressl failures.
>
> so, dedicated vm definetly will help in troubleshooting issues, for manual
> builds. It would save bunch of time. I do not mind if you will
> add my ssh to that vm.
>
>
> also, I requested access to Linaro.
>
>
>>
>> You can use Linaro for short term testing though.
>> https://www.linaro.cloud/
>> Here you can request free VM for short periods:
>> https://servicedesk.linaro.org/servicedesk/customer/portal/19/create/265
>> P.S. Linaro is not my employer!
>>
>>
>> Regards,
>> Martin
>>
>>
>>>
>>>>
>>>>>
>>>> @apache.org is just one of my several emails. And it is the default
>>>> one in my email client.
>>>> ASF is not related anyhow to my participation here.
>>>> If I used my work email then it might look like some kind of
>>>> advertisement. I'd like to avoid that!
>>>> Next time I will use my @gmail.com one, as more neutral. Actually I've
>>>> used the GMail one when registering to 

Re: ARM(64) builds

2020-01-16 Thread Martin Grigorov
Привет Илья,

On Thu, Jan 16, 2020 at 10:37 AM Илья Шипицин  wrote:

>
>
> чт, 16 янв. 2020 г. в 13:26, Martin Grigorov :
>
>> Hi Илья,
>>
>> On Thu, Jan 16, 2020 at 10:19 AM Илья Шипицин 
>> wrote:
>>
>>>
>>> Hello, Martin!
>>>
>>> btw, just curious, how is Apache Foundation (or you personally)
>>> interested in all that ?
>>> please do not blame me, I really like to know.
>>>
>>
> ok, so you work in some company that is interested in haproxy on ARM64.
> you do not want to tell it's name, at least is it legal ? is it related to
> some government ?
> if "no" and "no", I guess most people won't ask any more questions :)
>

It is legal and I do not work for a government of any country!


>
> personally, I do not work at Haproxy Inc, I use haproxy, sometimes I
> contribute to it.
> Please do not consider me as an "official representative".
>
>
> I'm interested in testing haproxy on ARM64, I planned to do so. I can
> priorierize it according to your interest to it.
> and yes, I can accept hardware donation (even considering I'm not part of
> Haproxy Inc).
>
> also, from my point of view, what would be really useful in your case is
> testing not just official reg-tests, but also
> your configs. reg-tests cover only partially. If you enable clang asan in
> your own workload there's a chance to catch
> something interesting (or, to make sure your own workload is ok). I can
> help with that as well.
>

Thanks for the offer!
I've discussed it with my managers.
Our offer is to donate a VM that could be used as an official CI agent for
the HAProxy project long term.

You can use Linaro for short term testing though.
https://www.linaro.cloud/
Here you can request free VM for short periods:
https://servicedesk.linaro.org/servicedesk/customer/portal/19/create/265
P.S. Linaro is not my employer!


Regards,
Martin


>
>>
>>>
>> @apache.org is just one of my several emails. And it is the default one
>> in my email client.
>> ASF is not related anyhow to my participation here.
>> If I used my work email then it might look like some kind of
>> advertisement. I'd like to avoid that!
>> Next time I will use my @gmail.com one, as more neutral. Actually I've
>> used the GMail one when registering to this mailing list, so probably the
>> post from @apache has been moderated. I'll be more careful next time!
>>
>> Thanks, Илья!
>>
>> Regards,
>> Martin
>>
>>
>>>
>>> чт, 16 янв. 2020 г. в 12:32, Martin Grigorov :
>>>
>>>> Hello HAProxy developers,
>>>>
>>>> At work we are going to use more and more ARM64 based servers and
>>>> HAProxy is one of the main products we rely on.
>>>> So I went checking whether HAProxy project has a CI environment for
>>>> testing on ARM architecture.
>>>>
>>>
>>> we are looking towards
>>> https://docs.travis-ci.com/user/multi-cpu-architectures
>>>
>>>
>>>> I've found this recent discussion:
>>>> https://www.mail-archive.com/haproxy@formilux.org/msg35302.html  (I
>>>> didn't find a way how to continue on the same mail thread, so I'm starting
>>>> a new one. Apologies for that!).
>>>>
>>>
>>> I played with arm64 for a while, the issue I encountered is travis-ci
>>> cache key, i.e. we cache openssl builds between our builds.
>>> so travis used the same cache key for both amd64 and arm64 builds (this
>>> might have changed recently, I did not check yet)
>>>
>>> arm64 is in my queue (as well as recent s390x arch from travis), hope to
>>> get back to it within month or so.
>>>
>>>
>>>> From this discussion and from
>>>> https://github.com/haproxy/haproxy/blob/master/.travis.yml I
>>>> understand that there is no public CI in use (i.e. TravisCI or CirrusCI)
>>>> but some of the developers run some tests locally regularly.
>>>>
>>>
>>> it is not completely true.
>>> there's public CI. we do not use github PR machinery, so sometimes tests
>>> fail after push to master branch. it is considered as ok, failures are
>>> fixed pretty fast.
>>> for example, see
>>> https://www.mail-archive.com/haproxy@formilux.org/msg35910.html
>>> it was just perfect, failure detected using CI and fixed within few
>>> days. no customers affected.
>>>
>>>
>>>> I've forked the project and tested on TravisCI (
>>>> https://travis-ci

Re: ARM(64) builds

2020-01-16 Thread Martin Grigorov
Hi Илья,

On Thu, Jan 16, 2020 at 10:19 AM Илья Шипицин  wrote:

>
> Hello, Martin!
>
> btw, just curious, how is Apache Foundation (or you personally) interested
> in all that ?
> please do not blame me, I really like to know.
>
>
@apache.org is just one of my several emails. And it is the default one in
my email client.
ASF is not related anyhow to my participation here.
If I used my work email then it might look like some kind of advertisement.
I'd like to avoid that!
Next time I will use my @gmail.com one, as more neutral. Actually I've used
the GMail one when registering to this mailing list, so probably the post
from @apache has been moderated. I'll be more careful next time!

Thanks, Илья!

Regards,
Martin


>
> чт, 16 янв. 2020 г. в 12:32, Martin Grigorov :
>
>> Hello HAProxy developers,
>>
>> At work we are going to use more and more ARM64 based servers and HAProxy
>> is one of the main products we rely on.
>> So I went checking whether HAProxy project has a CI environment for
>> testing on ARM architecture.
>>
>
> we are looking towards
> https://docs.travis-ci.com/user/multi-cpu-architectures
>
>
>> I've found this recent discussion:
>> https://www.mail-archive.com/haproxy@formilux.org/msg35302.html  (I
>> didn't find a way how to continue on the same mail thread, so I'm starting
>> a new one. Apologies for that!).
>>
>
> I played with arm64 for a while, the issue I encountered is travis-ci
> cache key, i.e. we cache openssl builds between our builds.
> so travis used the same cache key for both amd64 and arm64 builds (this
> might have changed recently, I did not check yet)
>
> arm64 is in my queue (as well as recent s390x arch from travis), hope to
> get back to it within month or so.
>
>
>> From this discussion and from
>> https://github.com/haproxy/haproxy/blob/master/.travis.yml I understand
>> that there is no public CI in use (i.e. TravisCI or CirrusCI) but some of
>> the developers run some tests locally regularly.
>>
>
> it is not completely true.
> there's public CI. we do not use github PR machinery, so sometimes tests
> fail after push to master branch. it is considered as ok, failures are
> fixed pretty fast.
> for example, see
> https://www.mail-archive.com/haproxy@formilux.org/msg35910.html
> it was just perfect, failure detected using CI and fixed within few days.
> no customers affected.
>
>
>> I've forked the project and tested on TravisCI (
>> https://travis-ci.org/martin-g/haproxy/builds) but unfortunately the
>> builds were not very stable:
>> 1) some tests fail sometimes. I guess it is because of some timing issues
>> For example:
>> - https://travis-ci.org/martin-g/haproxy/jobs/636745241
>> - https://travis-ci.org/martin-g/haproxy/jobs/636750676
>> - https://travis-ci.org/martin-g/haproxy/jobs/636763346
>>
>
> that's very interesting. I'll have a look.
>
>
>
>> 2) There was some weird issue on testing with LibreSSL
>> The output redirect at
>> https://github.com/haproxy/haproxy/blob/bb9da0b8e23c46caab34fe6005b66fa8065fe3ac/.travis.yml#L96
>>  for
>> some reason got stuck the build. I've removed temporarily the output
>> redirects and then it passed. So, it looks like some issue with TravisCI
>> environment.
>>
>
> arm64 is slower, I guess we should add "*travis*_*wait* *30" *to
> build-ssl.sh script
> thank for the hint
>
>
>>
>> In addition I've run the build and tests on one of our machines and all
>> was OK!
>>
>> My question to you is: Are you happy with your current way of testing ARM
>> architectures or you want to add more ?
>> Here are some options:
>> 1) enable TravisCI
>>
>
> already done
>
> https://travis-ci.com/haproxy/haproxy
>
>
>> 2) my company is willing to donate an ARM64 based VM, if you are
>> interested.
>>
>
> I do not work at Haproxy Inc :)
> Willy ?
>
>
>> You will have a SSH access and a user with sudo permissions to install
>> anything that is needed.
>> The spec is aarch64 8 cores CPU 2GHz (Kunpeng), 16GB RAM, 500G disk space
>> and 5M network bandwidth. The OS could be any of CentOS 7.4/7.5/7.6,
>> EulerOS 2.8, Fedora 29, OpenSuse 15.0 and Ubuntu 16.04/18.04.
>>
>> In both cases it will be ARM64. From the earlier mail discussion I
>> understand you would prefer ARM32.
>>
>
> as for myself, I prefer both arm64 and arm32.
> however, both AMD64 and ARM64 are the same 64 bits. both of them are
> little-endian. but you mentioned at least 3 builds with ARM64 failing (we
> have those tests passing on AMD64)!
>
>
>
>>
>> Kind regards,
>> Martin
>>
>


ARM(64) builds

2020-01-15 Thread Martin Grigorov
Hello HAProxy developers,

At work we are going to use more and more ARM64 based servers and HAProxy
is one of the main products we rely on.
So I went checking whether HAProxy project has a CI environment for testing
on ARM architecture.
I've found this recent discussion:
https://www.mail-archive.com/haproxy@formilux.org/msg35302.html  (I didn't
find a way how to continue on the same mail thread, so I'm starting a new
one. Apologies for that!).
>From this discussion and from
https://github.com/haproxy/haproxy/blob/master/.travis.yml I understand
that there is no public CI in use (i.e. TravisCI or CirrusCI) but some of
the developers run some tests locally regularly.
I've forked the project and tested on TravisCI (
https://travis-ci.org/martin-g/haproxy/builds) but unfortunately the builds
were not very stable:
1) some tests fail sometimes. I guess it is because of some timing issues
For example:
- https://travis-ci.org/martin-g/haproxy/jobs/636745241
- https://travis-ci.org/martin-g/haproxy/jobs/636750676
- https://travis-ci.org/martin-g/haproxy/jobs/636763346
2) There was some weird issue on testing with LibreSSL
The output redirect at
https://github.com/haproxy/haproxy/blob/bb9da0b8e23c46caab34fe6005b66fa8065fe3ac/.travis.yml#L96
for
some reason got stuck the build. I've removed temporarily the output
redirects and then it passed. So, it looks like some issue with TravisCI
environment.

In addition I've run the build and tests on one of our machines and all was
OK!

My question to you is: Are you happy with your current way of testing ARM
architectures or you want to add more ?
Here are some options:
1) enable TravisCI
2) my company is willing to donate an ARM64 based VM, if you are interested.
You will have a SSH access and a user with sudo permissions to install
anything that is needed.
The spec is aarch64 8 cores CPU 2GHz (Kunpeng), 16GB RAM, 500G disk space
and 5M network bandwidth. The OS could be any of CentOS 7.4/7.5/7.6,
EulerOS 2.8, Fedora 29, OpenSuse 15.0 and Ubuntu 16.04/18.04.

In both cases it will be ARM64. From the earlier mail discussion I
understand you would prefer ARM32.

Kind regards,
Martin