[Bug 1804069] Re: haproxy fails on arm64 due to alignment error
Bionic verification arm system: root@bionic-haproxy-1804069:~# uname -a Linux bionic-haproxy-1804069 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:32:18 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux using affected package at first: root@bionic-haproxy-1804069:~# apt-cache policy haproxy haproxy: Installed: 1.8.8-1ubuntu0.3 Candidate: 1.8.8-1ubuntu0.3 Version table: *** 1.8.8-1ubuntu0.3 500 500 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 Packages wget fails as expected: root@bionic-haproxy-1804069:~# wget -t1 http://localhost:8080 --2019-02-06 11:57:56-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... No data received. Giving up. root@bionic-haproxy-1804069:~# echo $? 4 haproxy logs show errors: Feb 6 11:57:57 bionic-haproxy-1804069 haproxy[10191]: [ALERT] 036/115744 (10191) : Current worker 10192 exited with code 135 Feb 6 11:57:57 bionic-haproxy-1804069 haproxy[10191]: [ALERT] 036/115744 (10191) : exit-on-failure: killing every workers with SIGTERM Feb 6 11:57:57 bionic-haproxy-1804069 haproxy[10191]: [WARNING] 036/115744 (10191) : All workers exited. Exiting... (135) Using updated package: root@bionic-haproxy-1804069:~# apt-cache policy haproxy haproxy: Installed: 1.8.8-1ubuntu0.4 Candidate: 1.8.8-1ubuntu0.4 Version table: *** 1.8.8-1ubuntu0.4 500 500 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/main arm64 Packages wget now works: root@bionic-haproxy-1804069:~# wget -t1 http://localhost:8080 --2019-02-06 12:00:17-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... 200 OK Length: 10918 (11K) [text/html] Saving to: ‘index.html’ index.html 100%[>] 10.66K --.-KB/sin 0s 2019-02-06 12:00:17 (165 MB/s) - ‘index.html’ saved [10918/10918] root@bionic-haproxy-1804069:~# echo $? 0 and haproxy logs stay silent. bionic verification succeeded -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1804069 Title: haproxy fails on arm64 due to alignment error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1804069/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1804069] Re: haproxy fails on arm64 due to alignment error
Cosmic verification Using arm64: root@cosmic-haproxy-1804069:~# uname -a Linux cosmic-haproxy-1804069 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:32:18 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux First, confirming the bug: root@cosmic-haproxy-1804069:~# apt-cache policy haproxy haproxy: Installed: 1.8.13-2ubuntu0.1 Candidate: 1.8.13-2ubuntu0.1 Version table: *** 1.8.13-2ubuntu0.1 500 500 http://ports.ubuntu.com/ubuntu-ports cosmic-updates/main arm64 Packages wget fails as expected: root@cosmic-haproxy-1804069:~# wget -t1 http://localhost:8080 --2019-02-06 11:43:52-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... No data received. Giving up. root@cosmic-haproxy-1804069:~# echo $? 4 Haproxy error logs show: Feb 6 11:43:52 cosmic-haproxy-1804069 haproxy[4539]: [ALERT] 036/114339 (4539) : Current worker 4540 exited with code 135 Feb 6 11:43:52 cosmic-haproxy-1804069 haproxy[4539]: [ALERT] 036/114339 (4539) : exit-on-failure: killing every workers with SIGTERM Feb 6 11:43:52 cosmic-haproxy-1804069 haproxy[4539]: [WARNING] 036/114339 (4539) : All workers exited. Exiting... (135) Upgraded haproxy package: root@cosmic-haproxy-1804069:~# apt-cache policy haproxy haproxy: Installed: 1.8.13-2ubuntu0.2 Candidate: 1.8.13-2ubuntu0.2 Version table: *** 1.8.13-2ubuntu0.2 500 500 http://ports.ubuntu.com/ubuntu-ports cosmic-proposed/main arm64 Packages wget now works: root@cosmic-haproxy-1804069:~# wget -t1 http://localhost:8080 --2019-02-06 11:46:19-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... 200 OK Length: 10918 (11K) [text/html] Saving to: ‘index.html’ index.html 100%[>] 10.66K --.-KB/sin 0s 2019-02-06 11:46:19 (154 MB/s) - ‘index.html’ saved [10918/10918] root@cosmic-haproxy-1804069:~# echo $? 0 root@cosmic-haproxy-1804069:~# And the haproxy logs stay quiet. Cosmic verification succeeded. ** Tags removed: verification-needed-cosmic ** Tags added: verification-done-cosmic -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1804069 Title: haproxy fails on arm64 due to alignment error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1804069/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1804069] Re: haproxy fails on arm64 due to alignment error
Ok, thanks for trying. I can complete the verification and will post results later today. -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1804069 Title: haproxy fails on arm64 due to alignment error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1804069/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1804069] Re: haproxy fails on arm64 due to alignment error
You can boot some other release for the host and use a lxd container for the cosmic test. -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1804069 Title: haproxy fails on arm64 due to alignment error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1804069/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1804069] Re: haproxy fails on arm64 due to alignment error
Jonathan, could you please detail what exactly you tested, and with which package version? The SRU team will appreciate that level of detail, which gives more confidence in the update. -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1804069 Title: haproxy fails on arm64 due to alignment error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1804069/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1804069] Re: haproxy fails on arm64 due to alignment error
** Description changed: [Impact] haproxy as shipped with bionic and cosmic doesn't work on arm64 architectures, crashing the moment it serves a request. [Test Case] * install haproxy and apache in an up-to-date ubuntu release you are testing, in an arm64 system: sudo apt update && sudo apt dist-upgrade -y && sudo apt install haproxy apache2 -y * Create /etc/haproxy/haproxy.cfg with the following contents: global chroot /var/lib/haproxy user haproxy group haproxy daemon maxconn 4096 defaults log global option dontlognull option redispatch retries 3 timeout client 50s timeout connect 10s timeout http-request 5s timeout server 50s maxconn 4096 frontend test-front bind *:8080 mode http default_backend test-back backend test-back mode http stick store-request src stick-table type ip size 256k expire 30m server test-1 localhost:80 * in one terminal, keep tailing the (still nonexistent) haproxy log file: tail -F /var/log/haproxy.log * in another terminal, restart haproxy: sudo systemctl restart haproxy * The haproxy log will become live, and already show errors: Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [WARNING] 023/191958 (2286) : Exiting Master process... Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [ALERT] 023/191958 (2286) : Current worker 2287 exited with code 143 Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [WARNING] 023/191958 (2286) : All workers exited. Exiting... (143) * Run wget to try to fetch the apache frontpage, via haproxy, limited to one attempt. It will fail: $ wget -t1 http://localhost:8080 --2019-01-24 19:23:51-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... No data received. Giving up. $ echo $? 4 * the haproxy logs will show errors: Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [ALERT] 023/192351 (6411) : Current worker 6412 exited with code 135 Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [ALERT] 023/192351 (6411) : exit-on-failure: killing every workers with SIGTERM Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [WARNING] 023/192351 (6411) : All workers exited. Exiting... (135) * Update the haproxy package and try the wget one more time. This time it will work, and the haproxy logs will stay silent: $ wget -t1 http://localhost:8080 --2019-01-24 19:26:14-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... 200 OK Length: 10918 (11K) [text/html] Saving to: ‘index.html’ index.html 100%[>] 10.66K --.-KB/sin 0s 2019-01-24 19:26:14 (75.3 MB/s) - ‘index.html’ saved [10918/10918] [Regression Potential] Patch was applied upstream in 1.8.15 and is available in the same form in the latest 1.8.17 release. The patch is a bit low level, but seems to have been well understood. [Other Info] - It's bad that our DEP8 test didn't catch this, since all it does is start the service and briefly talk to it, but there is no actual proxying going on. + After writing the testing instructions for this bug, I decided they could be easily converted to a DEP8 test, which I did and included in this SRU. This new test, very simple but effective, shows that arm64 is working, and that the other architectures didn't break. [Original Description] This fault was reported via the haproxy mailing list https://www.mail- archive.com/hapr...@formilux.org/msg31749.html And then patched in the haproxy code here https://github.com/haproxy/haproxy/commit/52dabbc4fad338233c7f0c96f977a43f8f81452a Without this patch haproxy is not functional on aarch64/arm64. Experimental deployments of openstack-ansible on arm64 fail because of this bug, and without a fix applied to the ubuntu bionic packages we cannot proceed further as the openstack CI only installs the upstream ubuntu distribution packages. -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1804069 Title: haproxy fails on arm64 due to alignment error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1804069/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1804069] Re: haproxy fails on arm64 due to alignment error
** Description changed: [Impact] haproxy as shipped with bionic and cosmic doesn't work on arm64 architectures, crashing the moment it serves a request. [Test Case] * install haproxy and apache in an up-to-date ubuntu release you are testing, in an arm64 system: sudo apt update && sudo apt dist-upgrade -y && sudo apt install haproxy apache2 -y * Create /etc/haproxy/haproxy.cfg with the following contents: global chroot /var/lib/haproxy user haproxy group haproxy daemon maxconn 4096 defaults log global option dontlognull option redispatch retries 3 timeout client 50s timeout connect 10s timeout http-request 5s timeout server 50s maxconn 4096 frontend test-front bind *:8080 mode http default_backend test-back backend test-back mode http stick store-request src stick-table type ip size 256k expire 30m server test-1 localhost:80 * in one terminal, keep tailing the (still nonexistent) haproxy log file: tail -F /var/log/haproxy.log * in another terminal, restart haproxy: sudo systemctl restart haproxy * The haproxy log will become live, and already show errors: Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [WARNING] 023/191958 (2286) : Exiting Master process... Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [ALERT] 023/191958 (2286) : Current worker 2287 exited with code 143 Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [WARNING] 023/191958 (2286) : All workers exited. Exiting... (143) * Run wget to try to fetch the apache frontpage, via haproxy, limited to one attempt. It will fail: $ wget -t1 http://localhost:8080 --2019-01-24 19:23:51-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... No data received. Giving up. $ echo $? 4 * the haproxy logs will show errors: Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [ALERT] 023/192351 (6411) : Current worker 6412 exited with code 135 Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [ALERT] 023/192351 (6411) : exit-on-failure: killing every workers with SIGTERM Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [WARNING] 023/192351 (6411) : All workers exited. Exiting... (135) * Update the haproxy package and try the wget one more time. This time it will work, and the haproxy logs will stay silent: $ wget -t1 http://localhost:8080 --2019-01-24 19:26:14-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... 200 OK Length: 10918 (11K) [text/html] Saving to: ‘index.html’ index.html 100%[>] 10.66K --.-KB/sin 0s 2019-01-24 19:26:14 (75.3 MB/s) - ‘index.html’ saved [10918/10918] [Regression Potential] - - * discussion of how regressions are most likely to manifest as a result - of this change. - - * It is assumed that any SRU candidate patch is well-tested before - upload and has a low overall risk of regression, but it's important - to make the effort to think about what ''could'' happen in the - event of a regression. - - * This both shows the SRU team that the risks have been considered, - and provides guidance to testers in regression-testing the SRU. + Patch was applied upstream in 1.8.15 and is available in the same form in the latest 1.8.17 release. The patch is a bit low level, but seems to have been well understood. [Other Info] - - * Anything else you think is useful to include - * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board - * and address these questions in advance + It's bad that our DEP8 test didn't catch this, since all it does is start the service and briefly talk to it, but there is no actual proxying going on. [Original Description] This fault was reported via the haproxy mailing list https://www.mail- archive.com/hapr...@formilux.org/msg31749.html And then patched in the haproxy code here https://github.com/haproxy/haproxy/commit/52dabbc4fad338233c7f0c96f977a43f8f81452a Without this patch haproxy is not functional on aarch64/arm64. Experimental deployments of openstack-ansible on arm64 fail because of this bug, and without a fix applied to the ubuntu bionic packages we cannot proceed further as the openstack CI only installs the upstream ubuntu distribution packages. -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to the bug report.
[Bug 1804069] Re: haproxy fails on arm64 due to alignment error
** Description changed: [Impact] - - * An explanation of the effects of the bug on users and - - * justification for backporting the fix to the stable release. - - * In addition, it is helpful, but not required, to include an - explanation of how the upload fixes this bug. + haproxy as shipped with bionic and cosmic doesn't work on arm64 architectures, crashing the moment it serves a request. [Test Case] * install haproxy and apache in an up-to-date ubuntu release you are testing, in an arm64 system: sudo apt update && sudo apt dist-upgrade -y && sudo apt install haproxy apache2 -y * Create /etc/haproxy/haproxy.cfg with the following contents: global chroot /var/lib/haproxy user haproxy group haproxy daemon maxconn 4096 defaults log global option dontlognull option redispatch retries 3 timeout client 50s timeout connect 10s timeout http-request 5s timeout server 50s maxconn 4096 frontend test-front bind *:8080 mode http default_backend test-back backend test-back mode http stick store-request src stick-table type ip size 256k expire 30m server test-1 localhost:80 * in one terminal, keep tailing the (still nonexistent) haproxy log file: tail -F /var/log/haproxy.log * in another terminal, restart haproxy: sudo systemctl restart haproxy * The haproxy log will become live, and already show errors: Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [WARNING] 023/191958 (2286) : Exiting Master process... Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [ALERT] 023/191958 (2286) : Current worker 2287 exited with code 143 Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [WARNING] 023/191958 (2286) : All workers exited. Exiting... (143) * Run wget to try to fetch the apache frontpage, via haproxy, limited to one attempt. It will fail: $ wget -t1 http://localhost:8080 --2019-01-24 19:23:51-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... No data received. Giving up. $ echo $? 4 * the haproxy logs will show errors: Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [ALERT] 023/192351 (6411) : Current worker 6412 exited with code 135 Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [ALERT] 023/192351 (6411) : exit-on-failure: killing every workers with SIGTERM Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [WARNING] 023/192351 (6411) : All workers exited. Exiting... (135) * Update the haproxy package and try the wget one more time. This time it will work, and the haproxy logs will stay silent: $ wget -t1 http://localhost:8080 --2019-01-24 19:26:14-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... 200 OK Length: 10918 (11K) [text/html] Saving to: ‘index.html’ index.html 100%[>] 10.66K --.-KB/sin 0s 2019-01-24 19:26:14 (75.3 MB/s) - ‘index.html’ saved [10918/10918] [Regression Potential] * discussion of how regressions are most likely to manifest as a result of this change. * It is assumed that any SRU candidate patch is well-tested before upload and has a low overall risk of regression, but it's important to make the effort to think about what ''could'' happen in the event of a regression. * This both shows the SRU team that the risks have been considered, and provides guidance to testers in regression-testing the SRU. [Other Info] * Anything else you think is useful to include * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board * and address these questions in advance [Original Description] This fault was reported via the haproxy mailing list https://www.mail- archive.com/hapr...@formilux.org/msg31749.html And then patched in the haproxy code here https://github.com/haproxy/haproxy/commit/52dabbc4fad338233c7f0c96f977a43f8f81452a Without this patch haproxy is not functional on aarch64/arm64. Experimental deployments of openstack-ansible on arm64 fail because of this bug, and without a fix applied to the ubuntu bionic packages we cannot proceed further as the openstack CI only installs the upstream ubuntu distribution packages. -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1804069 Title: haproxy fails on arm64 due to alignment error To
[Bug 1804069] Re: haproxy fails on arm64 due to alignment error
** Description changed: [Impact] * An explanation of the effects of the bug on users and * justification for backporting the fix to the stable release. * In addition, it is helpful, but not required, to include an explanation of how the upload fixes this bug. [Test Case] * install haproxy and apache in an up-to-date ubuntu release you are testing, in an arm64 system: sudo apt update && sudo apt dist-upgrade -y && sudo apt install haproxy apache2 -y * Create /etc/haproxy/haproxy.cfg with the following contents: global - chroot /var/lib/haproxy - user haproxy - group haproxy - daemon - maxconn 4096 + chroot /var/lib/haproxy + user haproxy + group haproxy + daemon + maxconn 4096 defaults - log global - option dontlognull - option redispatch - retries 3 - timeout client 50s - timeout connect 10s - timeout http-request 5s - timeout server 50s - maxconn 4096 + log global + option dontlognull + option redispatch + retries 3 + timeout client 50s + timeout connect 10s + timeout http-request 5s + timeout server 50s + maxconn 4096 frontend test-front - bind *:8080 - mode http - default_backend test-back + bind *:8080 + mode http + default_backend test-back backend test-back - mode http - stick store-request src - stick-table type ip size 256k expire 30m - server test-1 localhost:80 - + mode http + stick store-request src + stick-table type ip size 256k expire 30m + server test-1 localhost:80 * in one terminal, keep tailing the (still nonexistent) haproxy log file: tail -F /var/log/haproxy.log - * in another terminal, restart haproxy and tail its logs: + * in another terminal, restart haproxy: sudo systemctl restart haproxy - tail -f /var/log/haproxy.log * in another terminal, start haproxy: sudo systemctl restart haproxy * The haproxy log fill become live, and already show errors: Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [WARNING] 023/191958 (2286) : Exiting Master process... Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [ALERT] 023/191958 (2286) : Current worker 2287 exited with code 143 Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [WARNING] 023/191958 (2286) : All workers exited. Exiting... (143) * Run wget to try to fetch the apache frontpage, via haproxy, limited to one attempt. It will fail: $ wget -t1 http://localhost:8080 --2019-01-24 19:23:51-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... No data received. Giving up. $ echo $? 4 * the haproxy logs will show errors: - Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [ALERT] 023/192351 (6411) : Current worker 6412 exited with code 135 - Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [ALERT] 023/192351 (6411) : exit-on-failure: killing every workers with SIGTERM - Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [WARNING] 023/192351 (6411) : All workers exited. Exiting... (135) + Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [ALERT] 023/192351 (6411) : Current worker 6412 exited with code 135 + Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [ALERT] 023/192351 (6411) : exit-on-failure: killing every workers with SIGTERM + Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [WARNING] 023/192351 (6411) : All workers exited. Exiting... (135) * Update the haproxy package and try the wget one more time. This time it will work, and the haproxy logs will stay silent: wget -t1 http://localhost:8080 --2019-01-24 19:26:14-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... 200 OK Length: 10918 (11K) [text/html] Saving to: ‘index.html’ index.html 100%[>] 10.66K --.-KB/sin 0s 2019-01-24 19:26:14 (75.3 MB/s) - ‘index.html’ saved [10918/10918] - [Regression Potential] * discussion of how regressions are most likely to manifest as a result of this change. * It is assumed that any SRU candidate patch is well-tested before upload and has a low overall risk of regression, but it's important to make the effort to think about what ''could'' happen in the event of a
[Bug 1804069] Re: haproxy fails on arm64 due to alignment error
** Description changed: [Impact] - * An explanation of the effects of the bug on users and + * An explanation of the effects of the bug on users and - * justification for backporting the fix to the stable release. + * justification for backporting the fix to the stable release. - * In addition, it is helpful, but not required, to include an -explanation of how the upload fixes this bug. + * In addition, it is helpful, but not required, to include an + explanation of how the upload fixes this bug. [Test Case] - * detailed instructions how to reproduce the bug + * install haproxy and apache in an up-to-date ubuntu release you are testing, in an arm64 system: + sudo apt update && sudo apt dist-upgrade -y && sudo apt install haproxy apache2 -y - * these should allow someone who is not familiar with the affected -package to reproduce the bug and verify that the updated package fixes -the problem. + * Create /etc/haproxy/haproxy.cfg with the following contents: + global + chroot /var/lib/haproxy + user haproxy + group haproxy + daemon + maxconn 4096 + + defaults + log global + option dontlognull + option redispatch + retries 3 + timeout client 50s + timeout connect 10s + timeout http-request 5s + timeout server 50s + maxconn 4096 + + frontend test-front + bind *:8080 + mode http + default_backend test-back + + backend test-back + mode http + stick store-request src + stick-table type ip size 256k expire 30m + server test-1 localhost:80 + + + * in one terminal, keep tailing the (still nonexistent) haproxy log file: + tail -F /var/log/haproxy.log + + * in another terminal, restart haproxy and tail its logs: + sudo systemctl restart haproxy + tail -f /var/log/haproxy.log + + * in another terminal, start haproxy: + sudo systemctl restart haproxy + + * The haproxy log fill become live, and already show errors: + Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [WARNING] 023/191958 (2286) : Exiting Master process... + Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [ALERT] 023/191958 (2286) : Current worker 2287 exited with code 143 + Jan 24 19:22:23 cosmic-haproxy-1804069 haproxy[2286]: [WARNING] 023/191958 (2286) : All workers exited. Exiting... (143) + + * Run wget to try to fetch the apache frontpage, via haproxy, limited to one attempt. It will fail: + $ wget -t1 http://localhost:8080 + --2019-01-24 19:23:51-- http://localhost:8080/ + Resolving localhost (localhost)... 127.0.0.1 + Connecting to localhost (localhost)|127.0.0.1|:8080... connected. + HTTP request sent, awaiting response... No data received. + Giving up. + $ echo $? + 4 + + * the haproxy logs will show errors: + Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [ALERT] 023/192351 (6411) : Current worker 6412 exited with code 135 + Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [ALERT] 023/192351 (6411) : exit-on-failure: killing every workers with SIGTERM + Jan 24 19:24:36 cosmic-haproxy-1804069 haproxy[6411]: [WARNING] 023/192351 (6411) : All workers exited. Exiting... (135) + + * Update the haproxy package and try the wget one more time. This time it will work, and the haproxy logs will stay silent: + wget -t1 http://localhost:8080 + --2019-01-24 19:26:14-- http://localhost:8080/ + Resolving localhost (localhost)... 127.0.0.1 + Connecting to localhost (localhost)|127.0.0.1|:8080... connected. + HTTP request sent, awaiting response... 200 OK + Length: 10918 (11K) [text/html] + Saving to: ‘index.html’ + + index.html + 100%[>] + 10.66K --.-KB/sin 0s + + 2019-01-24 19:26:14 (75.3 MB/s) - ‘index.html’ saved [10918/10918] + [Regression Potential] - * discussion of how regressions are most likely to manifest as a result + * discussion of how regressions are most likely to manifest as a result of this change. - * It is assumed that any SRU candidate patch is well-tested before -upload and has a low overall risk of regression, but it's important -to make the effort to think about what ''could'' happen in the -event of a regression. + * It is assumed that any SRU candidate patch is well-tested before + upload and has a low overall risk of regression, but it's important + to make the effort to think about what ''could'' happen in the + event of a regression. - * This both shows the SRU team that the risks have been considered, -and provides guidance to testers in regression-testing the SRU. + * This both shows the SRU team that the risks have been
[Bug 1804069] Re: haproxy fails on arm64 due to alignment error
** Description changed: + [Impact] + + * An explanation of the effects of the bug on users and + + * justification for backporting the fix to the stable release. + + * In addition, it is helpful, but not required, to include an +explanation of how the upload fixes this bug. + + [Test Case] + + * detailed instructions how to reproduce the bug + + * these should allow someone who is not familiar with the affected +package to reproduce the bug and verify that the updated package fixes +the problem. + + [Regression Potential] + + * discussion of how regressions are most likely to manifest as a result + of this change. + + * It is assumed that any SRU candidate patch is well-tested before +upload and has a low overall risk of regression, but it's important +to make the effort to think about what ''could'' happen in the +event of a regression. + + * This both shows the SRU team that the risks have been considered, +and provides guidance to testers in regression-testing the SRU. + + [Other Info] + + * Anything else you think is useful to include + * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board + * and address these questions in advance + + + [Original Description] + This fault was reported via the haproxy mailing list https://www.mail- archive.com/hapr...@formilux.org/msg31749.html And then patched in the haproxy code here https://github.com/haproxy/haproxy/commit/52dabbc4fad338233c7f0c96f977a43f8f81452a Without this patch haproxy is not functional on aarch64/arm64. Experimental deployments of openstack-ansible on arm64 fail because of this bug, and without a fix applied to the ubuntu bionic packages we cannot proceed further as the openstack CI only installs the upstream ubuntu distribution packages. -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1804069 Title: haproxy fails on arm64 due to alignment error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1804069/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1804069] Re: haproxy fails on arm64 due to alignment error
Tests kicked off: cosmic: https://bileto.ubuntu.com/#/ticket/3610 bionic: https://bileto.ubuntu.com/#/ticket/3611 Those tickets have links to respective ppa builds, if someone wants to test early. -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1804069 Title: haproxy fails on arm64 due to alignment error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1804069/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1804069] Re: haproxy fails on arm64 due to alignment error
Bug reproduced, and fix confirmed. -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1804069 Title: haproxy fails on arm64 due to alignment error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1804069/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1804069] Re: haproxy fails on arm64 due to alignment error
** Changed in: haproxy (Ubuntu Bionic) Assignee: (unassigned) => Andreas Hasenack (ahasenack) ** Changed in: haproxy (Ubuntu Cosmic) Assignee: (unassigned) => Andreas Hasenack (ahasenack) ** Changed in: haproxy (Ubuntu Bionic) Status: Triaged => In Progress ** Changed in: haproxy (Ubuntu Cosmic) Status: Triaged => In Progress -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1804069 Title: haproxy fails on arm64 due to alignment error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1804069/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1804069] Re: haproxy fails on arm64 due to alignment error
This was fixed upstream in 1.8.15: 2018/12/13 : 1.8.15 - MINOR: threads: Make sure threads_sync_pipe is initialized before using it. ... - BUG/MEDIUM: Make sure stksess is properly aligned. <-- Marking main task as fix released, as disco has 1.8.17. But confirmed for bionic and cosmic (assuming it was introduced in the 1.8.x series and not present earlier). ** Also affects: haproxy (Ubuntu Cosmic) Importance: Undecided Status: New ** Changed in: haproxy (Ubuntu) Status: Triaged => Fix Released ** Changed in: haproxy (Ubuntu Cosmic) Status: New => Triaged ** Changed in: haproxy (Ubuntu Bionic) Importance: Undecided => High ** Changed in: haproxy (Ubuntu Cosmic) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1804069 Title: haproxy fails on arm64 due to alignment error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1804069/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
