Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Hi Stefan, Confirming again that your patch from Feb 4th fixed the issue. I've got now positive feedback from my customers and have upgraded all HTTPS-sites back to HTTP/2. Will this patch make it into Debian Stretch? Thanks, Philip
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Hi Stefan, Wow, this is great! I have applied your bug915103-try2.diff patch and it seems to fix the issue. Only did some rudimentary testing so far. I have patched Apache for 2hrs now and started to switch some crucial sites back to HTTP/2. Could not reproduce the problem any more. Very nice! Thank you S much! No worries about late response. It is a great response with a great patch and I totally appreciate. For the ones that are not used to patching Apache on Debian, here's my short HOWTO (it's enough to install apache2-bin package): $ cd /usr/src/apache2-bug915103 $ apt-get source apache2 $ cd apache2-2.4.25 $ patch -p1 < ../bug915103-try2.diff $ apt-get build-dep apache2 $ dpkg-buildpackage -b $ cd ../ $ dpkg -i apache2-bin_2.4.25-3+deb9u6_amd64.deb $ systemctl restart apache2 $ echo apache2-bin hold | dpkg --set-selections Cheers, Philip
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Hi Stefan Do you have any news about this? I had to downgrade the major part of my customers to HTTP/1.1 because of this bug, which is quite a disaster. I would greatly appreciate your help. Am also more than willing to pay you the hours you spend on this. Best regards, Philip
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Hi Stefan >> On 17 Dec 2018, at 22:55, Stefan Fritsch wrote: >> >> Yes, that's the problematic patch, not the fix. >> >> I have some hope that the fix for the issue is this upstream commit: >> https://svn.apache.org/viewvc?view=revision&revision=1843468 >> >> It would be nice if you could apply the attached patch to the debian source >> package, rebuild it, and check if it fixes the issue. Thanks. > > Thanks a lot for that patch. I have applied it to apache2 2.4.25-3+deb9u6, > compiled apache2 using dpkg-buildpackage, and installed apache2-bin package > on production webserver. So far no issues. > But I cannot tell you if it improved anything. The thing is, I couldn't > reproduce the previous issue under desktop Safari right before patching > Apache. I tried hard to reproduce it the same way I was able to reproduce it > on Dec 14th. I switched back the relevant sites to HTTP/2 (Protocols h2 > http/1.1) and tested in Safari checking web inspector console on a site where > previously a bunch of jpg images were not loaded at all. > So, it seems that magically, the problem went away by itself. Could the > original issue be related to any load / buffer issues on long running apache?? > I am sorry that I cannot give you any more detailed feedback. It works fine > with your patch as it did before... I need to disappoint you. I was now able to reproduce the same issue under Apache with applied patch. It was only a Safari browser caching thing why I couldn't reproduce it before. After emptying browser cache the problem occurred again. Switching back to http/1.1 resolves the issue. So it looks like your patch did not change anything here. Looking forward for your next trick! Cheers, Philip
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Hi Stefan > On 17 Dec 2018, at 22:55, Stefan Fritsch wrote: > > Yes, that's the problematic patch, not the fix. > > I have some hope that the fix for the issue is this upstream commit: > https://svn.apache.org/viewvc?view=revision&revision=1843468 > > It would be nice if you could apply the attached patch to the debian source > package, rebuild it, and check if it fixes the issue. Thanks. Thanks a lot for that patch. I have applied it to apache2 2.4.25-3+deb9u6, compiled apache2 using dpkg-buildpackage, and installed apache2-bin package on production webserver. So far no issues. But I cannot tell you if it improved anything. The thing is, I couldn't reproduce the previous issue under desktop Safari right before patching Apache. I tried hard to reproduce it the same way I was able to reproduce it on Dec 14th. I switched back the relevant sites to HTTP/2 (Protocols h2 http/1.1) and tested in Safari checking web inspector console on a site where previously a bunch of jpg images were not loaded at all. So, it seems that magically, the problem went away by itself. Could the original issue be related to any load / buffer issues on long running apache?? I am sorry that I cannot give you any more detailed feedback. It works fine with your patch as it did before... Can you explain why this issue only occurred in Safari? If I check the mod_h2 Github issues and threads referenced in upstream changelog, there is no hint about this being related to Safari: *) mod_http2: adding defensive code for stream EOS handling, in case the request handler missed to signal it the normal way (eos buckets). Addresses github issues https://github.com/icing/mod_h2/issues/164, https://github.com/icing/mod_h2/issues/167 and https://github.com/icing/mod_h2/issues/170. [Stefan Eissing] Cheers, Philip
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
> Could you please shed light on where I can find commit > bee2facd9343beda10677b139cd9b2e49e986f01 for Debian Stretch? > I did not find apache2 sources on https://salsa.debian.org - Where is the > official Debian apache2 source git repo? > If it is not public, please attach the patch. > > We are struggling hard with this bug and will need to downgrade all of our > customers from HTTP/2 to HTTP/1.1 if we don't find a fix very soon. I am fine > compiling apache2 package by myself as long as this fix does not make it into > Stretch. > > Can you confirm that this bug was only introduced in Debian 9.6 point > release? That issue was not popping up before but since then, people started > complaining. OK, in the meantime I found official Debian apache2 git repo: https://salsa.debian.org/apache-team/apache2 But the patch from bee2facd9343beda10677b139cd9b2e49e986f01 (https://salsa.debian.org/apache-team/apache2/commit/bee2facd9343beda10677b139cd9b2e49e986f01) was already applied to latest apache2 package in Debian 9.6 (modules/http2/h2_bucket_beam.c). How come this should fix the problem? Or did you rather mean this patch is the source of these issues. Best, Philip
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
> i'm still wrong: > da1d372d0d58474f2f5a71b9acd301abf9b11bc0 is the commit on the master branch > > On the stretch branch, the commit > is bee2facd9343beda10677b139cd9b2e49e986f01 Hi Cyr Could you please shed light on where I can find commit bee2facd9343beda10677b139cd9b2e49e986f01 for Debian Stretch? I did not find apache2 sources on https://salsa.debian.org - Where is the official Debian apache2 source git repo? If it is not public, please attach the patch. We are struggling hard with this bug and will need to downgrade all of our customers from HTTP/2 to HTTP/1.1 if we don't find a fix very soon. I am fine compiling apache2 package by myself as long as this fix does not make it into Stretch. Can you confirm that this bug was only introduced in Debian 9.6 point release? That issue was not popping up before but since then, people started complaining. Thanks, Philip
Bug#902906: closed by Stefan Fritsch (Bug#902906: fixed in apache2 2.4.25-3+deb9u6)
90172 > apache2_2.4.25-3+deb9u6.debian.tar.xz > 26ff2bc1b0d7dbe5b08d71f23633c4f9decf980fcfd0aa348ecf41cfc709ad7b 1186420 > apache2-bin_2.4.25-3+deb9u6_amd64.deb > c947d3889d33cfbb4b1e7c64f703c979830f4d53061d2966c0925e5e565d608f 162112 > apache2-data_2.4.25-3+deb9u6_all.deb > 4eb1c252b7efbb9f9d3254da546729a564f6eb5aa751662526347a776989b16e 4017542 > apache2-dbg_2.4.25-3+deb9u6_amd64.deb > b23d03dea9bcfa7c8f0f8534d193fa92837444e6d98d974d9858520707b52941 313942 > apache2-dev_2.4.25-3+deb9u6_amd64.deb > e87ecf4173d13aed62efce16521ac5f32ed5316f57ed7161470f5ccaa5b7a62f 3770774 > apache2-doc_2.4.25-3+deb9u6_all.deb > 53c2b3fe58ed0f232574a437f25302c052f798e9a3eec3ac8d7b617fddb65b22 2268 > apache2-ssl-dev_2.4.25-3+deb9u6_amd64.deb > 8901fea6f314719cd975e854c077f342f45d5143fe57082f969906f8667f68b4 155210 > apache2-suexec-custom_2.4.25-3+deb9u6_amd64.deb > 305a64e1a1871ca1e430dc2e164dc34c91581015540e8de71b758d07b848cf90 153732 > apache2-suexec-pristine_2.4.25-3+deb9u6_amd64.deb > 4c557dccd216f4c319a01b0d20e6315bd483999a1bbcca6488bd2e59990b046f 217058 > apache2-utils_2.4.25-3+deb9u6_amd64.deb > 8bde42135512e310cc1de367ae9375bb4e39625f2bb36dd14aff03a85284a18a 10163 > apache2_2.4.25-3+deb9u6_amd64.buildinfo > 42bbfcabaa49fcc458ec20569229adde1a8662aacd69b2e8107cfee69d5f9b59 235974 > apache2_2.4.25-3+deb9u6_amd64.deb > Files: > 0d89b47aef7b19975ae8387cb7d323d3 2986 httpd optional > apache2_2.4.25-3+deb9u6.dsc > 96fe0be15c776db7710d473acb7872b2 790172 httpd optional > apache2_2.4.25-3+deb9u6.debian.tar.xz > c36fee808ccdac5ec0cd2faae758bf14 1186420 httpd optional > apache2-bin_2.4.25-3+deb9u6_amd64.deb > dcfaef6cb1024be84c2f9be07b54fb4d 162112 httpd optional > apache2-data_2.4.25-3+deb9u6_all.deb > f0c4416e5244bab112201761a4f32d55 4017542 debug extra > apache2-dbg_2.4.25-3+deb9u6_amd64.deb > 1a0ae2576a3ba6b9e72b5a1432c38eee 313942 httpd optional > apache2-dev_2.4.25-3+deb9u6_amd64.deb > 7e6df0368dff1ee78c0232d8f9670262 3770774 doc optional > apache2-doc_2.4.25-3+deb9u6_all.deb > 94e03d511df7909bcd92a7a03073149c 2268 httpd optional > apache2-ssl-dev_2.4.25-3+deb9u6_amd64.deb > e159c61f9c7a050844852bc9ca056e77 155210 httpd extra > apache2-suexec-custom_2.4.25-3+deb9u6_amd64.deb > 4597c9c7e7733f8fd26712f57c125dfe 153732 httpd optional > apache2-suexec-pristine_2.4.25-3+deb9u6_amd64.deb > 2d3ac31dd972cf078b5493167e149839 217058 httpd optional > apache2-utils_2.4.25-3+deb9u6_amd64.deb > 8e16239cc29939450aa2af0cd22e2b9a 10163 httpd optional > apache2_2.4.25-3+deb9u6_amd64.buildinfo > 7fed7d6f182385772fbba22e615dcba2 235974 httpd optional > apache2_2.4.25-3+deb9u6_amd64.deb > > -BEGIN PGP SIGNATURE- > > iQIzBAEBCgAdFiEEOpiNza8JqByyYYsxxodfNUHO/eAFAlvd794ACgkQxodfNUHO > /eCtiw/+IHqUZt7sc/+RGQBeBbk8b7c9MSUJHhow+Eh03GIXbHZtY6gRqylH4tBA > EBcuLXpxbdevh8OiQhby9DCaqFmURZ434pd39EDgf2+mAPrwiIw93dkD1DBBSRvK > Z87/TaYRT7lI1CYPQBvyk4dZgKdrmAJfua5WXBCqLZNBknDgbq2dZ9M0OLbCsZSY > fdz96WVxhTopdug4Yu6T6nwmnFebsV90DtTQvdvPJdDumDoMp9docGx80ypkj/zE > fDJchBn2lb2x4m8+M8kcnlm/5+/yPyjMOd0Tlk3XdJxUQX6+/Dod/cqk4ooB+hdy > 7pjgFqBkDBu0fSktMFe2nfedTM4PUqy1BXLb42u3a3/FWaoCNK4HXsN7vbUgQQcN > FagHrjJ1dk/GqWgoYKeE4DOsdStJxZLL7ueSvl8x49DcQnZHYEtem0DXDrRKICOD > bK45JpDFcO8gwaGQFNhcnwBS4tBKdLBbID+Zj4+KI9fLmPBOO7XJIWznYrM8aXh6 > ePdhLKarksw4zUKYdFFVRDlAKLBcxo8hjS7SY82bwnMJ6AKGxwnj6myzhxNOGT7F > iVtFUKrruQ4j0lHQWEjhlPy11kWxcFGbV/4hADzOSyk0t8Ox4aGiHyC/dV45vfyj > TewDaFwqzBMbRGrfZLXY2H7ISQ2MRnPrbIZ7oZDTnpHetTUjekE= > =QKQK > -END PGP SIGNATURE- > > > From: Philip Iezzi > Subject: apache2-bin: mod_proxy_fcgi segfault on ap_fcgi_encoded_env_len if > an environment variable value is null > Date: 3 July 2018 at 11:53:41 CEST > To: Debian Bug Tracking System > > > Package: apache2-bin > Version: 2.4.25-3+deb9u4 > Severity: important > Tags: patch upstream > > Dear Maintainer, > > We got a lot of such segfaults in error.log, provoked by mod_proxy_fcgi: > > [core:notice] [pid 43086:tid 139897736885440] AH00051: child pid 43114 exit > signal Segmentation fault (11) > > As recommended on https://wiki.apache.org/httpd/PHP-FPM, we use the following > PHP-FPM invocation with SetHandler (running mpm_event): > > ``` > > > SetHandler > "proxy:unix:/run/fpm-pool-web999-php72.socket|fcgi://localhost" > > > ``` > > Analyzing coredump: > > ``` > $ gdb /usr/sbin/apache2 /tmp/coredump-apache2-11-33-33-43114-1530368206 > (...) > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > Core was generated by `/usr/sbin/apache2 -k start'. > Program terminated with signal SIGSEGV, Segmentation fault.
Bug#902906: apache2-bin: mod_proxy_fcgi segfault on ap_fcgi_encoded_env_len if an environment variable value is null
Package: apache2-bin
Version: 2.4.25-3+deb9u4
Severity: important
Tags: patch upstream
Dear Maintainer,
We got a lot of such segfaults in error.log, provoked by mod_proxy_fcgi:
[core:notice] [pid 43086:tid 139897736885440] AH00051: child pid 43114 exit
signal Segmentation fault (11)
As recommended on https://wiki.apache.org/httpd/PHP-FPM, we use the following
PHP-FPM invocation with SetHandler (running mpm_event):
```
SetHandler
"proxy:unix:/run/fpm-pool-web999-php72.socket|fcgi://localhost"
```
Analyzing coredump:
```
$ gdb /usr/sbin/apache2 /tmp/coredump-apache2-11-33-33-43114-1530368206
(...)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 strlen () at ../sysdeps/x86_64/strlen.S:106
106 ../sysdeps/x86_64/strlen.S: No such file or directory.
[Current thread is 1 (Thread 0x7f3c54ff9700 (LWP 43741))]
(gdb) bt
#0 strlen () at ../sysdeps/x86_64/strlen.S:106
#1 0x55b25cef8e57 in ap_fcgi_encoded_env_len (env=,
maxlen=maxlen@entry=16384, starting_elem=starting_elem@entry=0x7f3c54ff8ae0) at
util_fcgi.c:156
#2 0x7f3c74f4871d in send_environment (request_id=1,
temp_pool=0x7f3c49e1c028, r=0x7f3c49e196c0, conn=0x7f3c72bbb0a0) at
mod_proxy_fcgi.c:321
#3 fcgi_do_request (p=, origin=0x0, uri=,
url=, server_portstr=0x7f3c54ff8b40 "", conf=0x7f3c7ae24490,
conn=0x7f3c72bbb0a0, r=0x7f3c49e196c0) at mod_proxy_fcgi.c:848
#4 proxy_fcgi_handler (r=0x7f3c49e196c0, worker=,
conf=, url=, proxyname=,
proxyport=) at mod_proxy_fcgi.c:968
#5 0x7f3c751562bc in proxy_run_scheme_handler (r=r@entry=0x7f3c49e196c0,
worker=0x7f3c7ad7abf0, conf=conf@entry=0x7f3c7ae2bdd0,
url=0x7f3c49e13b08 "fcgi://localhost/var/www/shared/error_docs/400.php",
proxyhost=proxyhost@entry=0x0, proxyport=proxyport@entry=0) at mod_proxy.c:2880
#6 0x7f3c75157231 in proxy_handler (r=0x7f3c49e196c0) at mod_proxy.c:1230
#7 0x55b25cef1c40 in ap_run_handler (r=r@entry=0x7f3c49e196c0) at
config.c:170
#8 0x55b25cef21d6 in ap_invoke_handler (r=r@entry=0x7f3c49e196c0) at
config.c:434
#9 0x55b25cf090bc in ap_internal_redirect (new_uri=,
r=) at http_request.c:765
#10 0x55b25cedc5b5 in ap_read_request (conn=conn@entry=0x7f3c49e28348) at
protocol.c:1285
#11 0x55b25cf0604d in ap_process_http_async_connection (c=0x7f3c49e28348)
at http_core.c:146
#12 ap_process_http_connection (c=0x7f3c49e28348) at http_core.c:248
#13 0x55b25cefba70 in ap_run_process_connection (c=c@entry=0x7f3c49e28348)
at connection.c:42
#14 0x7f3c755786e8 in process_socket (my_thread_num=,
my_child_num=, cs=0x7f3c49e282b8, sock=,
p=0x7f3c49e28028, thd=) at event.c:1099
#15 worker_thread (thd=, dummy=) at event.c:2003
#16 0x7f3c7a3a4494 in start_thread (arg=0x7f3c54ff9700) at
pthread_create.c:333
#17 0x7f3c7a0e6acf in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
```
The issue was reported upstream, Apache Bug 60275, including a patch:
https://bz.apache.org/bugzilla/show_bug.cgi?id=60275
The patch made it into upstream Apache 2.4.26 (see
https://www.apache.org/dist/httpd/CHANGES_2.4):
*) mod_proxy_fcgi, mod_fcgid: Fix crashes in ap_fcgi_encoded_env_len() when
modules add empty environment variables to the request. PR 60275.
[]
I have applied the provided patch on apache2_2.4.25-3+deb9u4_amd64 and
installed apache2-bin. This resolved the issue 100% (Apache was previously
crashing on avg 15 times/h over months, since installing patched apache2-bin no
more single segfault!).
apache2-2.4.25-pr60275.patch:
```diff
diff -ur apache2-2.4.25/server/util_fcgi.c
apache2-2.4.25-patched/server/util_fcgi.c
--- apache2-2.4.25/server/util_fcgi.c 2015-07-20 12:28:13.0 +0200
+++ apache2-2.4.25-patched/server/util_fcgi.c 2018-07-01 09:16:08.122664970
+0200
@@ -153,7 +153,11 @@
envlen += keylen;
-vallen = strlen(elts[i].val);
+ if (!elts[i].val) {
+ vallen = 0;
+ } else {
+ vallen = strlen(elts[i].val);
+ }
if (vallen >> 7 == 0) {
envlen += 1;
@@ -226,7 +230,11 @@
buflen -= 4;
}
-vallen = strlen(elts[i].val);
+if (!elts[i].val) {
+vallen = 0;
+} else {
+ vallen = strlen(elts[i].val);
+ }
if (vallen >> 7 == 0) {
if (buflen < 1) {
@@ -262,8 +270,10 @@
rv = APR_ENOSPC; /* overflow */
break;
}
-memcpy(itr, elts[i].val, vallen);
-itr += vallen;
+ if (elts[i].val) {
+ memcpy(itr, elts[i].val, vallen);
+ itr += vallen;
+ }
if (buflen == vallen) {
(*starting_elem)++;
```
Please try to get this into the next Debian Stretch point release. It seems to
be critical as this bug renders mod_proxy_fcgi unusable for
Bug#863520: cyrus-imapd version 2.5.10-3 Fatal error with SSL
Dear maintainer, I would greatly appreciate if you could push this fix into current Debian Stretch. The problem still persists in Cyrus-imapd 2.5.10-3 and above patch from upstream fixes it. After having upgraded a mailserver to Debian Stretch we had a massive amount of negative customer feedback complaining about dropped connections. The patched and recompiled packages are now running for more than 2 weeks on two rather busy mail servers (datenpark.ch / onlime.ch) and all trouble has gone away, cyrus-imapd works stable again. Thanks Vladislav for your great support! Here's a short howto for people who never built a Deb package before: $ apt-get source cyrus-imapd $ wget https://github.com/cyrusimap/cyrus-imapd/commit/a1c917df8de04e108228f38f0010498bec3d81e8.patch -O cyrus-imapd-issue1872.patch $ cd cyrus-imapd-2.5.10/ $ patch -p1 < ../cyrus-imapd-issue1872.patch $ apt-get build-dep cyrus-imapd $ dpkg-buildpackage -b $ cd ../ # install at least the following and put those packages on hold: $ dpkg -i cyrus-common_2.5.10-3_amd64.deb cyrus-imapd_2.5.10-3_amd64.deb cyrus-pop3d_2.5.10-3_amd64.deb libcyrus-imap-perl_2.5.10-3_amd64.deb $ echo cyrus-common hold | dpkg --set-selections $ echo cyrus-imapd hold | dpkg --set-selections $ echo cyrus-pop3d hold | dpkg --set-selections $ echo libcyrus-imap-perl hold | dpkg --set-selections # check package state $ dpkg --get-selections | grep cyrus | grep -v deinstall This fixes the issue and the "lib/cyrusdb_twoskip.c" fatal errors no longer pop up in mail.log Best regards, Philip
