Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Looks like this is fixed in 2.4.25-3+deb9u7. Safari is not dropping http2 requests any more. > This update also contains bug fixes that were scheduled for inclusion in the > next stable point release. This includes a fix for a regression caused by a > security fix in version 2.4.25-3+deb9u6. https://www.debian.org/security/2019/dsa-4422
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Stefan Fritsch schrieb am 16.02.19 um 11:21: > On Tuesday, 12 February 2019 16:45:34 CET Philip Iezzi wrote: >> Hi Stefan, >> >> Confirming again that your patch from Feb 4th fixed the issue. I've got now >> positive feedback from my customers and have upgraded all HTTPS-sites back >> to HTTP/2. Will this patch make it into Debian Stretch? >> >> Thanks, >> Philip > > yes, I intend to have it included in the 9.9 stable point release. > Unfortunately, it was just a bit too late for the 9.8 which is due today. Thank you, but honestly i would have expected Debian to fix a regression like this immediately, not with a point release. That's the kind of stability i like and use Debian for... Andreas
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
On Tuesday, 12 February 2019 16:45:34 CET Philip Iezzi wrote: > Hi Stefan, > > Confirming again that your patch from Feb 4th fixed the issue. I've got now > positive feedback from my customers and have upgraded all HTTPS-sites back > to HTTP/2. Will this patch make it into Debian Stretch? > > Thanks, > Philip yes, I intend to have it included in the 9.9 stable point release. Unfortunately, it was just a bit too late for the 9.8 which is due today.
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Hi Stefan, Confirming again that your patch from Feb 4th fixed the issue. I've got now positive feedback from my customers and have upgraded all HTTPS-sites back to HTTP/2. Will this patch make it into Debian Stretch? Thanks, Philip
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Hi Stefan, got the same issue with Safari clients when updating from 2.4.25-3+deb9u5 to 2.4.25-3+deb9u6. Looking forward for a security regression update :) -- /* Mit freundlichem Gruß / With kind regards, Patrick Matthäi GNU/Linux Debian Developer Blog: http://www.linux-dev.org/ E-Mail: pmatth...@debian.org patr...@linux-dev.org */
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Hi Stefan, Wow, this is great! I have applied your bug915103-try2.diff patch and it seems to fix the issue. Only did some rudimentary testing so far. I have patched Apache for 2hrs now and started to switch some crucial sites back to HTTP/2. Could not reproduce the problem any more. Very nice! Thank you S much! No worries about late response. It is a great response with a great patch and I totally appreciate. For the ones that are not used to patching Apache on Debian, here's my short HOWTO (it's enough to install apache2-bin package): $ cd /usr/src/apache2-bug915103 $ apt-get source apache2 $ cd apache2-2.4.25 $ patch -p1 < ../bug915103-try2.diff $ apt-get build-dep apache2 $ dpkg-buildpackage -b $ cd ../ $ dpkg -i apache2-bin_2.4.25-3+deb9u6_amd64.deb $ systemctl restart apache2 $ echo apache2-bin hold | dpkg --set-selections Cheers, Philip
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Hi Philip,
sorry for the late respone, I have been quite busy with other things.
I could find no indication that any other upstream release has the same bug.
Therefore I hope that adding more fixes from upstream versions up to the
version from where I took the security fixes (2.4.34 and 2.4.35) should fix the
issue. That's how I picked the first patch I have sent you. There is one other
commit that may fit. A new patch is applied (leave out the first patch I sent).
If that does not work we need to find a more targeted approach. You could try
increasing the http2 log level and see if there are any log messages that
appear only when safari gives the error message. You could try a quite high
log level like
loglevel http2:trace1
or even trace2.
Cheers,
Stefan
diff --git a/debian/patches/http2-r1832566.diff b/debian/patches/http2-r1832566.diff
new file mode 100644
index 00..7ce7335100
--- /dev/null
+++ b/debian/patches/http2-r1832566.diff
@@ -0,0 +1,43 @@
+--- apache2.orig/modules/http2/h2_conn.c
apache2/modules/http2/h2_conn.c
+@@ -240,7 +240,19 @@ apr_status_t h2_conn_run(struct h2_ctx *
+ && mpm_state != AP_MPMQ_STOPPING);
+
+ if (c->cs) {
+-c->cs->state = CONN_STATE_LINGER;
++switch (session->state) {
++case H2_SESSION_ST_INIT:
++case H2_SESSION_ST_IDLE:
++case H2_SESSION_ST_BUSY:
++case H2_SESSION_ST_WAIT:
++c->cs->state = CONN_STATE_WRITE_COMPLETION;
++break;
++case H2_SESSION_ST_CLEANUP:
++case H2_SESSION_ST_DONE:
++default:
++c->cs->state = CONN_STATE_LINGER;
++break;
++}
+ }
+
+ return APR_SUCCESS;
+--- apache2.orig/modules/http2/h2_version.h
apache2/modules/http2/h2_version.h
+@@ -27,7 +27,7 @@
+ * @macro
+ * Version number of the http2 module as c string
+ */
+-#define MOD_HTTP2_VERSION "1.10.16"
++#define MOD_HTTP2_VERSION "1.10.20"
+
+ /**
+ * @macro
+@@ -35,7 +35,7 @@
+ * release. This is a 24 bit number with 8 bits for major number, 8 bits
+ * for minor and 8 bits for patch. Version 1.2.3 becomes 0x010203.
+ */
+-#define MOD_HTTP2_VERSION_NUM 0x010a10
++#define MOD_HTTP2_VERSION_NUM 0x010a14
+
+
+ #endif /* mod_h2_h2_version_h */
diff --git a/debian/patches/series b/debian/patches/series
index 014d958573..21ff3c5da4 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -30,3 +30,4 @@ mod_http2_mem_usage_32bit.diff
fcgi_crash.diff
CVE-2018-1333-mod_http2_DoS.diff
CVE-2018-11763-mod_http2_DoS-SETTINGS.diff
+http2-r1832566.diff
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
We are also able to reproduce this issue. We did several days of testing trying to pinpoint the issue and also determined it was for sure http/2 with Apache 2.4.25 on Debian Stretch only, and only a problem in Safari. We were unable to reproduce on other versions of Debian/Apache, with http/1.1, or with any other browser. We were able to duplicate as far back as Safari 9 on Mac and iOS 9 Safari (we didn't have older versions than that available to test). For now, we have disabled http/1.1 (better to load at all than to sometimes load faster), but this does hurt Google page speed/Lighthouse scores which in turn hurts search rank so we hope a solution can be found. I posted some additional details of our testing on https://serverfault.com/questions/942863/safari-fails-to-load-some-resources-over-http-2-with-apache. We were unable to figure out a way to disable http/2 just for Safari and came to the conclusion it wasn't possible.
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Hi Stefan Do you have any news about this? I had to downgrade the major part of my customers to HTTP/1.1 because of this bug, which is quite a disaster. I would greatly appreciate your help. Am also more than willing to pay you the hours you spend on this. Best regards, Philip
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Hi Stefan >> On 17 Dec 2018, at 22:55, Stefan Fritsch wrote: >> >> Yes, that's the problematic patch, not the fix. >> >> I have some hope that the fix for the issue is this upstream commit: >> https://svn.apache.org/viewvc?view=revision=1843468 >> >> It would be nice if you could apply the attached patch to the debian source >> package, rebuild it, and check if it fixes the issue. Thanks. > > Thanks a lot for that patch. I have applied it to apache2 2.4.25-3+deb9u6, > compiled apache2 using dpkg-buildpackage, and installed apache2-bin package > on production webserver. So far no issues. > But I cannot tell you if it improved anything. The thing is, I couldn't > reproduce the previous issue under desktop Safari right before patching > Apache. I tried hard to reproduce it the same way I was able to reproduce it > on Dec 14th. I switched back the relevant sites to HTTP/2 (Protocols h2 > http/1.1) and tested in Safari checking web inspector console on a site where > previously a bunch of jpg images were not loaded at all. > So, it seems that magically, the problem went away by itself. Could the > original issue be related to any load / buffer issues on long running apache?? > I am sorry that I cannot give you any more detailed feedback. It works fine > with your patch as it did before... I need to disappoint you. I was now able to reproduce the same issue under Apache with applied patch. It was only a Safari browser caching thing why I couldn't reproduce it before. After emptying browser cache the problem occurred again. Switching back to http/1.1 resolves the issue. So it looks like your patch did not change anything here. Looking forward for your next trick! Cheers, Philip
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Hi Stefan > On 17 Dec 2018, at 22:55, Stefan Fritsch wrote: > > Yes, that's the problematic patch, not the fix. > > I have some hope that the fix for the issue is this upstream commit: > https://svn.apache.org/viewvc?view=revision=1843468 > > It would be nice if you could apply the attached patch to the debian source > package, rebuild it, and check if it fixes the issue. Thanks. Thanks a lot for that patch. I have applied it to apache2 2.4.25-3+deb9u6, compiled apache2 using dpkg-buildpackage, and installed apache2-bin package on production webserver. So far no issues. But I cannot tell you if it improved anything. The thing is, I couldn't reproduce the previous issue under desktop Safari right before patching Apache. I tried hard to reproduce it the same way I was able to reproduce it on Dec 14th. I switched back the relevant sites to HTTP/2 (Protocols h2 http/1.1) and tested in Safari checking web inspector console on a site where previously a bunch of jpg images were not loaded at all. So, it seems that magically, the problem went away by itself. Could the original issue be related to any load / buffer issues on long running apache?? I am sorry that I cannot give you any more detailed feedback. It works fine with your patch as it did before... Can you explain why this issue only occurred in Safari? If I check the mod_h2 Github issues and threads referenced in upstream changelog, there is no hint about this being related to Safari: *) mod_http2: adding defensive code for stream EOS handling, in case the request handler missed to signal it the normal way (eos buckets). Addresses github issues https://github.com/icing/mod_h2/issues/164, https://github.com/icing/mod_h2/issues/167 and https://github.com/icing/mod_h2/issues/170. [Stefan Eissing] Cheers, Philip
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Hi Philip,
On Friday, 14 December 2018 22:49:13 CET Philip Iezzi wrote:
> But the patch from bee2facd9343beda10677b139cd9b2e49e986f01
> (https://salsa.debian.org/apache-team/apache2/commit/bee2facd9343beda10677b
> 139cd9b2e49e986f01) was already applied to latest apache2 package in Debian
> 9.6 (modules/http2/h2_bucket_beam.c). How come this should fix the problem?
> Or did you rather mean this patch is the source of these issues.
Yes, that's the problematic patch, not the fix.
I have some hope that the fix for the issue is this upstream commit:
https://svn.apache.org/viewvc?view=revision=1843468
It would be nice if you could apply the attached patch to the debian source
package, rebuild it, and check if it fixes the issue. Thanks.
Cheers,
Stefan
diff --git a/debian/patches/http-EOS-handling.diff b/debian/patches/http-EOS-handling.diff
new file mode 100644
index 00..501ab5a7b6
--- /dev/null
+++ b/debian/patches/http-EOS-handling.diff
@@ -0,0 +1,26 @@
+# https://svn.apache.org/viewvc?view=revision=1843468
+# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=915103
+--- apache2.orig/modules/http2/h2_session.c
apache2/modules/http2/h2_session.c
+@@ -1094,6 +1094,10 @@ static ssize_t stream_data_cb(nghttp2_se
+ case APR_SUCCESS:
+ break;
+
++case APR_EOF:
++eos = 1;
++break;
++
+ case APR_ECONNRESET:
+ case APR_ECONNABORTED:
+ return NGHTTP2_ERR_CALLBACK_FAILURE;
+--- apache2.orig/modules/http2/h2_stream.c
apache2/modules/http2/h2_stream.c
+@@ -915,7 +915,7 @@ apr_status_t h2_stream_out_prepare(h2_st
+ (long)*plen, *peos);
+ }
+ else {
+-status = APR_EAGAIN;
++status = (stream->output && h2_beam_is_closed(stream->output))? APR_EOF : APR_EAGAIN;
+ ap_log_cerror(APLOG_MARK, APLOG_TRACE1, 0, c,
+ H2_STRM_MSG(stream, "prepare, no data"));
+ }
diff --git a/debian/patches/series b/debian/patches/series
index 014d958573..93b77b7f35 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -30,3 +30,4 @@ mod_http2_mem_usage_32bit.diff
fcgi_crash.diff
CVE-2018-1333-mod_http2_DoS.diff
CVE-2018-11763-mod_http2_DoS-SETTINGS.diff
+http-EOS-handling.diff
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
> Could you please shed light on where I can find commit > bee2facd9343beda10677b139cd9b2e49e986f01 for Debian Stretch? > I did not find apache2 sources on https://salsa.debian.org - Where is the > official Debian apache2 source git repo? > If it is not public, please attach the patch. > > We are struggling hard with this bug and will need to downgrade all of our > customers from HTTP/2 to HTTP/1.1 if we don't find a fix very soon. I am fine > compiling apache2 package by myself as long as this fix does not make it into > Stretch. > > Can you confirm that this bug was only introduced in Debian 9.6 point > release? That issue was not popping up before but since then, people started > complaining. OK, in the meantime I found official Debian apache2 git repo: https://salsa.debian.org/apache-team/apache2 But the patch from bee2facd9343beda10677b139cd9b2e49e986f01 (https://salsa.debian.org/apache-team/apache2/commit/bee2facd9343beda10677b139cd9b2e49e986f01) was already applied to latest apache2 package in Debian 9.6 (modules/http2/h2_bucket_beam.c). How come this should fix the problem? Or did you rather mean this patch is the source of these issues. Best, Philip
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
> i'm still wrong: > da1d372d0d58474f2f5a71b9acd301abf9b11bc0 is the commit on the master branch > > On the stretch branch, the commit > is bee2facd9343beda10677b139cd9b2e49e986f01 Hi Cyr Could you please shed light on where I can find commit bee2facd9343beda10677b139cd9b2e49e986f01 for Debian Stretch? I did not find apache2 sources on https://salsa.debian.org - Where is the official Debian apache2 source git repo? If it is not public, please attach the patch. We are struggling hard with this bug and will need to downgrade all of our customers from HTTP/2 to HTTP/1.1 if we don't find a fix very soon. I am fine compiling apache2 package by myself as long as this fix does not make it into Stretch. Can you confirm that this bug was only introduced in Debian 9.6 point release? That issue was not popping up before but since then, people started complaining. Thanks, Philip
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
On Friday, 30 November 2018 15:54:07 CET Andreas Ziegler wrote: > Package: apache2 > Version: 2.4.25-3+deb9u6 > > When i load a picture using safari from an apache webserver with HTTP/2 > enabled and repeat that multiple times in a row (F5), > at least each 3rd request fails with > "Failed to load resource: The network connection was lost." > This happens regardless of the device (Mac, iPad) and regardless of the > ISP or the provider or hardware of the webserver in the datacenter. > The user experience when surfing on an affected webserver is really bad > with Safari. http://svn.apache.org/viewvc?view=revision=1843468 sounds like it could be a fix for this issue
Bug#915103: Apache2 HTTP/2 connection problems with Safari clients
Package: apache2 Version: 2.4.25-3+deb9u6 When i load a picture using safari from an apache webserver with HTTP/2 enabled and repeat that multiple times in a row (F5), at least each 3rd request fails with "Failed to load resource: The network connection was lost." This happens regardless of the device (Mac, iPad) and regardless of the ISP or the provider or hardware of the webserver in the datacenter. The user experience when surfing on an affected webserver is really bad with Safari. This does not happen after downgrading to version 2.4.25-3+deb9u5 The new HTTP/2 Anti-DoS patches should be optimized to work with Safari clients. We are using Debian GNU/Linux 9, Kernel 4.9.0-8-amd64.
