Bug report for Apache httpd-2 [2015/11/29]
+---+ | Bugzilla Bug ID | | +-+ | | Status: UNC=Unconfirmed NEW=New ASS=Assigned| | | OPN=ReopenedVER=Verified(Skipped Closed/Resolved) | | | +-+ | | | Severity: BLK=Blocker CRI=Critical REG=Regression MAJ=Major | | | | MIN=Minor NOR=NormalENH=Enhancement TRV=Trivial | | | | +-+ | | | | Date Posted | | | | | +--+ | | | | | Description | | | | | | | | 7483|Ass|Enh|2002-03-26|Add FileAction directive to assign a cgi interpret| | 8713|Inf|Min|2002-05-01|No Errorlog on PROPFIND/Depth:Infinity| | 8867|Opn|Cri|2002-05-07|exports.c generation fails when using a symlink to| |10747|New|Maj|2002-07-12|ftp SIZE command and 'smart' ftp servers results i| |11294|New|Enh|2002-07-30|desired vhost_alias option| |11580|Opn|Enh|2002-08-09|generate Content-Location headers | |12033|Opn|Nor|2002-08-26|Graceful restart immediately result in [warn] long| |12680|New|Enh|2002-09-16|Digest authentication with integrity protection | |13599|Inf|Nor|2002-10-14|autoindex formating broken for multibyte sequences| |13661|Ass|Enh|2002-10-15|Apache cannot not handle dynamic IP reallocation | |14104|Opn|Enh|2002-10-30|not documented: must restart server to load new CR| |14496|New|Enh|2002-11-13|Cannot upgrade any version on Windows. Must uninst| |14922|Inf|Enh|2002-11-28| is currently hardcoded to 'apache2' | |15719|Inf|Nor|2002-12-30|WebDAV MOVE to destination URI which is content-ne| |16761|Inf|Nor|2003-02-04|CustomLog with pipe spawns process during config | |16802|New|Enh|2003-02-05|Additional AllowOverride directive "Restrict" | |16811|Ass|Maj|2003-02-05|mod_autoindex always return webpages in UTF-8.| |17107|New|Min|2003-02-16|Windows should not install printenv | |17114|New|Enh|2003-02-17|Please add strip and install-strip targets to Make| |17244|Ass|Nor|2003-02-20|./configure --help gives false information regardi| |17497|Opn|Nor|2003-02-27|mod_mime_magic generates incorrect response header| |18325|New|Enh|2003-03-25|PAM support for suEXEC| |18334|Inf|Cri|2003-03-25|Server crashes when authenticating users against L| |18497|New|Min|2003-03-30|configure --help gives wrong default for sysconfdi| |19043|New|Min|2003-04-15|Interesting interaction between cern_meta module a| |19670|New|Enh|2003-05-05|content type header supplied upon PUT is thrown aw| |20036|Ass|Nor|2003-05-19|Trailing Dots stripped from PATH_INFO environment | |21253|New|Nor|2003-07-01|Mime magic doesn't continue if type is specifed fo| |21260|New|Nor|2003-07-02|CacheMaxExpire directive not enforced ! | |21533|Ass|Cri|2003-07-11|Multiple levels of htacces files can cause mod_aut| |22138|Inf|Cri|2003-08-05|Webdav is not preccessing special chars right.| |22237|New|Enh|2003-08-08|option to disable ServerSignature on index pages | |22484|Opn|Maj|2003-08-16|semaphore problem takes httpd down| |22686|Opn|Nor|2003-08-25|ab: apr_poll: The timeout specified has expired (7| |22898|Opn|Nor|2003-09-02|nph scripts with two HTTP header | |23167|Inf|Cri|2003-09-14|--enable-layout never goes to apr apr-util| |23181|New|Nor|2003-09-15|Status 304 (Not modified) and chunking leads to an| |23238|New|Cri|2003-09-18|non-async-signal-safe operations from signal handl| |23330|New|Enh|2003-09-22|Enhance ApacheMonitor to view and control Tomcat s| |23911|Opn|Cri|2003-10-18|CGI processes left defunct/zombie under 2.0.54| |24031|New|Enh|2003-10-23|Passphrase protected private key in SSLProxyMachin| |24095|Opn|Cri|2003-10-24|ERROR "Parent: child process exited with status 32| |24437|Opn|Nor|2003-11-05|mod_auth_ldap doubly-escapes backslash (\) charact| |24890|Opn|Nor|2003-11-21|Apache config parser should not be local aware ( g| |25014|New|Enh|2003-11-26|A flexible interface for mod_log_config | |25201|New|Enh|2003-12-04|Provide Cache Purge operation | |25240|Inf|Enh|2003-12-05|SSL Library Error: 336105671 logged as information| |25435|New|Enh|2003-12-11|sethandler and directoryindex not playing nice| |25469|Opn|Enh|2003-12-12|create AuthRoot for defining paths to auth files | |25484|Ass|Nor|2003-12-12|Non-service Apache cannot be stopped in WinXP | |25543|Inf|Nor|2003-12-15|mod_proxy_ajp overwrites existing response headers| |25667|New|Nor|
RE: No H2 Window updates!
> -Original Message- > From: Bert Huijben [mailto:b...@qqmail.nl] > Sent: zaterdag 28 november 2015 14:32 > To: stefan.eiss...@greenbytes.de; dev@httpd.apache.org > Subject: RE: No H2 Window updates! > In case of Subversion's real usage, I want to commit potentially hundreds of > MByte, so a connection level window of more than a few bytes would be > very > welcome With HTTP/1 we send out the data as fast as we can and the TCP > windowing handles this from the httpd side... Now the http/2 level > windowing > should handle this. > > > Mod_dav and mod_dav_svn are usually fast readers; just limited by trivial > disk io or xml processing in the common operations, so I don't expect real > problems there. For my commits against the svn-master.apache.org repository my latency/ping time is +- 142 ms. With the current simple algorithm and a maximum window of 64 Kbyte/connection, I can send a theoretical maximum of 64 Kbyte/142 ms per connection to that server... which would be about 450 Kbyte/s. (=1/0.142*65536/1024) That is < 1/30th of the bandwidth I have at my disposal (+- 150 Mbit). But currently I don't even get anywhere near that as my window continues to shrink because some data is missed in accounting even after my simple patch. For httpd we have to think carefully which algorithm we want to implement here. Preferably the algorithm should be better than TCP could, as we know the specifics of the http algorithm better than plain TCP could for HTTP/1,1. TCP does send a lot of window updates though... Almost every packet does. Solving the really hard problems, like this one, is the reason I didn't think the nghttp2 library would really be the solution for serf. It is nice for such a library to provide a head start on the protocol level, but there is no way a standard library can really solve this scheduling problem in a generic way. (If it could do it, we had used that solution for TCP... and never had to resort to designing a HTTP/2 in the first place). See https://en.wikipedia.org/wiki/TCP_window_scale_option for some introduction on how TCP was extended to work above the limit that currently applies to H2. Dynamic window sizing/scaling will have to be implemented at some point, at both the stream and the connection level... This will involve timing, measuring, etc. etc. Things nghttp2 can't do for us right now. And if I'm right this might take continuous optimizing for years to come. When I connect to sites like google I immediately receive a large connection level window, to allow me to post huge blobs without a delay. (Haven't tested their stream level windows behavior yet) In serf I do something similar... and then apply a bit of throttling at the stream level. I would guess some clients have already implemented some of this, so we might be able to learn from their implementations... Clients will see much more incoming data than servers of course :). Bert
RE: No H2 Window updates!
> -Original Message-
> From: Bert Huijben [mailto:b...@qqmail.nl]
> Sent: zaterdag 28 november 2015 14:09
> To: stefan.eiss...@greenbytes.de; dev@httpd.apache.org
> Subject: RE: No H2 Window updates!
>
>
>
> > -Original Message-
> > From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de]
> > Sent: zaterdag 28 november 2015 13:01
> > To: dev@httpd.apache.org
> > Subject: Re: No H2 Window updates!
> >
> > I am not really here, but...
> >
> > the window updates are sent out via update_window(), line 1001,
> > h2_session.c. If you do not see any window updates with a client, it may
> > be that the server app you called does not read its input. I have
> > several test cases with uploads and they work with nghttp and curl.
>
> In my case it doesn't...
>
> ---
> if (h2_stream_set_has_open_input(session->streams)) {
> /* Check that any pending window updates are sent. */
> status = h2_mplx_in_update_windows(session->mplx,
> update_window, session);
> if (APR_STATUS_IS_EAGAIN(status)) {
> status = APR_SUCCESS;
> }
> else if (status == APR_SUCCESS) {
> /* need to flush window updates onto the connection
asap
> */
> h2_conn_io_flush(&session->io);
> }
> }
>
>
>
> Looks like that ' h2_stream_set_has_open_input()' always returns false for
> me, with those tiny requests of only 1 data frame.
>
> The streams are marked as closed the moment a data frame with EOS is
> received... that is: before the frame is processed.
>
> In this testcase all data frames have EOS set, so this value would only be
> true for me if some other outstanding request did not receive its data
yet.
> (Which doesn't happen yet as Subversion still assumes http/1 like
> processing)
If I just replace the
if (h2_stream_set_has_open_input(session->streams))
with if(TRUE) {
then I do receive a few window updates... Of a few bytes each though...
I would have expected a few huge returns (Kbyte+) every few requests that
send data.
So over the course of hundreds of requests my connection level window
shrinks to almost zero... at some points I receive a few updates of a few
hundred bytes... I even see one of a single byte.
Data on closed streams must be accounted towards the connection window, so
conditionalizing the whole processing on 'are there any readable streams' is
a clear bug. Skipping all data in the last frame is another one.
Perhaps the code can skip some of the processing in this case, but (as a
client) I would like to see my outgoing window at the connection updated
sooner.
In case of Subversion's real usage, I want to commit potentially hundreds of
MByte, so a connection level window of more than a few bytes would be very
welcome With HTTP/1 we send out the data as fast as we can and the TCP
windowing handles this from the httpd side... Now the http/2 level windowing
should handle this.
Mod_dav and mod_dav_svn are usually fast readers; just limited by trivial
disk io or xml processing in the common operations, so I don't expect real
problems there.
Bert
Re: No H2 Window updates!
Steffen in gmane.comp.apache.devel (Sat, 28 Nov 2015 10:44:44 +0100): >I saw issues on boxes with the default of number of concurrent >streams. > >Lowering the default 100 value of the directive H2MaxSessionStreams >solved it. I know that, because it is in this topic: https://www.apachelounge.com/viewtopic.php?p=32186#32186 What is worrying me is that I do not have problems with mod_http2 1.0.5, but mod_http2 1.0.8 in a default installation causes trouble with Drupal7. It would be great if I could test it with mod_http2.so 1.0.8 from a Apachelounge build. -- Jan
RE: No H2 Window updates!
> -Original Message-
> From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de]
> Sent: zaterdag 28 november 2015 13:01
> To: dev@httpd.apache.org
> Subject: Re: No H2 Window updates!
>
> I am not really here, but...
>
> the window updates are sent out via update_window(), line 1001,
> h2_session.c. If you do not see any window updates with a client, it may
> be that the server app you called does not read its input. I have
> several test cases with uploads and they work with nghttp and curl.
In my case it doesn't...
---
if (h2_stream_set_has_open_input(session->streams)) {
/* Check that any pending window updates are sent. */
status = h2_mplx_in_update_windows(session->mplx,
update_window, session);
if (APR_STATUS_IS_EAGAIN(status)) {
status = APR_SUCCESS;
}
else if (status == APR_SUCCESS) {
/* need to flush window updates onto the connection asap
*/
h2_conn_io_flush(&session->io);
}
}
Looks like that ' h2_stream_set_has_open_input()' always returns false for
me, with those tiny requests of only 1 data frame.
The streams are marked as closed the moment a data frame with EOS is
received... that is: before the frame is processed.
In this testcase all data frames have EOS set, so this value would only be
true for me if some other outstanding request did not receive its data yet.
(Which doesn't happen yet as Subversion still assumes http/1 like
processing)
Bert
RE: No H2 Window updates!
> -Original Message-
> From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de]
> Sent: zaterdag 28 november 2015 13:01
> To: dev@httpd.apache.org
> Subject: Re: No H2 Window updates!
>
> I am not really here, but...
>
> the window updates are sent out via update_window(), line 1001,
> h2_session.c. If you do not see any window updates with a client, it may
> be that the server app you called does not read its input. I have
> several test cases with uploads and they work with nghttp and curl.
>
> Basics of mod_http2 flow control:
> - the auto udates of nghttp2 are disabled because nghttp2 would
> continously update the window for the client, letting the client sent
> more and more - until we run out of memory.
> - instead, input reads from workers against the h2_mplx io are recorded
> and lead to regular window update being sent out. So clients can only
> send more when the data is actually consumed by someone.
>From h2_session.c around line 1640.
===
switch (status) {
case APR_SUCCESS: /* successful read, reset our idle
timers */
have_read = 1;
wait_micros = 0;
break;
case APR_EAGAIN: /* non-blocking read, nothing
there */
break;
default:
if (APR_STATUS_IS_ETIMEDOUT(status)
|| APR_STATUS_IS_ECONNABORTED(status)
|| APR_STATUS_IS_ECONNRESET(status)
|| APR_STATUS_IS_EOF(status)
|| APR_STATUS_IS_EBADF(status)) {
/* common status for a client that has left */
ap_log_cerror( APLOG_MARK, APLOG_DEBUG, status,
session->c,
"h2_session(%ld): terminating",
session->id);
/* Stolen from mod_reqtimeout to speed up lingering
when
* a read timeout happened.
*/
apr_table_setn(session->c->notes,
"short-lingering-close", "1");
}
else {
/* uncommon status, log on INFO so that we see this
*/
ap_log_cerror( APLOG_MARK, APLOG_INFO, status,
session->c,
APLOGNO(02950)
"h2_session(%ld): error reading,
terminating",
session->id);
}
h2_session_abort(session, status, 0);
goto end_process;
}
===
I'm not familiar enough with differences in bucket handling between serf and
httpd to really make the call, but as the serf buckets were designed by the
same group:
I'm guessing that there might be successful reads with different status
values than just APR_SUCCESS.
In serf I would expect to see an immediate APR_EOF when there is only a
single frame to be read (or perhaps a few intermediate APR_EAGAINS and then
a EOF), which may both imply successful reading 0 or more bytes, followed by
their status code.
Bert
RE: No H2 Window updates!
> -Original Message- > From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de] > Sent: zaterdag 28 november 2015 13:01 > To: dev@httpd.apache.org > Subject: Re: No H2 Window updates! > > I am not really here, but... > > the window updates are sent out via update_window(), line 1001, > h2_session.c. If you do not see any window updates with a client, it may > be that the server app you called does not read its input. I have > several test cases with uploads and they work with nghttp and curl. I do see window updates against other servers. And I'm pretty sure Subversion reads its input... things are committed correctly and the propfinds return their result. In the testcase at hand I'm sending very small requests (+- 100 bytes max), so the per stream window updates are not really used... They get nowhere near the 64K limit and the first data frame has EOS set. Perhaps this also skips sending window updates on the connection level. I do remember testing larger uploads as a single stream... So perhaps it is a combination that makes them not appear for me. Bert
Re: No H2 Window updates!
I am not really here, but... the window updates are sent out via update_window(), line 1001, h2_session.c. If you do not see any window updates with a client, it may be that the server app you called does not read its input. I have several test cases with uploads and they work with nghttp and curl. Basics of mod_http2 flow control: - the auto udates of nghttp2 are disabled because nghttp2 would continously update the window for the client, letting the client sent more and more - until we run out of memory. - instead, input reads from workers against the h2_mplx io are recorded and lead to regular window update being sent out. So clients can only send more when the data is actually consumed by someone. What is not well tested, I think, is the timeout/cleanup behaviour on this. So, as long as the connection is alive, individual stream could appear to hang infinitely *if* no one ever reads the data. Maybe you see something like this? //Stefan Am 27.11.2015 um 14:23 schrieb Bert Huijben: -Original Message- From: Bert Huijben [mailto:b...@qqmail.nl] Sent: vrijdag 27 november 2015 13:56 To: b...@qqmail.nl Subject: Hi, I finally took the time to diagnose that segfault I had, and I think it points to a serious bug in httpd. To summarize this: I don’t receive window updates. In this specific test we set a very huge amount of small requests (bodies of 95 and 113 bytes), until we get out of the 65535 (or 65536) bytes of window space I get from httpd at the connection level. (Each stream doesn’t get near its limit. I can try if I can receive window updates there… but currently I can’t reproduce ever receiving a window update) Originally this caused a segfault in my code, but I fixed that one. But now I’m just stuck waiting to receive a window update from httpd… My last testing was against 2.4.x (to get the 2.4.18 goodness) And I think the combination of: === h2_session.c around line 707 === /* We need to handle window updates ourself, otherwise we * get flooded by nghttp2. */ nghttp2_option_set_no_auto_window_update(options, 1); And not a single call to nghttp2_submit_window_update() to find, explains the situation. I haven't tried what happens when I disable this auto_window call... but sending window updates is really required by the H2 specs. And I totally understand that this wasn't high priority... I worked around not sending updates in my implementation until yesterday :-) Bert
RE: No H2 Window updates!
I saw issues on boxes with the default of number of concurrent
streams.
Lowering the default 100 value of the directive H2MaxSessionStreams
solved it.
On Saturday 28/11/2015 at 10:29, Bert Huijben wrote:
-Original Message-
From: Jan Ehrhardt [mailto:php...@ehrhardt.nl]
Sent: vrijdag 27 november 2015 22:35
To: dev@httpd.apache.org
Subject: Re: No H2 Window updates!
Bert Huijben in gmane.comp.apache.devel (Fri, 27 Nov 2015 20:04:14
+0100):
Well. it is not a regression, so can it be a show stopper? ?.
But I would like to see this fixed.
Curious: are you still testing this on Windows? If so, I guess you
compiled your own httpd. I tried to do the same a couple of days ago,
but
ran into problems with Drupal7: the admin menu sometimes showed and
sometimes did not show at all. I could not lay my finger on what went
wrong.
Because I did not have the problems with Apachelounge's 2.4.18-dev at
https://www.apachelounge.com/viewtopic.php?t=6842 I checked out an
earlier
revision of the alpha branch:
| svn co -r 1715218
http://svn.apache.org/repos/asf/httpd/httpd/branches/2.4-http2-alpha
That revision compiled into a httpd with no problems. I am waiting now
for
Stefan Eissing to finish his work on mod_http2.
BTW: did you switch to nghttp2 1.5.0 already?
Hi Jan,
No I didn't switch to nghttp2 1.5.0 yet. Thanks for reminding me to
check if
there is a new version :)
The code I wrote for Serf code doesn't use nghttp2... After reading
the
specs I didn't think building on nghttp2 would win me a lot of time,
and
having multiple separate implementations of the same specification/RFC
in
the open source world would be a good thing. (Almost every recent H2
project
I see builds on top of nghttp2)
It is entirely possible that you hit the same problem as I did. (I'm
actually very surprised that I didn't hit this problem much earlier
on.
There is just one test in the Subversion testsuite that sends more
than 64
Kbyte of request bodies over a single connection... I'll fix that)
But back to that problem... There is an easy workaround, which I used
on the
other side until two days ago: just making the H2 default window big
enough
that you never get near it.
Just configure 'H2WindowSize' to be something like 1 GB and you
probably
never have to think about window updates. (The max allowed value is 2
GB -1)
This windowing allows the server to throttle incoming DATA from the
other
side, so servers like httpd really want to tune this dynamically. Note
that
headers and new requests are not counted to this limit, so on the
server
side it is really just request bodies... (Technically data on already
closed
streams should be counted too... but that is an implementation
detail).
I remember reading that Stefan Eissing is away for some time ('Re:
NOTICE:
Intent to T&R 2.4.18' thread), so perhaps I should spend some time
looking
at this myself.
Disabling that line that explicitly disables Window updates from the
nghttp
library, could be an easy fix... but it might require some
compensating
actions, like lowering the number of supported concurrent streams if
that
comment is still up to date. Allowing up to 100 concurrent streams per
connection could be a bit high, although this really depends on what
these
connections are used for. I don't know how to test against that 'gets
flooded' problem though, as that isn't measurable by itself.
And yes I build my own binaries for Subversion and all its
dependencies...
All my scripting that I use for that is in
https://sharpsvn.open.collab.net/svn/sharpsvn/trunk/imports (username
'guest', no password). The default build doesn't build httpd, but if
you use
a Subversion dev build (copy dev-default.build to a directory one
level
above imports) it builds httpd. My scripts should work for VS2005 upto
2015
and require nant, and some python and perl versions. Everything else
is
built from the scripts.
These same scripts drive the Subversion and Serf win32 buildbots that
I
maintain on ci.apache.org... and they also deliver the SharpSvn and
'Slik
Subversion Client' binaries.
(I currently explicitly don't deliver httpd itself or anything that
depends
on that though)
Thanks [ / Groeten ;-)],
Bert
--
Jan
RE: No H2 Window updates!
> -Original Message-
> From: Jan Ehrhardt [mailto:php...@ehrhardt.nl]
> Sent: vrijdag 27 november 2015 22:35
> To: dev@httpd.apache.org
> Subject: Re: No H2 Window updates!
>
> Bert Huijben in gmane.comp.apache.devel (Fri, 27 Nov 2015 20:04:14 +0100):
> >Well. it is not a regression, so can it be a show stopper? ?.
> >But I would like to see this fixed.
>
> Curious: are you still testing this on Windows? If so, I guess you
> compiled your own httpd. I tried to do the same a couple of days ago, but
> ran into problems with Drupal7: the admin menu sometimes showed and
> sometimes did not show at all. I could not lay my finger on what went
> wrong.
>
> Because I did not have the problems with Apachelounge's 2.4.18-dev at
> https://www.apachelounge.com/viewtopic.php?t=6842 I checked out an
> earlier
> revision of the alpha branch:
> | svn co -r 1715218
> http://svn.apache.org/repos/asf/httpd/httpd/branches/2.4-http2-alpha
>
> That revision compiled into a httpd with no problems. I am waiting now for
> Stefan Eissing to finish his work on mod_http2.
>
> BTW: did you switch to nghttp2 1.5.0 already?
Hi Jan,
No I didn't switch to nghttp2 1.5.0 yet. Thanks for reminding me to check if
there is a new version :)
The code I wrote for Serf code doesn't use nghttp2... After reading the
specs I didn't think building on nghttp2 would win me a lot of time, and
having multiple separate implementations of the same specification/RFC in
the open source world would be a good thing. (Almost every recent H2 project
I see builds on top of nghttp2)
It is entirely possible that you hit the same problem as I did. (I'm
actually very surprised that I didn't hit this problem much earlier on.
There is just one test in the Subversion testsuite that sends more than 64
Kbyte of request bodies over a single connection... I'll fix that)
But back to that problem... There is an easy workaround, which I used on the
other side until two days ago: just making the H2 default window big enough
that you never get near it.
Just configure 'H2WindowSize' to be something like 1 GB and you probably
never have to think about window updates. (The max allowed value is 2 GB -1)
This windowing allows the server to throttle incoming DATA from the other
side, so servers like httpd really want to tune this dynamically. Note that
headers and new requests are not counted to this limit, so on the server
side it is really just request bodies... (Technically data on already closed
streams should be counted too... but that is an implementation detail).
I remember reading that Stefan Eissing is away for some time ('Re: NOTICE:
Intent to T&R 2.4.18' thread), so perhaps I should spend some time looking
at this myself.
Disabling that line that explicitly disables Window updates from the nghttp
library, could be an easy fix... but it might require some compensating
actions, like lowering the number of supported concurrent streams if that
comment is still up to date. Allowing up to 100 concurrent streams per
connection could be a bit high, although this really depends on what these
connections are used for. I don't know how to test against that 'gets
flooded' problem though, as that isn't measurable by itself.
And yes I build my own binaries for Subversion and all its dependencies...
All my scripting that I use for that is in
https://sharpsvn.open.collab.net/svn/sharpsvn/trunk/imports (username
'guest', no password). The default build doesn't build httpd, but if you use
a Subversion dev build (copy dev-default.build to a directory one level
above imports) it builds httpd. My scripts should work for VS2005 upto 2015
and require nant, and some python and perl versions. Everything else is
built from the scripts.
These same scripts drive the Subversion and Serf win32 buildbots that I
maintain on ci.apache.org... and they also deliver the SharpSvn and 'Slik
Subversion Client' binaries.
(I currently explicitly don't deliver httpd itself or anything that depends
on that though)
Thanks [ / Groeten ;-)],
Bert
> --
> Jan
