Re: [VOTE] Switch read/write repository from Subversion to Git

[Roy T. Fielding]

> On May 4, 2023, at 1:34 AM, Ruediger Pluem  wrote:
> 
> [X]: Move the read/write repository from Subversion to Git and leverage the 
> features of Github (for now Actions and PR).

I trust subversion more as a vcs, but that is outweighed by the
convenience of Github's PR and issue management. Their integration is
far too complete to consider retaining bugzilla as well, except as a
source for legacy issues.

Roy

Re: [VOTE] [VOTE] Release httpd-2.4.56-rc1 as httpd-2.4.56

[Roy T. Fielding]

On Mar 10, 2023, at 8:56 AM, Yann Ylavic  wrote:

> On Fri, Mar 10, 2023 at 4:34 PM Eric Covener  wrote:
>> 
>> Saw another report on users@
>> 
>> Any thoughts on something like this to just allow spaces?
>> http://people.apache.org/~covener/patches/rewrite-lax.diff
> 
> What about:
> 
> Index: modules/mappers/mod_rewrite.c
> ===
> --- modules/mappers/mod_rewrite.c(revision 1908254)
> +++ modules/mappers/mod_rewrite.c(working copy)
> @@ -4814,7 +4814,8 @@ static int hook_uri2file(request_rec *r)
> apr_size_t flen;
> int to_proxyreq;
> 
> -if (r->args && *(ap_scan_vchar_obstext(r->args))) {
> +if (rulestatus == ACTION_NOESCAPE
> +&& r->args && *(ap_scan_vchar_obstext(r->args))) {
> /*
>  * We have a raw control character or a ' ' in r->args.
>  * Correct encoding was missed.
> ?
> 
> Regards;
> Yann.

Allowing a space to be sent within the proxied request target is not an option,
regardless of how the user has configured the server. The CVE fix was just to
prevent an invalid target sent from us.

Why don't we fix the source of the spaces? The place where the variable is 
decoding
the matched string being inserted. I find that bit surprising, since it doesn't 
behave
like a proper regex.

Likewise, the rewrite mapper should always pct-encode or reject embedded spaces
long before we get to the proxy (or internal redirect) request.

Roy

fixed autolinks from cve process

[Roy T. Fielding]

FYI, I made some minor changes to the post-release description
on httpd-site and within 2.4.x/CHANGES for CVE-2023-2569

The form we use for editing the CVE json info has a
feature for autolinking anything that looks like a
URL reference. Unfortunately, it's buggy and cannot be
turned off. When the form was updated yesterday, the
autolinks came back. This causes every example URL to
be repeated in the text as a link, which of course
makes them an invalid example.

I have fixed the JSON on httpd-site, rebuilding

  https://httpd.apache.org/security/vulnerabilities_24.html

and committed a fix in CHANGES.  I have not sent an update for
the official CVE database, since I have no idea what that would break.

No worries, but some users might complain about the weird examples.

Cheers, and thanks for the release,

Roy

Re: [VOTE] [VOTE] Release httpd-2.4.56-rc1 as httpd-2.4.56

[Roy T. Fielding]

> On Mar 5, 2023, at 1:31 PM, Eric Covener  wrote:
> 
> Hi all,
> 
> Please find below the proposed release tarball and signatures:
> 
> https://dist.apache.org/repos/dist/dev/httpd/
> 
> I would like to call a VOTE over the next few days to release
> this candidate tarball httpd-2.4.56-rc1 as 2.4.56:
> [X] +1: It's not just good, it's good enough!

Verified sigs, compiled and installed locally (Macbook Pro M1, macos Ventura 
13.2.1),
and tested around the relevant changes. Everything looks good.

+1 for release.

Roy T. Fielding

Re: svn commit: r1906618 - /httpd/httpd/branches/2.2.x/docs/manual/rewrite/intro.xml

[Roy T. Fielding]

I am pretty sure that this change makes the example incorrect.

That regex is not supposed to match "/foo/" (even if one might
want to do that for directory examples, this particular example
is not about directories). /foo is the whole path.

If a directory example is desired, perhaps a new example would
be better?

Roy

> On Jan 11, 2023, at 12:13 PM, rbo...@apache.org wrote:
> 
> Author: rbowen
> Date: Wed Jan 11 20:13:31 2023
> New Revision: 1906618
> 
> URL: http://svn.apache.org/viewvc?rev=1906618=rev
> Log:
> Fixes example to actually work.
> 
> Modified:
>httpd/httpd/branches/2.2.x/docs/manual/rewrite/intro.xml
> 
> Modified: httpd/httpd/branches/2.2.x/docs/manual/rewrite/intro.xml
> URL: 
> http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/docs/manual/rewrite/intro.xml?rev=1906618=1906617=1906618=diff
> ==
> --- httpd/httpd/branches/2.2.x/docs/manual/rewrite/intro.xml (original)
> +++ httpd/httpd/branches/2.2.x/docs/manual/rewrite/intro.xml Wed Jan 11 
> 20:13:31 2023
> @@ -205,7 +205,7 @@ like the A
> 2. A web-path to a resource
> 
> 
> -RewriteRule ^/foo$ /bar
> +RewriteRule ^/foo/?$ /bar
> 
> If DocumentRoot is set
> to /usr/local/apache2/htdocs, then this directive would
> 
> 

Re: svn commit: r1904269 - in /httpd/httpd/trunk: changes-entries/ docs/manual/mod/ modules/http2/ test/modules/http2/

[Roy T. Fielding]

> On Oct 19, 2022, at 2:28 AM, Stefan Eissing via dev  
> wrote:
>> Am 18.10.2022 um 21:20 schrieb Roy T. Fielding :
>> 
>>> On Oct 6, 2022, at 2:17 AM, Stefan Eissing via dev  
>>> wrote:
>>> 
>>>> Am 05.10.2022 um 19:34 schrieb Stefan Eissing via dev 
>>>> :
>>>> 
>>>> 
>>>> 
>>>>> Am 05.10.2022 um 18:48 schrieb Eric Covener :
>>>>> 
>>>>> On Wed, Oct 5, 2022 at 12:44 PM Roy T. Fielding  wrote:
>>>>>> 
>>>>>>> On Sep 26, 2022, at 5:29 AM, ic...@apache.org wrote:
>>>>>>> 
>>>>>>> Author: icing
>>>>>>> Date: Mon Sep 26 12:29:47 2022
>>>>>>> New Revision: 1904269
>>>>>>> 
>>>>>>> URL: http://svn.apache.org/viewvc?rev=1904269=rev
>>>>>>> Log:
>>>>>>> *) mod_http2: new directive "H2HeaderStrictness" to control the 
>>>>>>> compliance
>>>>>>> level of header checks as defined in the HTTP/2 RFCs. Default is 7540.
>>>>>>> 9113 activates the checks for forbidden leading/trailing whitespace in
>>>>>>> field values (available from nghttp2 v1.50.0 on).
>>>>>> 
>>>>>> I am not seeing why that should be optional. It adds overhead, but it 
>>>>>> reduces
>>>>>> variability (for HPACK) and might prevent some downstream 
>>>>>> vulnerabilities, IIRC.
>>>>>> Maybe an internal switch for testing with default on?
>>> 
>>> To add some more detail:
>>> 
>>> - rfc9113 ch 8.2.1 states: "A field value MUST NOT start or end with an 
>>> ASCII whitespace character"
>>> - nghttp2 v1.49.0 implemented that check, non-optional. Things broke.
>>> - nghttp2 v1.50.0 added to its API so that the application can control the 
>>> behaviour on request of Daniel Stenberg
>>> - I added "H2HeaderStrictness" (in trunk only for now) to steer that.
>>> 
>>> This is not a normalization issue, it's a hard "MUST NOT" e.g. rejection of 
>>> the request/response carrying such a field. While I agree that there are 
>>> many advantages in having fields more strict, the way to get there is not 
>>> clear.
>>> 
>>> I am totally in favour of avoiding "H2HeaderStrictness", but just enforcing 
>>> rfc9113 here will not do. What would our response be for people whose 
>>> legacy clients/fronted applications will no longer work?
>> 
>> Well, normally, I don't have a problem with just breaking them.
>> They are broken. Someone can fix them.
>> 
>> It isn't a normalization issue. Whitespace that magically appeared when h2
>> changed the framing of header field values immediately became a security
>> vulnerability for all downstream recipients of h2/h3 messages that use
>> generic HTTP (semantic) field values expecting that stuff to have
>> been excluded during parsing.
>> 
>> IOW, it's a security hole and our code either fixes it or gets a CVE.
>> We MUST NOT forward the malformed data in fields. That is not an option.
>> 
>> OTOH, how we deal with a request containing malformed data in fields
>> (after making it well-formed) is a SHOULD send 400, not a MUST.
>> If we want to be super nice to the folks shipping bad code (or running pen
>> testers) and have an option to strip the naughty bits out while forwarding
>> the message, that's fine with me as an optional directive. But that won't
>> help them interop with the rest of the Internet.
> 
> Speaking about our implementation, I just added tests to trunk. Configuring 
> 'Header add name "$value"' and making http/1.1 requests, curl sees the 
> returned headers as:
> 
> configured, returned
> ['123 ', '123 '],  # trailing space stays
> ['123\t', '123\t'],# trailing tab stays
> [' 123', '123'],   # leading space is stripped
> ['  123', '123'],  # leading spaces are stripped
> ['\t123', '123'],  # leading tab is stripped
> 
> Test case 'test_h1_007_02'. 
> 
> Linking nghttp2 v1.49.0 and newer, without further config, will RST_STREAM 
> cases 1 and 2 when using HTTP/2. While succeeding with HTTP/1.1. I would find 
> that highly confusing.
> 
> I understand that in the definition of Core HTTP, leading/trailing whitespace 
> MUST NOT carry any semantics and SHOULD be avoided. If Apache httpd, in its 
> version-independent field handling, should decide to universa

Re: svn commit: r1904269 - in /httpd/httpd/trunk: changes-entries/ docs/manual/mod/ modules/http2/ test/modules/http2/

[Roy T. Fielding]

> On Oct 19, 2022, at 4:50 AM, Ruediger Pluem  wrote:
> 
> 
> 
> On 10/19/22 11:28 AM, Stefan Eissing via dev wrote:
>> 
>> 
>>> Am 18.10.2022 um 21:20 schrieb Roy T. Fielding :
>>> 
>>>> On Oct 6, 2022, at 2:17 AM, Stefan Eissing via dev  
>>>> wrote:
>>>> 
>>>>> Am 05.10.2022 um 19:34 schrieb Stefan Eissing via dev 
>>>>> :
>>>>> 
>>>>> 
>>>>> 
>>>>>> Am 05.10.2022 um 18:48 schrieb Eric Covener :
>>>>>> 
>>>>>> On Wed, Oct 5, 2022 at 12:44 PM Roy T. Fielding  
>>>>>> wrote:
>>>>>>> 
>>>>>>>> On Sep 26, 2022, at 5:29 AM, ic...@apache.org wrote:
>>>>>>>> 
>>>>>>>> Author: icing
>>>>>>>> Date: Mon Sep 26 12:29:47 2022
>>>>>>>> New Revision: 1904269
>>>>>>>> 
>>>>>>>> URL: http://svn.apache.org/viewvc?rev=1904269=rev
>>>>>>>> Log:
>>>>>>>> *) mod_http2: new directive "H2HeaderStrictness" to control the 
>>>>>>>> compliance
>>>>>>>> level of header checks as defined in the HTTP/2 RFCs. Default is 7540.
>>>>>>>> 9113 activates the checks for forbidden leading/trailing whitespace in
>>>>>>>> field values (available from nghttp2 v1.50.0 on).
>>>>>>> 
>>>>>>> I am not seeing why that should be optional. It adds overhead, but it 
>>>>>>> reduces
>>>>>>> variability (for HPACK) and might prevent some downstream 
>>>>>>> vulnerabilities, IIRC.
>>>>>>> Maybe an internal switch for testing with default on?
>>>> 
>>>> To add some more detail:
>>>> 
>>>> - rfc9113 ch 8.2.1 states: "A field value MUST NOT start or end with an 
>>>> ASCII whitespace character"
>>>> - nghttp2 v1.49.0 implemented that check, non-optional. Things broke.
>>>> - nghttp2 v1.50.0 added to its API so that the application can control the 
>>>> behaviour on request of Daniel Stenberg
>>>> - I added "H2HeaderStrictness" (in trunk only for now) to steer that.
>>>> 
>>>> This is not a normalization issue, it's a hard "MUST NOT" e.g. rejection 
>>>> of the request/response carrying such a field. While I agree that there 
>>>> are many advantages in having fields more strict, the way to get there is 
>>>> not clear.
>>>> 
>>>> I am totally in favour of avoiding "H2HeaderStrictness", but just 
>>>> enforcing rfc9113 here will not do. What would our response be for people 
>>>> whose legacy clients/fronted applications will no longer work?
>>> 
>>> Well, normally, I don't have a problem with just breaking them.
>>> They are broken. Someone can fix them.
>>> 
>>> It isn't a normalization issue. Whitespace that magically appeared when h2
>>> changed the framing of header field values immediately became a security
>>> vulnerability for all downstream recipients of h2/h3 messages that use
>>> generic HTTP (semantic) field values expecting that stuff to have
>>> been excluded during parsing.
>>> 
>>> IOW, it's a security hole and our code either fixes it or gets a CVE.
>>> We MUST NOT forward the malformed data in fields. That is not an option.
>>> 
>>> OTOH, how we deal with a request containing malformed data in fields
>>> (after making it well-formed) is a SHOULD send 400, not a MUST.
>>> If we want to be super nice to the folks shipping bad code (or running pen
>>> testers) and have an option to strip the naughty bits out while forwarding
>>> the message, that's fine with me as an optional directive. But that won't
>>> help them interop with the rest of the Internet.
>> 
>> Speaking about our implementation, I just added tests to trunk. Configuring 
>> 'Header add name "$value"' and making http/1.1 requests, curl sees the 
>> returned headers as:
>> 
>> configured, returned
>> ['123 ', '123 '],  # trailing space stays
>> ['123\t', '123\t'],# trailing tab stays
>> [' 123', '123'],   # leading space is stripped
>> ['  123', '123'],  # leading spaces are stripped
>> ['\t123', '123'],  # leading tab is stripped
>> 
>> Test case 'test_h1_007_0

Re: svn commit: r1904269 - in /httpd/httpd/trunk: changes-entries/ docs/manual/mod/ modules/http2/ test/modules/http2/

[Roy T. Fielding]

> On Oct 6, 2022, at 2:17 AM, Stefan Eissing via dev  
> wrote:
> 
>> Am 05.10.2022 um 19:34 schrieb Stefan Eissing via dev :
>> 
>> 
>> 
>>> Am 05.10.2022 um 18:48 schrieb Eric Covener :
>>> 
>>> On Wed, Oct 5, 2022 at 12:44 PM Roy T. Fielding  wrote:
>>>> 
>>>>> On Sep 26, 2022, at 5:29 AM, ic...@apache.org wrote:
>>>>> 
>>>>> Author: icing
>>>>> Date: Mon Sep 26 12:29:47 2022
>>>>> New Revision: 1904269
>>>>> 
>>>>> URL: http://svn.apache.org/viewvc?rev=1904269=rev
>>>>> Log:
>>>>> *) mod_http2: new directive "H2HeaderStrictness" to control the compliance
>>>>>  level of header checks as defined in the HTTP/2 RFCs. Default is 7540.
>>>>>  9113 activates the checks for forbidden leading/trailing whitespace in
>>>>>  field values (available from nghttp2 v1.50.0 on).
>>>> 
>>>> I am not seeing why that should be optional. It adds overhead, but it 
>>>> reduces
>>>> variability (for HPACK) and might prevent some downstream vulnerabilities, 
>>>> IIRC.
>>>> Maybe an internal switch for testing with default on?
> 
> To add some more detail:
> 
> - rfc9113 ch 8.2.1 states: "A field value MUST NOT start or end with an ASCII 
> whitespace character"
> - nghttp2 v1.49.0 implemented that check, non-optional. Things broke.
> - nghttp2 v1.50.0 added to its API so that the application can control the 
> behaviour on request of Daniel Stenberg
> - I added "H2HeaderStrictness" (in trunk only for now) to steer that.
> 
> This is not a normalization issue, it's a hard "MUST NOT" e.g. rejection of 
> the request/response carrying such a field. While I agree that there are many 
> advantages in having fields more strict, the way to get there is not clear.
> 
> I am totally in favour of avoiding "H2HeaderStrictness", but just enforcing 
> rfc9113 here will not do. What would our response be for people whose legacy 
> clients/fronted applications will no longer work?

Well, normally, I don't have a problem with just breaking them.
They are broken. Someone can fix them.

It isn't a normalization issue. Whitespace that magically appeared when h2
changed the framing of header field values immediately became a security
vulnerability for all downstream recipients of h2/h3 messages that use
generic HTTP (semantic) field values expecting that stuff to have
been excluded during parsing.

IOW, it's a security hole and our code either fixes it or gets a CVE.
We MUST NOT forward the malformed data in fields. That is not an option.

OTOH, how we deal with a request containing malformed data in fields
(after making it well-formed) is a SHOULD send 400, not a MUST.
If we want to be super nice to the folks shipping bad code (or running pen
testers) and have an option to strip the naughty bits out while forwarding
the message, that's fine with me as an optional directive. But that won't
help them interop with the rest of the Internet.

Cheers,

Roy

Re: svn commit: r1904269 - in /httpd/httpd/trunk: changes-entries/ docs/manual/mod/ modules/http2/ test/modules/http2/

[Roy T. Fielding]

> On Sep 26, 2022, at 5:29 AM, ic...@apache.org wrote:
> 
> Author: icing
> Date: Mon Sep 26 12:29:47 2022
> New Revision: 1904269
> 
> URL: http://svn.apache.org/viewvc?rev=1904269=rev
> Log:
>  *) mod_http2: new directive "H2HeaderStrictness" to control the compliance
> level of header checks as defined in the HTTP/2 RFCs. Default is 7540.
> 9113 activates the checks for forbidden leading/trailing whitespace in
> field values (available from nghttp2 v1.50.0 on).

I am not seeing why that should be optional. It adds overhead, but it reduces
variability (for HPACK) and might prevent some downstream vulnerabilities, IIRC.
Maybe an internal switch for testing with default on?

Roy

Re: backports

[Roy T. Fielding]

> On Mar 4, 2022, at 6:17 AM, Eric Covener  wrote:
> 
> On Fri, Mar 4, 2022 at 9:05 AM Jim Jagielski  wrote:
>> 
>> A question: Would it be easier for all this if we moved to being Github 
>> canon?
> 
> I think it is much more straightforward.  The original work, reviews
> and travis results are all in the same place and nothing is copied
> around.
> I have had the same thought a few times this week. But I was hesitant
> to reopen that thread/discussion because I'm pessimistic we can get
> anywhere on it.

I think we are far beyond that point where staying with svn/bugzilla is actively
hurting the project for little or no benefit.

I'd +1 a switch just to get real issue management and PRs.

Roy

Re: svn commit: r1897872 - in /httpd/httpd/trunk: changes-entries/http2_request_scheme.txt modules/http2/h2_stream.c test/modules/http2/test_003_get.py

[Roy T. Fielding]

> On Feb 9, 2022, at 1:28 AM, Stefan Eissing  wrote:
>> Am 09.02.2022 um 10:15 schrieb Ruediger Pluem :
>> On 2/8/22 7:10 PM, Roy T. Fielding wrote:
>>> As noted in
>>> 
>>>  https://github.com/icing/mod_h2/issues/230#issuecomment-1032905432
>>> 
>>> This doesn't look right to me. I think what you want is to verify that 
>>> https is
>>> in a secured connection. This should have no effect on other schemes, and
>>> certainly not require all schemes to be http or https.
>>> 
>>> Literally, the scheme is a naming system, not a protocol. "http" and "https"
>>> and "foo" schemes can be resolved by any protocol that performs requests
>>> on an absolute URI, including HTTP/2. "https" only requires the connection
>>> to be secured end-to-end.
>> 
>> With respect to our HTTP/1 handling r1895921 
>> http://svn.apache.org/viewvc?view=revision=1895921 added
>> a check that the scheme for non forward proxied requests either needs to be 
>> http or https, but we don't check
>> for a matching with the actual connection whether this is secured or not.
> 
> Thanks for pointing that out, Ruediger. I think I need to change mod_http2 to
> populate the uri.scheme when a :scheme header was sent in the request. Then
> we have the check in one place.
> 
> The question with matching "http" and "https" concerns:
> A. do we select the correct server_rec matching the scheme?
> B. do we want to deny access to https: resources on a non-secured connection?
> 
> I'll add a test for A. My opinion on B is that we should.
> 
> Kind Regards,
> Stefan

The problem with B is that the TLS parts may have already been removed
by a trusted gateway or TLS offload device. I don't know how we recognize
that in our own server config, if at all. Basically, we need the config to state
that the service URL is https even though the message is just http, and
we need to be sure that the above check can be overridden by such a config.

And don't forget that proxies can also receive ftp, ftps, doi, and urn as
schemes, depending entirely on how the modules are mapped and what
kinds of clients are being serviced.

It's important to keep in mind that IETF specs only define the protocol as
it crosses the Internet. Our server also has to support network configs for
inside a colo, non-TCP networks, localhost, and symmetric cyphers, etc.
Hence, some of the TLS-only requirements have no meaning to us outside
the default config, and shouldn't be enforced by the protocol module.

Roy

Re: svn commit: r1897872 - in /httpd/httpd/trunk: changes-entries/http2_request_scheme.txt modules/http2/h2_stream.c test/modules/http2/test_003_get.py

[Roy T. Fielding]

As noted in

   https://github.com/icing/mod_h2/issues/230#issuecomment-1032905432

This doesn't look right to me. I think what you want is to verify that https is
in a secured connection. This should have no effect on other schemes, and
certainly not require all schemes to be http or https.

Literally, the scheme is a naming system, not a protocol. "http" and "https"
and "foo" schemes can be resolved by any protocol that performs requests
on an absolute URI, including HTTP/2. "https" only requires the connection
to be secured end-to-end.

Roy



> On Feb 8, 2022, at 5:38 AM, ic...@apache.org wrote:
> 
> Author: icing
> Date: Tue Feb  8 13:38:49 2022
> New Revision: 1897872
> 
> URL: http://svn.apache.org/viewvc?rev=1897872=rev
> Log:
>  *) mod_http2: when a h2 request carries a ':scheme' pseudoheader,
>it gives a 400 response if the scheme does not match the
>connection. Fixes .
> 
> 
> Added:
>httpd/httpd/trunk/changes-entries/http2_request_scheme.txt
> Modified:
>httpd/httpd/trunk/modules/http2/h2_stream.c
>httpd/httpd/trunk/test/modules/http2/test_003_get.py
> 
> Added: httpd/httpd/trunk/changes-entries/http2_request_scheme.txt
> URL: 
> http://svn.apache.org/viewvc/httpd/httpd/trunk/changes-entries/http2_request_scheme.txt?rev=1897872=auto
> ==
> --- httpd/httpd/trunk/changes-entries/http2_request_scheme.txt (added)
> +++ httpd/httpd/trunk/changes-entries/http2_request_scheme.txt Tue Feb  8 
> 13:38:49 2022
> @@ -0,0 +1,3 @@
> +  *) mod_http2: when a h2 request carries a ':scheme' pseudoheader,
> +it gives a 400 response if the scheme does not match the
> +connection. Fixes .
> 
> Modified: httpd/httpd/trunk/modules/http2/h2_stream.c
> URL: 
> http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/http2/h2_stream.c?rev=1897872=1897871=1897872=diff
> ==
> --- httpd/httpd/trunk/modules/http2/h2_stream.c (original)
> +++ httpd/httpd/trunk/modules/http2/h2_stream.c Tue Feb  8 13:38:49 2022
> @@ -23,6 +23,7 @@
> #include 
> #include 
> #include 
> +#include 
> 
> #include 
> 
> @@ -773,12 +774,10 @@ apr_status_t h2_stream_end_headers(h2_st
> status = h2_request_end_headers(stream->rtmp, stream->pool, eos, 
> raw_bytes);
> if (APR_SUCCESS == status) {
> set_policy_for(stream, stream->rtmp);
> -stream->request = stream->rtmp;
> -stream->rtmp = NULL;
> -
> +
> ctx.maxlen = stream->session->s->limit_req_fieldsize;
> ctx.failed_key = NULL;
> -apr_table_do(table_check_val_len, , stream->request->headers, 
> NULL);
> +apr_table_do(table_check_val_len, , stream->rtmp->headers, NULL);
> if (ctx.failed_key) {
> if (!h2_stream_is_ready(stream)) {
> ap_log_cerror(APLOG_MARK, APLOG_INFO, 0, stream->session->c1,
> @@ -790,6 +789,15 @@ apr_status_t h2_stream_end_headers(h2_st
> /* keep on returning APR_SUCCESS, so that we send a HTTP response 
> and
>  * do not RST the stream. */
> }
> +if (stream->rtmp->scheme && strcasecmp(stream->rtmp->scheme,
> +ap_ssl_conn_is_ssl(stream->session->c1)? "https" : "http")) {
> +ap_log_cerror(APLOG_MARK, APLOG_INFO, 0, stream->session->c1,
> +  H2_STRM_LOG(APLOGNO(), stream,"Request :scheme 
> '%s' and "
> +  "connection do not match."), 
> stream->rtmp->scheme);
> +set_error_response(stream, HTTP_BAD_REQUEST);
> +}
> +stream->request = stream->rtmp;
> +stream->rtmp = NULL;
> }
> return status;
> }
> 
> Modified: httpd/httpd/trunk/test/modules/http2/test_003_get.py
> URL: 
> http://svn.apache.org/viewvc/httpd/httpd/trunk/test/modules/http2/test_003_get.py?rev=1897872=1897871=1897872=diff
> ==
> --- httpd/httpd/trunk/test/modules/http2/test_003_get.py (original)
> +++ httpd/httpd/trunk/test/modules/http2/test_003_get.py Tue Feb  8 13:38:49 
> 2022
> @@ -211,3 +211,11 @@ content-type: text/html
> assert 1024 == len(r.response["body"])
> assert "content-length" in h
> assert clen == h["content-length"]
> +
> +# use an invalid scheme
> +def test_h2_003_51(self, env):
> +url = env.mkurl("https", "cgi", "/")
> +opt = ["-H:scheme: http"]
> +r = env.nghttp().get(url, options=opt)
> +assert r.exit_code == 0, r
> +assert r.response['status'] == 400
> 
> 

Re: http and http/1.x separation

[Roy T. Fielding]

> On Jan 24, 2022, at 6:53 AM, Stefan Eissing  wrote:
> 
> 
> 
>> Am 24.01.2022 um 15:40 schrieb Yann Ylavic > >:
>> 
>> On Mon, Jan 24, 2022 at 3:28 PM Stefan Eissing > > wrote:
>>> 
>>> FYI: I am busy hacking away at the separation between our HTTP and HTTP/1.x 
>>> handling code. I'll make a PR once all tests have passed, so we can talk 
>>> about the changes.
>> 
>> Looking forward to seeing it ;)
> 
> 
> 
>> 
>>> 
>>> The goals are pretty simple:
>>> 
>>> 1. have mod_http focus on HTTP semantics, header checks, ap_die conversions,
>>>  not chunking and CRLFs.
>>> 2. have a mod_http1 that installs TRANSCODE filters when the connection is 
>>> h1.
>>>  those do the chunking and the header serialization.
>>> 3. Have interim responses that send META buckets, not DATA to the transcode 
>>> filters.
>>>  No content filter will be confused by those.
>>> 4. Pass FOOTERS in/out as META buckets in a similar way.
>>> 
>>> The main idea is to introduce a META "HEADERS" bucket that is used for
>>> final/interim responses and footers as well. This will safely pass through
>>> all filters that do not know about it. This is similar to the ERROR bucket
>>> type we already have.
>> 
>> There are some assumptions in httpd (and possibly third-party modules)
>> that meta buckets have a no length, so if "HEADERS" buckets have one
>> you might need to embed it in the bucket->data struct..
> 
> Yes, it's 0 length like ERROR and has a struct as that holds:
> - int status, opt 0 
> - const char *reason, opt NULL
> - apr_table_t *headers
> - apr_table_t *notes
> 
> on responses, shallow copied from the request_rec, etc.
> 
> The nice thing is that the HTTP/1.x transcode out filter can
> now check in one place that
> - there is indeed a HEADERS before data buckets
> - there is only one HEADERS with status >= 200
> 
> And "above" the transcode layer, a data bucket is content, period.

And trailers can be implemented this way too, right?

Roy

Re: svn commit: r1895955 - in /httpd/httpd/branches/2.4.x: ./ CHANGES include/ap_mmn.h include/http_protocol.h modules/http/http_request.c modules/http2/h2_request.c modules/proxy/mod_proxy.c modules/

[Roy T. Fielding]

> On Dec 14, 2021, at 2:22 PM, Yann Ylavic  wrote:
> 
> On Tue, Dec 14, 2021 at 6:26 PM Roy T. Fielding  wrote:
>> 
>> This is a little confusing. It looks from the comment and code like the
>> change is restricting the request target that can be sent through a proxy,
>> which would be wrong.
> 
> Yeah, the hunk I pointed to in the other message is doing this and I'm
> going to fix it.

Thanks.

>> OTOH, it would make sense that the proxy itself (the thing to which the
>> proxied request is being sent) is limited to http(s) because that is a 
>> feature
>> of HTTP. Is that what was intended?
> 
> And the ap_post_read_request() part of the patch is enforcing that
> yes, so keeping it would be fine right?

Yep, that should be fine, at least until someone implements proxying via some 
other protocol.

Roy

Re: svn commit: r1895955 - in /httpd/httpd/branches/2.4.x: ./ CHANGES include/ap_mmn.h include/http_protocol.h modules/http/http_request.c modules/http2/h2_request.c modules/proxy/mod_proxy.c modules/

[Roy T. Fielding]

This is a little confusing. It looks from the comment and code like the
change is restricting the request target that can be sent through a proxy,
which would be wrong.

OTOH, it would make sense that the proxy itself (the thing to which the
proxied request is being sent) is limited to http(s) because that is a feature
of HTTP. Is that what was intended?

Roy

> On Dec 14, 2021, at 9:10 AM, Roy T. Fielding  wrote:
> 
> I am pretty sure that this isn't correct, or at least seems like overkill.
> We should definitely block unix: from being forwarded, but why would
> we want to block things like a urn: resolver?
> 
> To be clear, I'd rather remove all proxy functionality from httpd than
> suggest to the world that http(s) schemes are the only names that
> can be proxied through HTTP. It would break the Web architecture,
> so -1 to that.
> 
> Roy

Re: svn commit: r1895955 - in /httpd/httpd/branches/2.4.x: ./ CHANGES include/ap_mmn.h include/http_protocol.h modules/http/http_request.c modules/http2/h2_request.c modules/proxy/mod_proxy.c modules/

[Roy T. Fielding]

I am pretty sure that this isn't correct, or at least seems like overkill.
We should definitely block unix: from being forwarded, but why would
we want to block things like a urn: resolver?

To be clear, I'd rather remove all proxy functionality from httpd than
suggest to the world that http(s) schemes are the only names that
can be proxied through HTTP. It would break the Web architecture,
so -1 to that.

Roy


> On Dec 14, 2021, at 7:35 AM, yla...@apache.org wrote:
> 
> Author: ylavic
> Date: Tue Dec 14 15:35:56 2021
> New Revision: 1895955
> 
> URL: http://svn.apache.org/viewvc?rev=1895955=rev
> Log:
> Merge r1895914, r1895921 from trunk:
> 
>  *) http: Enforce that fully qualified uri-paths not to be forward-proxied
> have an http(s) scheme, and that the ones to be forward proxied have a
> hostname, per HTTP specifications.
> trunk patch: http://svn.apache.org/r1895914
>  http://svn.apache.org/r1895921
> 2.4.x patch: 
> https://patch-diff.githubusercontent.com/raw/apache/httpd/pull/286.patch
> backport PR: https://github.com/apache/httpd/pull/286
> +1: ylavic, minfrin, gbechis
> 
> 
> mod_proxy: Detect unix: scheme syntax errors at load time.
> 
> * modules/proxy/mod_proxy.c(add_pass, add_member, set_proxy_param,
>proxysection):
>  Check return value of ap_proxy_de_socketfy().
> 
> * modules/proxy/proxy_util.c(ap_proxy_get_worker_ex):
>  Check return value of ap_proxy_de_socketfy().
> 
> 
> 
> http: Enforce that fully qualified uri-paths not to be forward-proxied
>  have an http(s) scheme, and that the ones to be forward proxied have a
>  hostname, per HTTP specifications.
> 
> The early checks avoid failing the request later on and thus save cycles
> for those invalid cases.
> 
> 
> Submitted by: ylavic
> Reviewed by: ylavic, minfrin, gbechis
> Closes #286
> 
> Modified:
>httpd/httpd/branches/2.4.x/   (props changed)
>httpd/httpd/branches/2.4.x/CHANGES
>httpd/httpd/branches/2.4.x/include/ap_mmn.h
>httpd/httpd/branches/2.4.x/include/http_protocol.h
>httpd/httpd/branches/2.4.x/modules/http/http_request.c
>httpd/httpd/branches/2.4.x/modules/http2/h2_request.c
>httpd/httpd/branches/2.4.x/modules/proxy/mod_proxy.c
>httpd/httpd/branches/2.4.x/modules/proxy/proxy_util.c
>httpd/httpd/branches/2.4.x/server/protocol.c
> 
> Propchange: httpd/httpd/branches/2.4.x/
> --
>  Merged /httpd/httpd/trunk:r1895914,1895921
> 
> Modified: httpd/httpd/branches/2.4.x/CHANGES
> URL: 
> http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/CHANGES?rev=1895955=1895954=1895955=diff
> ==
> --- httpd/httpd/branches/2.4.x/CHANGES [utf-8] (original)
> +++ httpd/httpd/branches/2.4.x/CHANGES [utf-8] Tue Dec 14 15:35:56 2021
> @@ -1,6 +1,10 @@
>  -*- coding: utf-8 -*-
> Changes with Apache 2.4.52
> 
> +  *) http: Enforce that fully qualified uri-paths not to be forward-proxied
> + have an http(s) scheme, and that the ones to be forward proxied have a
> + hostname, per HTTP specifications.  [Ruediger Pluem, Yann Ylavic]
> +
>   *) OpenSSL autoconf detection improvement: pick up openssl.pc in the
>  specified openssl path. [Joe Orton]
> 
> 
> Modified: httpd/httpd/branches/2.4.x/include/ap_mmn.h
> URL: 
> http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/include/ap_mmn.h?rev=1895955=1895954=1895955=diff
> ==
> --- httpd/httpd/branches/2.4.x/include/ap_mmn.h (original)
> +++ httpd/httpd/branches/2.4.x/include/ap_mmn.h Tue Dec 14 15:35:56 2021
> @@ -586,6 +586,7 @@
>  *   dav_find_attr().
>  * 20120211.120 (2.4.51-dev) Add dav_liveprop_elem structure and
>  *   dav_get_liveprop_element().
> + * 20120211.121 (2.4.51-dev) Add ap_post_read_request()
>  * 
>  */
> 
> @@ -594,7 +595,7 @@
> #ifndef MODULE_MAGIC_NUMBER_MAJOR
> #define MODULE_MAGIC_NUMBER_MAJOR 20120211
> #endif
> -#define MODULE_MAGIC_NUMBER_MINOR 120 /* 0...n */
> +#define MODULE_MAGIC_NUMBER_MINOR 121 /* 0...n */
> 
> /**
>  * Determine if the server's current MODULE_MAGIC_NUMBER is at least a
> 
> Modified: httpd/httpd/branches/2.4.x/include/http_protocol.h
> URL: 
> http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/include/http_protocol.h?rev=1895955=1895954=1895955=diff
> ==
> --- httpd/httpd/branches/2.4.x/include/http_protocol.h (original)
> +++ httpd/httpd/branches/2.4.x/include/http_protocol.h Tue Dec 14 15:35:56 
> 2021
> @@ -96,6 +96,13 @@ AP_DECLARE(void) ap_get_mime_headers(req
> AP_DECLARE(void) ap_get_mime_headers_core(request_rec *r,
>   

Re: disallow HTTP 0.9 by default?

[Roy T. Fielding]

> On Jul 22, 2021, at 12:29 AM, Stefan Eissing  
> wrote:
>> Am 21.07.2021 um 22:04 schrieb Eric Covener :
>> 
>> I was chasing an unrelated thread about close_notify alerts and
>> reminded me -- is it time to change the default for
>> HttpProtocolOptions from Allow0.9 to Require1.0?
>> 
>> As the manual says, the requirement was dropped in RFC 7230. It seems
>> like the kind of potential gadget in future desynch/smuggling kind of
>> attacks that shouldn't be on by default today.
>> 
>> Any opinions?
> 
> +1
> 
> I think the internet is a different place now from when 2.4 came out.

Yep, we have long past the point where the Internet depends on header fields
like Host being present to avoid various attacks. +1

Roy

where do we want to send website bot notices?

[Roy T. Fielding]

I was about to update the site config so that it wouldn't send notices
to dev, but I don't know whether they should instead go to cvs, docs, or
a new list (like notices at httpd).  Any opinions?

Roy

Re: Question about APR trunk and httpd ldap modules

[Roy T. Fielding]

> On May 28, 2021, at 9:59 AM, William A Rowe Jr  wrote:
> 
> AIUI, as he remains a PMC member, the veto remains binding per Roy's 
> conclusion, whether it was made 9 weeks ago or 9 years ago. I do not, so just 
> sharing historical pointers for those raising questions, no opinion remaining 
> of HTTP Server PMC choices at all. But I did want to point out that the 
> project did choose to ignore the vastly more secure PCRE 10 rewrite and is 
> still stuck at PCRE 8 (although I run bleed builds all the time of httpd 
> trunk X apr 2 trunk X pcre 10 with no issues at all.) In theory, if the APR 
> project has enough maintainers (minimum 3, more + than - votes), then 
> apr[+util] 1.x might persist for years after a 2.0 release, if such a release 
> occurs.

The veto, like any veto, was specific to both the context at the time (2.4.0)
and the change being made. The way to resolve it is to work together
towards either a common solution (in APR) or a narrow change (in httpd).

The way to not get anything done in ten years is to claim that someone
doesn't have the right to veto a change, and then insist on having the
high ground instead of working toward anything.

Projects don't *do* things; people do, preferably while working together
within the same project. It is much harder for people to do things together
when individuals are being painted into a corner, having their concerns
disregarded, or repeatedly being attacked just because you don't agree
with one decision they made.

I suggest that the way to fix this tiny little problem is to create a new
LDAP secure client library in httpd that has the very specific purpose
we need (call it ap_ldapsc_*) and then change httpd's current usage to
make use of that library instead. If that leads to enough energy to
eventually become a common utility on its own, then it can migrate
to a common LDAP library (not necessarily APR) at that later time.

Likewise, the way to update to PCRE 10 is to do the work necessary for
the update, including backwards compatible shims. It would make for
a good entry/student project.

Roy

Re: [VOTE] Release httpd-2.4.47

[Roy T. Fielding]

On Apr 29, 2021, at 5:18 AM, Ruediger Pluem  wrote:
> On 4/29/21 2:09 PM, Eric Covener wrote:
>> On Thu, Apr 29, 2021 at 7:40 AM Ivan Zhakov  wrote:
>>> I have noticed regression in ETag response header handling in httpd 2.4.47: 
>>> ETag response header is not set for HTTP 304 responses. While RFC 7232, 4.1 
>>> requires them for 304 responses [1]
>>> [[[
>>> The server generating a 304 response MUST generate any of the
>>> following header fields that would have been sent in a 200 (OK)
>>> response to the same request: Cache-Control, Content-Location,
>>> Date, ETag, Expires, and Vary.
>>> ]]]
>>> 
>>> httpd 2.4.46 and before sets ETag header for HTTP 304 responses.
>>> 
>>> Unfortunately, I don't have time right now to investigate this issue 
>>> further, but I think it's a critical regression for the patch release.
>>> 
>>> [1] https://tools.ietf.org/html/rfc7232#section-4.1
>>> 
>> 
>> Thanks for catching, I have revived the BZ thread here:
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=61820
>> In the issue  we were working from a draft of the caching RFC, maybe
>> part of how we got mixed up.
> 
> Probably we are doing it wrong and should not touch ANY of the headers for 
> 304 and instead need to fix
> 
> modules/cache/cache_storage::cache_accept_headers
> 
> such that we don't update the following headers of the cached entity:
> 
> Content-Encoding
> Content-Length
> Content-MD5
> Content-Range
> ETag

Yes.

> Given the current state I would wait announcing the release until this got 
> clarified and if we got it wrong fixed.

It's hard to tell what the current state is based on a bunch of deltas,
which is why I don't like discussions in bugzilla. Yes, it looks like people
keep getting confused on what should be *generated* on a 304 versus
what should be updated in a cache for 304.

As I said before, the bug is in removing header fields from a 304
response in the HTTP filter just as it is being sent. That is wrong and
unnecessary. It's not the HTTP filter's job to remove information
from a message (unless the filter itself is generating the message,
as in a canned error response).

The spec says SHOULD NOT send unnecessary metadata because
there are cases when they might need to be sent. Just remove the
entire conditional and let the generating code adhere to the spec.

Note that the current specs are in 

  
https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#status.304
 


Roy

Re: svn commit: r1884280 - in /httpd/httpd/trunk: CHANGES modules/proxy/mod_proxy_ftp.c modules/proxy/mod_proxy_http.c modules/proxy/proxy_util.c

[Roy T. Fielding]

> On Dec 16, 2020, at 2:52 AM, Graham Leggett  wrote:
> 
> On 12 Dec 2020, at 01:59, Roy T. Fielding  <mailto:field...@gbiv.com>> wrote:
> 
>> That is too many questions. The purpose of the cache requirement is so that 
>> the cache
>> does not deliver a non-validated entry after receiving a failed validation. 
>> It doesn't really
>> matter what code is returned so long as it is 5xx, so the specs are 
>> over-constraining here.
>> 
>> The CoAdvisor test suite is overly pedantic.
>> 
>> And, as stated, this only applies when the request contains max-revalidate 
>> and the
>> action being done is a cache revalidation. No other code should behave this 
>> way.
> 
> To clarify, the change under discussion covers behaviour when a proxy of any 
> kind (including an ftp proxy) suffers a low level network error of any kind, 
> from DNS errors to low level network errors.

Yes, and as I said, that is wrong.

> Whether ultimately correct or not, CoAdvisor was very specific as to the 
> codes to be returned in these cases.

I am very happy for their financial success. However, they are frequently
wrong, even though they are supposed to be based on the standard.

>> Reverting the change is the correct call, regardless, but it is also the 
>> right choice.
>> I have filed a bug on the Cache spec to change that MUST send 504 to a MUST 
>> send 5xx.
>> 
>>https://github.com/httpwg/http-core/issues/608 
>> <https://github.com/httpwg/http-core/issues/608>
>> 
>> If you think we need to change other things in the specs, please file bugs 
>> now.
> 
> The above issue relates to must-revalidate only, while the wider issue is for 
> which 5xx error to be returned in which situation. Can you clarify?
> 
> The patch in this thread covers each case, and basically boils down to a 
> choice between “Bad Gateway” or “Timed Out”.

The patch changed BAD_GATEWAY to Timed Out for no reason whatsoever.

The spec text that you pointed to is about cache and cache control when
the cache is disconnected from the network. The spec says that the cache
must generate a 504 when it is disconnected, meaning it has NO response
other than the entry in its cache. I fully understand why CoAdvisor is
confused and unable to properly test for that condition, given that it isn't
an internal cache testing suite, but this has nothing to do with a gateway
receiving or generating a 5xx code upon gateway fail. And, in any case,
the requirement in the spec is overly constraining when it is supposed to
be minimally preventing the cache from reusing its cached response.

IOW, gateway != cache.

> Given the long list here http://coad.measurement-factory.com/clients.html 
> <http://coad.measurement-factory.com/clients.html> I would be keen to make 
> sure httpd worked the same way as all those other servers.

That is not an interop concern.

Roy

Re: svn commit: r1884280 - in /httpd/httpd/trunk: CHANGES modules/proxy/mod_proxy_ftp.c modules/proxy/mod_proxy_http.c modules/proxy/proxy_util.c

[Roy T. Fielding]

> On Dec 11, 2020, at 6:28 AM, Ruediger Pluem  wrote:
> On 12/11/20 1:13 PM, Yann Ylavic wrote:
>> On Fri, Dec 11, 2020 at 12:49 PM Graham Leggett  wrote:
>>> 
>>> On 10 Dec 2020, at 18:04, yla...@apache.org wrote:
>>> 
>>> Author: ylavic
>>> Date: Thu Dec 10 16:04:34 2020
>>> New Revision: 1884280
>>> 
>>> URL: http://svn.apache.org/viewvc?rev=1884280=rev
>>> Log:
>>> Revert r1480058, -1'ed on dev@ and STATUS.
>>> 
>>> Never backported (and never will supposedly), while often creating
>>> merge conflicts.
>>> 
>>> You’ve just reverted a fix to an RFC violation that was picked up by the 
>>> CoAdvisor test suite.
>> 
>> Where is this test suite?
>> Which RFC violation, a proxy socket connection error should return 504
>> Gateway Timeout??
>> I see that RFC2616 14.9.4 is about cache, why don't you fix this in 
>> mod_cache?
>> 
>>> 
>>> “Creating merge conflicts” is not a sufficient technical justification for 
>>> a veto that results in the re-introduction of an RFC violation.
>>> 
>>> Please back this out.
>>> 
>>> See 
>>> https://lists.apache.org/thread.html/be0e7bdc3510fddd2dd80accece44917eba361ef4fcc713dd0f7f7fa%401367999236%40%3Cdev.httpd.apache.org%3E
>>> and 
>>> https://lists.apache.org/thread.html/6e63271b308a2723285d288857318e7bb51b6756690514d9bc75a71b%401371148914%40%3Ccvs.httpd.apache.org%3E
>>> 
>>> Please resolve the discussion above.
> 
> Just as an update for restarting the discussion:
> 
> Old RFC2616 Section   New RFC and section Link
> 14.9.4RFC7234, 5.2.2.1
> https://tools.ietf.org/html/rfc7234#section-5.2.2 
> 
> 10.5.3RFC7231, 6.6.3  
> https://tools.ietf.org/html/rfc7231#section-6.6.3 
> 
> 10.5.5RFC7231, 6.6.5  
> https://tools.ietf.org/html/rfc7231#section-6.6.5 
> 

Heh, sorry, the current version is

  
https://httpwg.org/http-core/draft-ietf-httpbis-cache-latest.html#cache-response-directive.must-revalidate
 

  
https://httpwg.org/http-core/draft-ietf-httpbis-cache-latest.html#rfc.section.4.3.3
 


  
https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#status.502
 


and we have until Monday to fix it, if needed.

> After rereading the sections on the new RFC's my opinion is still the same as 
> in
> 
> https://lists.apache.org/thread.html/be0e7bdc3510fddd2dd80accece44917eba361ef4fcc713dd0f7f7fa%401367999236%40%3Cdev.httpd.apache.org%3E
>  
> 
> 
> IMHO RFC7231, 6.6.3 and RFC7231, 6.6.5 apply for the proxy/gateway. In the 
> revalidate case (RFC7234, 5.2.2.1) the issue for the
> cache may be even wider: What if the reply on the revalidation request from 
> the backend is not cachable for whatever reason (like
> the 502 here without appropriate headers that allow caching)?
> Does the cache just pass the reply on and does not update its cache? Does it 
> remove the stale entry from the cache?
> 
> Apart from this the proxy could add a note to r->notes if there was a network 
> issue with the backend such that the cache can reply
> with 504 in this case. But of course this only helps in the case that the 
> next hop was not reachable. If the 502 is created by a
> further gateway between us and the origin server the issue remains.

That is too many questions. The purpose of the cache requirement is so that the 
cache
does not deliver a non-validated entry after receiving a failed validation. It 
doesn't really
matter what code is returned so long as it is 5xx, so the specs are 
over-constraining here.

The CoAdvisor test suite is overly pedantic.

And, as stated, this only applies when the request contains max-revalidate and 
the
action being done is a cache revalidation. No other code should behave this way.

>> You should do that, it's not my veto. Failing to resolve the
>> discussion, the commit should be reverted right?
>> 
>>> 
>>> The last on the thread is that Roy was asked for advice, but he was busy. 
>>> In the light of RFC7230 it would be useful to get a new answer on this 
>>> question.
>> 
>> Sure.
> 
> Can you please help us resolving this Roy?

Reverting the change is the correct call, regardless, but it is also the right 
choice.
I have filed a bug on the Cache spec to change that MUST send 504 to a MUST 
send 5xx.

   https://github.com/httpwg/http-core/issues/608 


If you think we need to change other things in the specs, please file bugs now.

Re: Reject HTTP protocols >= 2.0 in ap_parse_request_line?

[Roy T. Fielding]

> On Jun 18, 2020, at 9:03 AM, Stefan Eissing  
> wrote:
>> Am 18.06.2020 um 16:51 schrieb William A Rowe Jr :
>> 
>> 
>>>>>> On 6/18/20 12:09 AM, Roy T. Fielding wrote:
>>>>>>>> On Jun 8, 2020, at 12:56 AM, Ruediger Pluem  wrote:
>>>>>>>> 
>>>>>>>> I came across the question if we should not reject HTTP protocols >= 
>>>>>>>> 2.0 in the request line when we parse it
>>>>>>>> in ap_parse_request_line.
>>>>>>>> This does not affect mod_http2 if loaded as HTTP/2.0 connections 
>>>>>>>> itself are not parsed via ap_parse_request_line
>>>>>>>> and sending a
>>>>>>>> 
>>>>>>>> GET /something HTTP/2.0
>>>>>>>> 
>>>>>>>> as request line is not a valid way to start a HTTP 2.0 connection and 
>>>>>>>> I doubt that it will be for future major versions.
>> 
>> Correct, it starts an HTTP/1.1 connection, and the response should reflect 
>> HTTP/1.1.
>> 
>>>>>>> That isn't how these things typically work. New protocols are
>>>>>>> advanced with either deliberate backwards-compat or deliberate
>>>>>>> backwards-break, with an expectation that it will either do
>>>>>>> something useful on an older-protocol server or cause a safe
>>>>>>> error in an expected way.
>>>>>>> 
>>>>>>> Hence, we might still see an HTTP/4.0 that is designed to be
>>>>>>> parsed like HTTP/1.1 (by an old server) while at the same time
>>>>>>> work perfectly for a new server. That would be some hefty magic,
>>>>>>> but it remains possible. Likewise, we might want to deploy a
>>>>>>> version of h2 or HTTP/3 that works on unix domain sockets or
>>>>>>> localhost over non-Internet TCP.
>>>>>>> 
>>>>>>> This is why the existing code did not error on protocols >= 2.0.
>>>>>>> Doing so is both unnecessary and counterproductive. If parsing
>>>>>>> fails for some other reason, we want that other reason to be
>>>>>>> in the response (because that's what the new protocol will be
>>>>>>> expecting from an old protocol server). If it doesn't fail, we
>>>>>>> want to provide the successful response because the request
>>>>>>> was deliberately crafted that way to save a round trip.
>>>>>>> 
>>>>>>> Note that the incoming request protocol version should always
>>>>>>> be distinct from any forwarded request protocol or response
>>>>>>> protocol versions.
>> 
>> Precisely. If mod_http2 or quic/mod_http3 can do something with the 
>> connection
>> based on the request line, it's up to them through the hook facility to take 
>> ownership
>> of the connection.
> 
> That is not issue. That works well. 
> 
>> If they cannot/do not, then the core http1 connection/request processors 
>> remain
>> in place and in response to "please speak in HTTP/4.0" this server will 
>> respond,
>> "sure, here is your HTTP/1.1 response" as expected and defined by the RFC.
> 
> There are now several RFCs and they differentiate between HTTP/1.1 transport 
> and
> the pure HTTP semantics. This split is not reflected in our code, yet. We 
> have 
> functions that mix both. Not as a mistake, it's historical.
> 
> ap_parse_request_line() for example, checks the initial HTTP/1.1 request line 
> *and*
> the method names, uri, header_only and other request_rec fields.
> 
> We can either copy the latter into mod_http2 and maintain it in two places or
> have a core function for it to be invoked by mod_http and mod_http2. That 
> seems 
> to be the design decision to make.

I think that is the right way forward, though we have always tried
to minimize overhead for the common case. I guess the extra cycles
are irrelevant now.

> I used ap_parse_request_line() from mod_http2 to *not* duplicate the other 
> code,
> and someone added checks in trunk that imply it only ever gets called for 
> HTTP/1.x.
> So, now it pukes. Which is good, as it brought up this discussion.

Yep. The new RFCs should be "finished" by next month due to HTTP/3
being in WGLC this month.  If anyone's interested:

   https://github.com/httpwg/http-core <https://github.com/httpwg/http-core>
   https://github.com/quicwg/base-drafts <https://github.com/quicwg/base-drafts>

...Roy

Re: Reject HTTP protocols >= 2.0 in ap_parse_request_line?

[Roy T. Fielding]

> On Jun 8, 2020, at 12:56 AM, Ruediger Pluem  wrote:
> 
> I came across the question if we should not reject HTTP protocols >= 2.0 in 
> the request line when we parse it
> in ap_parse_request_line.
> This does not affect mod_http2 if loaded as HTTP/2.0 connections itself are 
> not parsed via ap_parse_request_line
> and sending a
> 
> GET /something HTTP/2.0
> 
> as request line is not a valid way to start a HTTP 2.0 connection and I doubt 
> that it will be for future major versions.

That isn't how these things typically work. New protocols are
advanced with either deliberate backwards-compat or deliberate
backwards-break, with an expectation that it will either do
something useful on an older-protocol server or cause a safe
error in an expected way.

Hence, we might still see an HTTP/4.0 that is designed to be
parsed like HTTP/1.1 (by an old server) while at the same time
work perfectly for a new server. That would be some hefty magic,
but it remains possible. Likewise, we might want to deploy a
version of h2 or HTTP/3 that works on unix domain sockets or
localhost over non-Internet TCP.

This is why the existing code did not error on protocols >= 2.0.
Doing so is both unnecessary and counterproductive. If parsing
fails for some other reason, we want that other reason to be
in the response (because that's what the new protocol will be
expecting from an old protocol server). If it doesn't fail, we
want to provide the successful response because the request
was deliberately crafted that way to save a round trip.

Note that the incoming request protocol version should always
be distinct from any forwarded request protocol or response
protocol versions.

Roy

Re: Migrate to git?

[Roy T. Fielding]

> On Oct 5, 2019, at 1:09 PM, Jim Jagielski  wrote:
> 
> Various PMCs have made their default/de-facto SCM git and have seen an 
> increase in contributions and contributors...
> 
> Is this something the httpd project should consider? Especially w/ the 
> foundation officially supporting Github, it seems like time to have a 
> discussion about it, especially as we start thinking about the next 25 years 
> of this project :)

I don't like git. I don't like isolating work on branches. If we want to work 
on it as a base,
we need to agree on how to manage the release branches first and their names 
(so that
the existing history can be merged accordingly).

I hate our existing bugzilla blackhole and manual process of PRs.

I do like Github. I like the flexibility and openness of Github issues and 
tracking with commits.
I like the PRs and review mechanisms.  I am not sure what to do about security 
issues.
I don't know how to handle folks who don't have Github IDs already (or don't 
want them).

Currently, I am +0 on the idea, but +1 if the way forward is worked out and the 
process
of migrating history is extensively automated/tested first.  I did that for 
httpwg and it was
a pain in the ass even with only five committers.  But it can be done.

Roy

Re: [PATCH] Return HTTP 431 (Request Header Fields Too Large) for requests with large headers

[Roy T. Fielding]

> On Aug 27, 2019, at 5:19 AM, Ivan Zhakov  wrote:
> 
> On Wed, 14 Mar 2018 at 10:05, Ivan Zhakov  wrote:
>> 
>> Hi,
>> 
>> Please find patch that changes HTTPD to return HTTP 431 (Request
>> Header Fields Too Large) for requests with large headers. This status
>> code is defined by RFC 6585 [1]. Currently HTTPD returns generic HTTP
>> 400 (Bad Request) status code for requests with large headers.
>> 
>> [1] https://tools.ietf.org/html/rfc6585#section-5
>> 
> Hey folks,
> 
> Is there any feedback on the proposed patch?
> 
> I believe that using the custom status code would be more appropriate
> in these cases.

I missed the patch, but +1 to the idea.  We should always use a specific code 
when available.

Roy

Re: keep-alive and vary in 304 responses

[Roy T. Fielding]

> On Apr 10, 2019, at 3:10 AM, Stefan Eissing  
> wrote:
> 
> 
>> Am 09.04.2019 um 18:48 schrieb Roy T. Fielding :
>> 
>>> 
>>> 2. Validation responses lose the "Vary" header from the unconditional 
>>> response. This happens on resources where mod_deflate is active.
>>> The 200 response without any "if-modified-since" etc. carries "Vary: 
>>> Accept-Encoding" as it should.
>>> The 304 response with "if-modified-since" has no "Vary" header.
>>> The code in mod_deflate looks as if it tries to add it in any case, where 
>>> is it lost?
>> 
>> That's one of many bugs related to design problems with mod_deflate.  
>> Basically,
>> it chooses to insert itself into the response handling after the other 
>> checks are done,
>> which means a 304 doesn't even know yet whether the response would use gzip.
>> The solution (which I never found time to implement) is to replace dynamic 
>> gzip with
>> a transfer encoding generator, and/or to rewrite the mod_deflate CE encoder 
>> to act more
>> like mod_cache.
> 
> Thinking out loud here:
> 
> Acting more like mod_cache would then split the deflate output filter into a 
> resource and a protocol type one, I assume?

Yeah, probably.  What I meant was that deflate would act like a proxy to the 
resource
backend, provide variants transparently regardless of the client's request, and 
maintain
a cache of (at least) the metadata for available variants. That would allow the 
metadata
to be consistent over time, and possibly allow the dynamically compressed 
variants
to also be cached for later use.

It's also reasonable (though perhaps less cache efficient) to add "Vary: 
Accept-Encoding"
on any response that has a chance of being dynamically compressed even if we 
decide
that *this* response will not be.

Roy

Re: keep-alive and vary in 304 responses

[Roy T. Fielding]

> On Apr 9, 2019, at 3:30 AM, Stefan Eissing  
> wrote:
> 
> I just did some tests with https://redbot.org/ (the site tester by Mark 
> Nottingham) against our server and it notifies of 2 things:
> 
> 1. The "Keep-Alive" header is deprecated. I tried to "Header unset 
> Keep-Alive" but that has no effect. Seems to be added very late. 
>   Do we have a way to suppress it?

Please hold off on that one.  We received several requests that it not be 
deprecated
because the information that Apache sends is useful for avoiding request loss 
when
the max is reached.

> 2. Validation responses lose the "Vary" header from the unconditional 
> response. This happens on resources where mod_deflate is active.
>   The 200 response without any "if-modified-since" etc. carries "Vary: 
> Accept-Encoding" as it should.
>   The 304 response with "if-modified-since" has no "Vary" header.
>   The code in mod_deflate looks as if it tries to add it in any case, where 
> is it lost?

That's one of many bugs related to design problems with mod_deflate.  Basically,
it chooses to insert itself into the response handling after the other checks 
are done,
which means a 304 doesn't even know yet whether the response would use gzip.
The solution (which I never found time to implement) is to replace dynamic gzip 
with
a transfer encoding generator, and/or to rewrite the mod_deflate CE encoder to 
act more
like mod_cache.

Roy

Re: Host header checking too strict?

[Roy T. Fielding]

> On Jun 25, 2018, at 8:57 AM, William A Rowe Jr  wrote:
> 
> On Mon, Jun 25, 2018 at 5:31 AM, Joe Orton  > wrote:
> On Fri, Jun 22, 2018 at 05:21:08PM -0400, Eric Covener wrote:
> > After CVE-2016-8743 we only accept hostnames that are valid in DNS,
> > which notably excludes underscores.  But it seems like 7230 does not
> > require HTTP Host: to use a DNS registry, and excluding  '_' should
> > have broken IDN (punycode) international domain names.
> > 
> > Meanwhile I have seen several reports of e.g. departmental servers or
> > proxypreservehost=off-like failures with hostnames w/ underscores.
> > 
> > Should we be more tolerant here, or offer an option?
> > 
> > [ ] No
> > [X] Just underscores, which seems to come up alot?
> 
> Yup, we had Fedora users complain about this as well after 2.6.25, +1 
> for underscores in hostnames allowed by default.
> 
> I'll concur, I see no problem "violating" the spec in this single respect.
> Note that the same is not true of, say, http field names. There, ambiguity
> between - and _ due to CGI is an actual problem.

The spec is at

  https://tools.ietf.org/html/rfc3986#section-3.2.2 


and 

  
https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#header.host
 


Whatever we are doing, underscore is allowed by the spec.  DNS is irrelevant 
here
because hostnames are not limited to DNS names.

It is reasonable for us to limit Host to be the set of allowed virtual hosts we 
are
willing to match, so we can certainly exclude the weird delimiters, but we
don't want to prevent access to hosts we allow to be configured.

BTW, note that the second link above is to the current editors' draft of HTTP,
which is being revised now.  If anyone wants to reduce the grammar here or
elsewhere, now is the time to make it an issue at

  https://github.com/httpwg/http-core 

Cheers,

Roy

Re: The Case for Managed Domains

[Roy T. Fielding]

> On Feb 9, 2018, at 12:06 PM, Stefan Eissing  
> wrote:
>> Am 09.02.2018 um 16:12 schrieb Daniel :
>> 
>> I'm getting lost.
>> 
>> What would VirtualServer tag mean exactly?
>> 
>> Thanks in advance and apologies for my slowness :)
> 
> The intention of VirtualServer is to solve common properties
> for a range of VirtualHosts. It's a level between the global
> settings and individual vhosts. Think of it as having several 
> sets of (nowadays) global configurations for a number of vhosts.

Hmm, I think reuse of that VS name is confusing.  How about
HostSet, HostGroup, GroupHost, or ServerGroup?

FWIW, I think it is terribly poor practice to serve the same resources
via both https and http.  Setting an auto-redirect would make more sense.

Roy

Re: svn commit: r1764961 - in /httpd/httpd/trunk: docs/manual/mod/core.xml modules/http/http_filters.c server/core.c server/gen_test_char.c server/protocol.c server/util.c

[Roy T. Fielding]

> On Oct 15, 2016, at 2:10 AM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
> 
> On Sat, Oct 15, 2016 at 3:54 AM, William A Rowe Jr <wr...@rowe-clan.net 
> <mailto:wr...@rowe-clan.net>> wrote:
> On Fri, Oct 14, 2016 at 4:44 PM, Roy T. Fielding <field...@gbiv.com 
> <mailto:field...@gbiv.com>> wrote:
> Right, though several people have requested it now as errata. Seems likely to 
> be in the final update for STD.
> 
> In the HttpProtocolOptions Unsafe mode, it is tolerated.
> 
> Should it be the proper 'Strict' behavior to parse (never generate) such 
> noise? 
> 
> FWIW, I see very little harm in potentially unsafe chunk headers because
> it becomes a serious chore to inject between alternating \r-only vs \n-only 
> vs space trailing chunk headers. I'm not suggesting it can't be done, but
> most requests-with-body are intrinsically not idempotent, so one must be
> extremely clever to affect cache history. 
> 
> But it isn't impossible, so if the editors follow the way of BWS vs. follow 
> the absolute explicit statements about HTTP request field names and
> the trailing ':', I'd be somewhat disappointed. Tighten ambiguity where
> there was little ambiguity before. Make explicit the real ambiguity for
> all user-agents and servers to implement. /shrug.

We tried.  People complained.

In any case, BWS only includes *( SP / HTAB ).  Not much ambiguity there.

Roy

Re: svn commit: r1764961 - in /httpd/httpd/trunk: docs/manual/mod/core.xml modules/http/http_filters.c server/core.c server/gen_test_char.c server/protocol.c server/util.c

[Roy T. Fielding]

Right, though several people have requested it now as errata. Seems likely to 
be in the final update for STD.

Roy


> On Oct 14, 2016, at 2:16 PM, William A Rowe Jr  wrote:
> 
>> On Fri, Oct 14, 2016 at 3:48 PM,  wrote:
>> Author: wrowe
>> Date: Fri Oct 14 20:48:43 2016
>> New Revision: 1764961
>> 
>> URL: http://svn.apache.org/viewvc?rev=1764961=rev
>> Log:
>> [...]
>> Apply HttpProtocolOptions Strict to chunk header parsing, invalid
>> whitespace is invalid, line termination must follow CRLF convention.
>> 
>> [...]
>  
>> static apr_status_t parse_chunk_size(http_ctx_t *ctx, const char *buffer,
>> [...]
>  
>> -else if (c == ' ' || c == '\t') {
>> +else if (!strict && (c == ' ' || c == '\t')) {
>>  /* Be lenient up to 10 BWS (term from rfc7230 - 3.2.3).
>>   */
>>  ctx->state = BODY_CHUNK_CR;
> 
> I'm not sure where this myth came from... 
> 
> https://tools.ietf.org/html/rfc7230#section-4.1
> 
> has *NO* provision for BWS in the chunk size.

Re: StrictURI in the wild [Was: Backporting HttpProtocolOptions survey]

[Roy T. Fielding]

> On Sep 14, 2016, at 6:28 AM, William A Rowe Jr  wrote:
> 
> On Tue, Sep 13, 2016 at 5:07 PM, Jacob Champion  > wrote:
> On 09/13/2016 12:25 PM, Jacob Champion wrote:
> What is this? Is this the newest "there are a bunch of almost-right
> implementations so let's make yet another standard in the hopes that it
> won't make things worse"? Does anyone know the history behind this spec?
> 
> (My goal in asking this question is not to stare and point and laugh, but 
> more to figure out whether we are skating to where the puck is going. It 
> would be nice for users to know which specification StrictURI is being strict 
> about.)
> 
> RFC3986 as incorporated by and expanded upon by reference in RFC7230. 
> 
> IP, TCP, HTTP and it's data and framing are defined by the IETF. HTTP's
> definition depends on the meaning of many things, including ASCII, URI
> syntax, etc, see its table of citations. The things it depends on simply
> can't be moving targets any more than those definitions that the TCP 
> protocol is dependant upon. The IETF process is to correct a broken 
> underlying spec with a newly revised spec subject to peer review, and 
> then update the consuming specs to leverage the changes in the 
> underlying, where necessary (in some cases the revised underlying
> spec, once applied, has no impact on the consuming spec.)
> 
> HTML folks use URL's, and therefore forked the spec they percieved as
> too rigid and inflexible. In fact, it wasn't, but it appears so if you read 
> the
> spec as requiring -users- to -type- valid URI's, which was never the case.
> Although it gets prickly if you consider handling badly authored href= links 
> in html. HTML became a "living spec" subject to perpetual evolution;
> this results in a state where all implementations are perpetually broken.
> But the key take-away is that whattfwg URI does not and cannot
> supercede RFC3986 for the purposes of RFC7230. Rather than improve
> the underlying spec, the group decided to overlay an unrelated spec.
> 
> https://daniel.haxx.se/blog/2016/05/11/my-url-isnt-your-url/ 
>  does one
> decent job explaining some of this. Google "URI whatwg vs. ietf" for
> an excessively long list of references.
> 
> So in short, whatwg spec describes URI's anywhere someone wants
> to apply their defintion; HTML5 is based upon this. The wire protocol 
> of talking to an http: schema server is defined by RFC7230, which 
> subordinates to the RFC3986 definition of a URI. How you choose to 
> apply these two specs depends on your position in the stack.

I don't consider the WHATWG to be a standards organization, nor should
anyone else. It is just a selective group (a clique) with opinions about
software that they didn't write and a desire to document it in a way that
excludes the interests of everyone other than browser developers.

The main distinction between the WHATWG "URL standard" (it isn't)  and
the IETF URI standard (it is, encompassing URL and URN) is that HTML5
needs to define the url object in DOM (what is basically an object containing
a parsed URI reference), whereas the IETF needs to define a grammar for
the set of uniform identifiers believed to be interoperable on the Internet.

Obviously, if one spec wants to define everything a user might input as a
reference and call that "URL", while the other wants to define the interoperable
identifier output after uniform parsing of a reference relative to a base URI
as a "URL", the two specs are not going to be compatible.

Do you think the empty string ("") is a URL?  I don't.

A normal author would have used two different terms to define the two
different things (actually, four different things, since the URL spec also uses
url to describe two other things related to URL processing). The IETF chose a
different term, 23 years ago, when it created the term URL instead of just
defining them as "WWW Addresses" or universal document identifiers.

Instead of making a rational effort to document references in HTML, the
WHATWG decided to go on an ego trip about what "real developers" call
a "URL", and then embarked on yet another political effort to reject IETF
standards (that represent the needs of all Internet users, not just
browser developers) in favor of their own "living standards" that only
reflect a figment of the author's imagination (not implementations).

Yes, a user agent will send invalid characters in a request URI.  That is a bug
in the user agent.  Even if every browser chose to do it, that is still a bug in
the browser (not a bug in the spec). The spec knows that those addresses
are unsafe on the command-line and therefore unable to be properly
handled by many parts of the Internet that are not browsers, whereas
the correctly encoded equivalent is known to be interoperable. Hence,
the real standard requires that they be sent in an interoperable form.

Re: svn commit: r1756531 - /httpd/httpd/trunk/modules/proxy/proxy_util.c

[Roy T. Fielding]

> On Aug 16, 2016, at 9:51 AM, Eric Covener <cove...@gmail.com> wrote:
> 
> On Tue, Aug 16, 2016 at 12:26 PM, Roy T. Fielding <field...@gbiv.com> wrote:
>> It used to be that we always log INFO because we only use it for noting
>> configuration details.  Has that changed?
> 
> You're probably thinking of the special handling of NOTICE level, so n/a here.

Oh, right. Brain fart.

Roy

Re: svn commit: r1756531 - /httpd/httpd/trunk/modules/proxy/proxy_util.c

[Roy T. Fielding]

> On Aug 16, 2016, at 9:21 AM, yla...@apache.org wrote:
> 
> Author: ylavic
> Date: Tue Aug 16 16:21:13 2016
> New Revision: 1756531
> 
> URL: http://svn.apache.org/viewvc?rev=1756531=rev
> Log:
> Follow up to r1750392: reduce AH03408 level to INFO as suggested by wrowe/jim.

It used to be that we always log INFO because we only use it for noting
configuration details.  Has that changed?

Roy

Re: HTTP/1.1 strict ruleset

[Roy T. Fielding]

> On Aug 11, 2016, at 9:59 AM, William A Rowe Jr  wrote:
> 
> On Thu, Aug 11, 2016 at 11:54 AM, Eric Covener  > wrote:
> On Thu, Aug 11, 2016 at 12:44 PM, William A Rowe Jr  > wrote:
> > Just to be clear, that is now 2 votes for eliminating the 'classic parser'
> > from all
> > of trunk, 2.4.x and 2.2.x branches, and using only the strict parser,
> > unconditionally.
> >
> > That's actually 3 votes for removing it from trunk, which I planned to do
> > already,
> > after 2.4 and 2.2 backports are in-sync with trunk.
> 
> Without yet reviewing the votes, I would (personally) think this kind
> of split makes it your call as the one neck deep in the issue & doing
> all the work.Thank you for your work on this.
> 
> Maybe one last summary of your call, and a short window for strong
> objection/veto?
> 
> Certainly, that's what the backport proposal of everything from the initial
> commit by sf all the way to the present state will accomplish in STATUS.
> 
> With so many evolutions of various bits, a summary patch will be provided,
> of course. But it's helpful to me to know the opinions of Jim and Roy and
> everyone else in advance of proposing that backport.

I am having trouble keeping up while dealing with summer parenting issues.

I have no doubt that a strict parser is necessary, for some definition of 
strict.
I have no idea why there is any need to discuss EBCDIC here, since HTTP itself
is never EBCDIC.  We should not be transforming any input to EBCDIC until
after the request has been parsed.

I am not convinced that we need a wholesale rewrite of the parser code to
accomplish anything. Since most of the changes to trunk were tossed and repeated
multiple times due to unfamiliarity with the train parsing code or unawareness
that the read is already handling obs-fold and spurious newlines, I still think
we should just commit the simple fix (with your added logging statements)
and remove the bits from trunk that we don't actually need.

That doesn't mean we shouldn't attempt a better parser.  However, I would like
to review that as one clean diff with demonstrated performance tests.
That means setting up a test harness and proving that it is actually better.
For example, we might want to try using (or at least learning from) other 
parsers
that can work on the entire received buffer in one pass, rather than limit 
ourselves
to the existing line-at-a-time process, and simultaneously deprecate or remove
handling of obs-fold and HTTP/0.9.

In any case, if you have a working parser implementation, I will be happy to
review it regardless of my preferences.  If it is better than what we have, then
it will still get my +1 for trunk regardless of longer term plans.

Roy

Re: HTTP/1.1 strict ruleset

[Roy T. Fielding]

> On Aug 4, 2016, at 3:02 PM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
> 
> On Thu, Aug 4, 2016 at 3:46 PM, Roy T. Fielding <field...@gbiv.com 
> <mailto:field...@gbiv.com>> wrote:
> > On Aug 3, 2016, at 4:33 PM, William A Rowe Jr <wr...@rowe-clan.net 
> > <mailto:wr...@rowe-clan.net>> wrote:
> >
> > So it seems pretty absurd we are coming back to this over
> > three years later, but is there any reason to preserve pre-RFC 2068
> > behaviors? I appreciate that Stefan was trying to avoid harming
> > existing deployment scenarios, but even as I'm about to propose
> > that we backport all of this to 2.4 and 2.2, I have several questions;
> 
> In general, I don't see a need for any "strict" options. The only changes we
> made to parsing in RFC7230 were for the sake of security and known failures
> to interoperate. This is exactly the feature we are supposed to be handling
> automatically on behalf of our users: secure, correct, and interoperable
> handling and generation of HTTP messaging.  We should not need to configure 
> it.
> 
> Note that the MUST requirements in RFC7230 are not optional. We either 
> implement
> them as specified or we are not compliant with HTTP. 
>  
> Understood. And that describes my attitude toward 2.6/3.0 next release.
> 
> We live in an ecosystem with literally hundreds of thousands of legitimate
> custom http clients asking httpd server for datum. Most projects would
> effectively declare their last major.minor release static, and fix the defects
> while doing all enhancement in their next release. This isn't that project.

By that logic, we can't make any changes in minor versions.  I disagree that
we have ever treated versioning in that way.  If a user doesn't like a bug fix,
they don't have to install the new version, or they have the code to unfix it.

> Because httpd fixes and introduces dozens of bugs each major.minor
> subversion release, and we truly agree that we want every user to move
> to the most recently released major.minor, breaking hundreds of these
> applications with *no recourse* in their build or configuration is 
> frustrating.

Leaving existing users in a broken state of non-compliance with the primary
Internet standard we are claiming to implement just because of unsubstantiated
FUD is far more frustrating.  Bugs get fixed. Users choose whether or not to 
install.
If we find a real problem in a deployed client that causes the bug fix to be 
intolerable,
then of course we will need to configure workarounds.  But we are not in that 
place.

> If consensus here agrees that no out-of-spec behavior should be tolerated
> anymore, I'll jump on board. I'm already in the consensus block that says
> we should not ship a new major.minor without disallowing all of this garbage.
> 
> It would be helpful if other PMC members would weigh in yea or nay on
> dropping out-of-spec behaviors from 2.4 and 2.2 maintenance branches. 

That would be weird.  One of us is going to create a patch.  That specific 
patch is
going to be voted upon for backport.  If anyone wants to veto it, they are free
to do so with justification.

Roy

Re: HTTP/1.1 strict ruleset

[Roy T. Fielding]

> On Aug 3, 2016, at 4:33 PM, William A Rowe Jr  wrote:
> 
> So it seems pretty absurd we are coming back to this over
> three years later, but is there any reason to preserve pre-RFC 2068
> behaviors? I appreciate that Stefan was trying to avoid harming
> existing deployment scenarios, but even as I'm about to propose
> that we backport all of this to 2.4 and 2.2, I have several questions;

In general, I don't see a need for any "strict" options. The only changes we
made to parsing in RFC7230 were for the sake of security and known failures
to interoperate. This is exactly the feature we are supposed to be handling
automatically on behalf of our users: secure, correct, and interoperable
handling and generation of HTTP messaging.  We should not need to configure it.

Note that the MUST requirements in RFC7230 are not optional. We either implement
them as specified or we are not compliant with HTTP.  So, the specific issues of

https://tools.ietf.org/html/rfc7230#section-3

   A sender MUST NOT send whitespace between the start-line and the
   first header field.  A recipient that receives whitespace between the
   start-line and the first header field MUST either reject the message
   as invalid or consume each whitespace-preceded line without further
   processing of it (i.e., ignore the entire line, along with any
   subsequent lines preceded by whitespace, until a properly formed
   header field is received or the header section is terminated).

   The presence of such whitespace in a request might be an attempt to
   trick a server into ignoring that field or processing the line after
   it as a new request, either of which might result in a security
   vulnerability if other implementations within the request chain
   interpret the same message differently.  Likewise, the presence of
   such whitespace in a response might be ignored by some clients or
   cause others to cease parsing.

and

https://tools.ietf.org/html/rfc7230#section-3.2.4

   No whitespace is allowed between the header field-name and colon.  In
   the past, differences in the handling of such whitespace have led to
   security vulnerabilities in request routing and response handling.  A
   server MUST reject any received request message that contains
   whitespace between a header field-name and colon with a response code
   of 400 (Bad Request).  A proxy MUST remove any such whitespace from a
   response message before forwarding the message downstream.


must be complied with regardless of any "strict" config setting.

Some of those other things under "strict" seem a bit wonky. For example,
changing the Host header field when the incoming request URI is absolute
is fine by default but needs to be a configurable option for gateways.
Trying to validate IPv4/6 vs DNS doesn't work in intranet environments
that use local name servers.  The Location field-value is no longer required
to be absolute ("https://tools.ietf.org/html/rfc7231#section-7.1.2;).

> 1. offer a logging-only option? Why? It seems like a simple
>choice, follow the spec, or don't. If you want to see what's
>going on, Wireshark, Fiddler and dozens of other tools let
>you inspect the conversation.
> 
> 2. leave the default as 'not-strict'? Seems we should most
>strongly recommend that the server observe RFC's 2068,
>2616 and 723x, and not tolerate ancient behavior by default
>unless the admin insists on being foolish.

As far as the Internet is concerned, RFC723x is the new law of the land.
There is no reason to support obsolete RFCs.  No reason at all.  This has
nothing to do with semantic versioning or binary compatibility -- it is
simply doing what the product says it does: serve HTTP.

> 3. retain these legacy faulty behaviors in httpd 2.next/3.0?
>Seems that once we agree on a backport, the ancient
>side of this logic should all just disappear from trunk.
> 
> 4. detail the error to the error log? Again, there are inspection
>tools, but more importantly, no visual user-agent is going
>to send this garbage, and automated requests are going
>to discard the 400 response. Seems we can save a lot of
>code throwing away the details that just don't help, and
>are generally the product of abusive traffic.
> 
> Thoughts?

I think we just need to state in the log the reason for a 400 error. I don't 
like
sending invalid client-provided data back in a response, even when encoded.

Whitespace before the first header field can log a static message.
Whitespace after a field-name could log the field-name (no need to log the
field value). Invalid characters can be noted as "in a field-name" without
further data, or as "in a field-value" with only the field-name logged.

These are all post-error details off the critical path, so I don't buy the CPU
argument.  However, I do think our error handling in protocol.c has become
so verbose that it obscures the rest of the code.  Maybe it would be better if
we just 

Re: svn commit: r1754548 - /httpd/httpd/trunk/server/protocol.c

[Roy T. Fielding]

> On Aug 3, 2016, at 2:28 PM, William A Rowe Jr  wrote:
> 
> So AIUI, the leading SP / TAB whitespace in a field is a no-op (usually
> represented by a single space by convention), and trailing whitespace 
> in the field value is a no-op, all leading tabs/spaces (beyond one SP) 
> in the obs-fold line is a no-op. Is there any reason to preserve trailing 
> spaces before the obs-fold?

Not given our implementation.  The buffer efficiency argument is for other
kinds of parsers that are not reading just one line at a time.

> If not, then stripping trailing whitespace from the line prior to obs-fold and
> eating all leading whitespace on the obs-fold line will result in a single SP
> character, which should be just fine unless spaces were significant within
> a quoted value. The only way for the client to preserve such significant 
> spaces would be to place them after the opening quote before the obs-fold.

obs-fold is not allowed inside quoted text, so we need not worry about
messing with such a construct.

Note that obs-fold has been formally deprecated outside of message/http.
We can remove its handling at any time we are willing to accept the risk
of strange error reports.  I do not believe it is part of our versioning 
contract.

Roy

Re: svn commit: r1754548 - /httpd/httpd/trunk/server/protocol.c

[Roy T. Fielding]

> On Aug 3, 2016, at 11:44 AM, Jacob Champion  wrote:
> 
> On 07/31/2016 09:18 AM, William A Rowe Jr wrote:
>>> So all the trailing SP/HTAB are part of obs-fold IMHO.
>>> Should we replace all of them (plus the CRLF) with a single SP or with
>>> as many SP?
>> 
>> Hmmm... Good point. Advancing over them in our HTTP_STRICT mode seems
>> best, if we have a consensus on this.
> 
> Agreed that we should process all the obs-fold whitespace, and not just one 
> byte.
> 
> Replacing each byte with a separate space (as opposed to condensing into a 
> single space) *might* help prevent adversaries from playing games with header 
> length checks in more complicated/layered systems. That's probably a stretch 
> though. And if we consume the CRLF in a different layer of logic, adding on 
> two spaces just to keep everything "consistent" may also be a stretch. I'm 
> not feeling strongly either way.

What the spec is trying to say is that we can either replace all those bytes
with a single SP (semantically speaking they are the same) or we we can replace
them all with a sequence of SP (still the same, but doesn't require splitting
or recomposing the buffer).

> >> > So the obs-fold itself consists of CR LF [ SP | TAB ]
> >>
> >>obs-fold = CRLF 1*( SP / HTAB )
> >>
> 
> Note that this section of the spec has Errata associated with it; I'm reading 
> through the conversation [1] and it's seeming like they *may* want to treat 
> OWS preceding the CRLF as part of the obs-fold as well. I don't know what our 
> position is on adopting pieces of Errata that have been Held for Document 
> Update.

No, that is just an ABNF issue for matching purposes.  We don't use it.

Roy

Re: "Upgrade: h2" header for HTTP/1.1 via TLS (Bug 59311)

[Roy T. Fielding]

> On Apr 20, 2016, at 4:29 AM, Stefan Eissing  
> wrote:
> 
>> 
>> Am 20.04.2016 um 13:16 schrieb Yann Ylavic :
>> 
>> On Wed, Apr 20, 2016 at 1:09 PM, Yann Ylavic  wrote:
>>> On Wed, Apr 20, 2016 at 11:25 AM, Stefan Eissing
>>>  wrote:
 Done in r1740075.
 
 I was thinking of a nicer solution, but that involved inventing new hooks 
 which seems not worth it.
 
 Since this area of protocol negotiation has already been talked about in 
 regard to TLS upgrades
 and websockets, I do not want to invest in the current way of handling 
 this too much time.
>>> 
>>> I really don't see why we couldn't upgrade to h2 from "http:" (not
>>> "https:" since ALPN did not take place already, or would have done
>>> it).
>>> ISTM that "Upgrade: h2" could be valid in response to a (plain) HTTP/1
>>> request, and the client could upgrade from there...
>> 
>> More on this and Michael's quote of RFC 7540 ("A server MUST ignore an
>> "h2" token...").
>> An HTTP/2 server must indeed ignore the inner HTTP/1 request's
>> "Upgrade: h2" header since it's RFC states it, but and HTTP/1 server
>> (AFAICT) is not concerned by this RFC, and should not...
> 
> Totally agree. And, although untested, in principle we would upgrade
> such a request, as our protocol negotiation framework is somewhat agnostic
> to which RFC can leak water farther than others.

I think we need to be clear when the RFC applies and when it does not.

For example, ANY requirement in the RFC about TLS must be ignored by our
code (but not our default config) because we intend to implement h2c over
anything within some environments.  The RFC was written for the sole
use case of browsers making requests over the open Internet and we should
expect to deviate where its requirements are clearly for that context
rather than for technical reasons.

In any case, RFC7540 does allow an "Upgrade: h2" to be sent by the server
when the connection is already TLS; the fact that it actually means
"Upgrade: HTTP/2.0, TLS" can be handled as we see fit.  There is no
need to be pedantic because we are doing HTTP/1.1 at that point, not h2.

If there is a problem with sending h2 or h2c in a non-TLS response, then
we can send

   Upgrade: HTTP/2.0

instead (with or without the TLS, depending on how we are configured).
It says the same thing as h2c (or h2), as far as HTTP/1.x is concerned.

If that is still a problem with NodeJS, then we should have a conditional
workaround that is limited to the NodeJS client string (including its version);
we cannot allow broken clients to define the protocol in the long term.

Cheers,

Roy

Fwd: RFC 7804 on Salted Challenge Response HTTP Authentication Mechanism

[Roy T. Fielding]

For folks looking for a new feature to develop,

Roy


> Begin forwarded message:
> 
> From: rfc-edi...@rfc-editor.org
> Subject: RFC 7804 on Salted Challenge Response HTTP Authentication Mechanism
> Date: March 9, 2016 at 11:01:55 AM PST
> To: ietf-annou...@ietf.org, rfc-d...@rfc-editor.org
> Cc: drafts-update-...@iana.org, http-a...@ietf.org, rfc-edi...@rfc-editor.org
> Reply-To: i...@ietf.org
> List-Archive: 
> 
> A new Request for Comments is now available in online RFC libraries.
> 
> 
>RFC 7804
> 
>Title:  Salted Challenge Response HTTP Authentication 
>Mechanism 
>Author: A. Melnikov
>Status: Experimental
>Stream: IETF
>Date:   March 2016
>Mailbox:alexey.melni...@isode.com
>Pages:  18
>Characters: 39440
>Updates/Obsoletes/SeeAlso:   None
> 
>I-D Tag:draft-ietf-httpauth-scram-auth-15.txt
> 
>URL:https://www.rfc-editor.org/info/rfc7804
> 
>DOI:http://dx.doi.org/10.17487/RFC7804
> 
> This specification describes a family of HTTP authentication
> mechanisms called the Salted Challenge Response Authentication
> Mechanism (SCRAM), which provides a more robust authentication
> mechanism than a plaintext password protected by Transport Layer
> Security (TLS) and avoids the deployment obstacles presented by
> earlier TLS-protected challenge response authentication mechanisms.
> 
> This document is a product of the Hypertext Transfer Protocol Authentication 
> Working Group of the IETF.
> 
> 
> EXPERIMENTAL: This memo defines an Experimental Protocol for the
> Internet community.  It does not specify an Internet standard of any
> kind. Discussion and suggestions for improvement are requested.
> Distribution of this memo is unlimited.
> 
> This announcement is sent to the IETF-Announce and rfc-dist lists.
> To subscribe or unsubscribe, see
>  https://www.ietf.org/mailman/listinfo/ietf-announce
>  https://mailman.rfc-editor.org/mailman/listinfo/rfc-dist
> 
> For searching the RFC series, see https://www.rfc-editor.org/search
> For downloading RFCs, see https://www.rfc-editor.org/retrieve/bulk
> 
> Requests for special distribution should be addressed to either the
> author of the RFC in question, or to rfc-edi...@rfc-editor.org.  Unless
> specifically noted otherwise on the RFC itself, all RFCs are for
> unlimited distribution.
> 
> 
> The RFC Editor Team
> Association Management Solutions, LLC
> 
> 

where to put update_mime_types.pl?

[Roy T. Fielding]

I have a perl script (see below) for updating the mime.types file with the 
latest
registered IANA media types.  I would like to add it to our version control,
but I am unsure whether to place it in

  httpd/trunk/support/

or in

  httpd/docs-build/trunk/

I guess it depends on whether we want to distribute it as part of the product
or just use it ourselves as an occasional tool.  It is generally useful, though
not intended to be bullet proof.

Roy

==

#!/usr/bin/perl
#
# update_mime_types.pl: Read an existing Apache mime.types file and
# merge its entries with any new types discovered within an
# IANA media-types.xml file (see below for obtaining it).
#
# All existing mime.types entries are preserved as is (aside from sorting).
# Any new registered types are merged as a commented-out entry without
# an assigned extension, and then the entire file is printed to stdout.
#
# Typical use would be something like:
# 
#  wget -N http://www.iana.org/assignments/media-types/media-types.xml
#  ./update_mime_types.pl > new.types
#  diff -u mime.types new.types   ; check the differences
#  rm mime.types && mv new.types mime.types   ; only if diffs are good
#
# Note that we assume all files are in the current working directory
# and efficiency is not an issue.
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
my $mity = 'mime.types';
my $medy = 'media-types.xml';

die "no $mity here\n" unless (-e $mity);
die "no $medy here\n" unless (-e $medy);

my $in_head = 1;
my @header = ();
my %mtype = ();

# Read through the Apache httpd mime.types file to create tables
# keyed on the minor type names.  We save the entire input line as
# the hash value so that existing configs won't change when output.
# We assume the type names are already lowercased tokens.
#
die "cannot open $mity: $!" unless open (MIME, "<", $mity);

while () {
if ($in_head) {
push @header, $_;
if (/^# =/) {
$in_head = 0;
}
next;
}
if (/^(# )?([a-z_\+\-\.]+\/\S+)/) {
$mtype{$2} = $_;
}
else {
warn "Skipping: ", $_;
}
}
close MIME;

# Read through the IANA media types registry, in XML form, and extract
# whatever looks to be a registered type based on the element structure.
# Yes, this is horribly fragile, but the format isn't expected to change.
#
die "cannot open $medy: $!" unless open (IANA, "<", $medy);

my $major= 'examples';
my $thistype = '';

while () {
last if (/^\s*/);
next if (/(OBSOLETE|DEPRECATE)/);

if (/^\s*([^<]+)<\/name>/) {
$thistype = lc "$major/$1";
if (!defined($mtype{$thistype})) {
$mtype{$thistype} = "# $thistype\n";
}
}
}
close IANA;

# Finally, output a replacement for Apache httpd's mime.types file
#
print @header;

foreach $key (sort(keys %mtype)) {
print $mtype{$key};
}

exit 0;

Re: svn commit: r1725349 - /httpd/httpd/trunk/docs/manual/env.xml

[Roy T. Fielding]

I don't understand this comment.  RFC7230 doesn't recommend sending HTTP/1.0.
It certainly allows it as a workaround for a broken client, but 
force-response-1.0
is not recommended for general use.

Roy

> On Jan 18, 2016, at 1:14 PM, cove...@apache.org wrote:
> 
> Author: covener
> Date: Mon Jan 18 21:14:46 2016
> New Revision: 1725349
> 
> URL: http://svn.apache.org/viewvc?rev=1725349=rev
> Log:
> emphasize http/1.0 clients, mention RFC7230 calling this
> envvar a SHOULD.
> 
> --This line, and those below, will be inored--
> 
> Menv.xml
> 
> Modified:
>httpd/httpd/trunk/docs/manual/env.xml
> 
> Modified: httpd/httpd/trunk/docs/manual/env.xml
> URL: 
> http://svn.apache.org/viewvc/httpd/httpd/trunk/docs/manual/env.xml?rev=1725349=1725348=1725349=diff
> ==
> --- httpd/httpd/trunk/docs/manual/env.xml (original)
> +++ httpd/httpd/trunk/docs/manual/env.xml Mon Jan 18 21:14:46 2016
> @@ -322,12 +322,15 @@
> 
> force-response-1.0
> 
> -  This forces an HTTP/1.0 response to clients making an HTTP/1.0
> -  request. It was originally
> -  implemented as a result of a problem with AOL's proxies. Some
> +  This forces an HTTP/1.0 response to clients making an 
> +  HTTP/1.0 request. It was originally
> +  implemented as a result of a problem with AOL's proxies during the
> +  early days of HTTP/1.1. Some
>   HTTP/1.0 clients may not behave correctly when given an HTTP/1.1
> -  response, and this can be used to interoperate with them.
> -
> +  response, and this can be used to interoperate with them.  Later
> +  revisions of the HTTP/1.1 spec (RFC 7230) recommend this behavior 
> +  for HTTP/1.0 clients.
> + 
> 
> 
> 
> 
> 

Re: Upgrade Summary

[Roy T. Fielding]

> On Dec 8, 2015, at 2:07 AM, Stefan Eissing  
> wrote:
> 
> Trying to summarize the status of the discussion and where the issues are 
> with the current Upgrade implementation.
> 
> Clarified:
> A. any 100 must be sent out *before* a 101 response
> B. request bodies are to be read in the original protocol, input filters like 
> chunk can be used, indeed are necessary, as if the request is being processed 
> normally

Yes.

> C. if a protocol supports upgrade on request bodies is up to the protocol 
> implementation and needs to be checked in the "propose" phase

In some respects, yes, but Upgrade is defined by HTTP/1 and no other
protocol applies until after it is done.  That means ignoring any
idiotic handshake requirements of the "new" protocol that aren't
consistent with HTTP/1.  It also means h2's requirements on Upgrade
are irrelevant except for when it requires not upgrading to h2.

The client's assumption must be that the Upgrade will fail and any
attempt to use Expect will timeout, so the entire message will be
sent eventually regardless of anyone's stupid hacks to try and game
the protocol.  Hence, the server has no choice but to receive the
entire message as HTTP/1.1 even if it thinks it could have responded
fast enough to interrupt the client in mid-send.

> 
> Open:
> 1. Protocols like Websocket need to take over the 101 sending themselves in 
> the "switch protocol" phase. (correct, Jacob?). Should we delegate the 
> sending of the 101 to the protocol switch handler?

That seems unlikely.

> 2. General handling of request bodies. Options:
>  a setaside in core of up to nnn bytes before switch invocation
>  b do nothing, let protocol switch handler care about it

I think folks are confusing implementation with protocol.  There is no need
for the protocol being read on a connection to be the same protocol that is
being written in response.  In other words, the incoming connection remains
reading HTTP/1 bits until the message is finished, regardless of the decision
to upgrade the response stream -- the other protocol engine doesn't care.

This should be easily handled by adding a filter that translates the HTTP/1
incoming body (if any) to a single channel of the new protocol.  Just fake it.
There is no need to wait or set aside the bytes, unless that is desired for
other reasons (e.g., denial of denial of service attacks).

> 3. When to do the upgrade dance:
>  a post_read_request: upgrade precedes authentication
>  b handler: upgrade only honored on authenticated and otherwise ok requests
>  c both: introduce separate hooks? have an additional parameter? more 
> complexity

(a).  We do want to upgrade non-ok responses.  If the "new" protocol wants to
send a canned HTTP/1.1 error, it can do so without our help.

> 4. status code of protocol switch handler: if we move the task of 101 
> sending, the switch handler might not do it and keep the connection on the 
> "old" protocol. Then a connection close is not necessary. So, we would do the 
> close only when the switch handler returns APR_EOF.

Eh?

> 5. Will it be possible to migrate the current TLS upgrade handling to this 
> revised scheme?

TLS upgrade is never done with a body (and is broken code anyway).  Just fix it.
Note that the upgrade token is "TLS" -- it does not have to be "TLS/1.0".

Roy

Re: svn commit: r1710723 - in /httpd/httpd/trunk: CHANGES modules/cache/cache_util.h

[Roy T. Fielding]

> On Oct 26, 2015, at 11:45 PM, jaillet...@apache.org wrote:
> 
> Author: jailletc36
> Date: Tue Oct 27 06:45:03 2015
> New Revision: 1710723
> 
> URL: http://svn.apache.org/viewvc?rev=1710723=rev
> Log:
> RFC2616 defines #rules as:
>   #rule
>  A construct "#" is defined, similar to "*", for defining lists of
>  elements. The full form is "#element" indicating at least
>   and at most  elements, each separated by one or more commas
>  (",") and OPTIONAL linear white space (LWS). This makes the usual
>  form of lists very easy; a rule such as
> ( *LWS element *( *LWS "," *LWS element ))
>  can be shown as
> 1#element
> 
> It also defines Linear White Space (LWS) as:
>   LWS= [CRLF] 1*( SP | HT )
> 
> 
> The actual implementation only accepts SP (Space) and not HT (Horizontal Tab) 
> when parsing cache related header fields (i.e. "Vary", "Cache-Control" and 
> "Pragma")

Well, to be more accurate: RFC7230 defines these (2616 no longer applies) and
the original algorithm did handle HT.  My bet is that someone screwed up an
automated TAB -> two space conversion and the code change got lost in the noise.

Your fix looks right, though.

Roy

Re: Enforce rewriting of Host header when an absolue URI is given

[Roy T. Fielding]

> On Oct 26, 2015, at 10:33 AM, Jacob Champion  wrote:
> 
> Yann,
> 
> I found this while trying to understand the corner cases for Origin header 
> checks for mod_websocket, and I do actually have some thoughts on it...
> 
> On 03/04/2015 07:21 AM, Yann Ylavic wrote:
>> (by default, not only with "HttpProtocol strict", which is trunk only btw).
>> 
>> Per RFC7230#section-5.4 :
>>When a proxy receives a request with an absolute-form of
>>request-target, the proxy MUST ignore the received Host header field
>>(if any) and instead replace it with the host information of the
>>request-target.  A proxy that forwards such a request MUST generate a
>>new Host field-value based on the received request-target rather than
>>forward the received Host field-value.
>> 
>> The first part is already honored, but not the forwarding part: the
>> Host header is not rewritten with the one extracted from the absolute
>> URI, neither at the protocol (core) level nor at proxy level.
>> 
>> There are PR56718 (and duplicate PR57563) about this.
>> Still the same question about whether providing the headers to some
>> CGI or module is a forward or not,
> 
> I don't buy this. IMO, CGI/plugins/modules/etc. are implementation details of 
> the server.

Correct.  The word "proxy" in HTTP only applies to to the forwarding of HTTP
messages by a client-selected (forward) proxy.  There was no such thing as
a "reverse proxy" (an idiotic marketing term invented by Netscape) when the
original HTTP specs were written.  Those are gateways (as in Common Gateway 
Interface).

>> I think personally it would be sane
>> to do this at the protocol level (beginning of the request).
>> I proposed a patch there and refined it a bit (attached), so that
>> section 5.4 is applied in vhost.c::fix_hostname().
>> 
>> It also implements this part of section 5.4 (second point, _underlined_) :
>>A server MUST respond with a 400 (Bad Request) status code to any
>>HTTP/1.1 request message that lacks a Host header field and _to any
>>request message that contains more than one Host header field_ or a
>>Host header field with an invalid field-value.
>> 
>> The first point is already handled, and the third is "HttpProtocol
>> strict" dependent (which is enough I think, but maybe deserve a
>> backport too).

We should always be strict on received Host handling because misplaced routing
information is often used to bypass security filters.  That is, we should not 
allow
an invalid Host header field to pass through. It should at least be rejected
by default (if non-reject is configurable).

Roy

Re: svn commit: r1708593 - in /httpd/httpd/trunk: docs/manual/mod/mod_http2.xml modules/http2/h2_config.c modules/http2/h2_config.h modules/http2/h2_conn.c modules/http2/h2_h2.c modules/http2/h2_h2.h

[Roy T. Fielding]

Can you please choose a more specific directive name? Like "LimitTLSunderH2".

We don't have switches for RFC compliance. We do have switches for stupid WG 
political positions that contradict common sense and are not applicable to 
non-Internet deployments.

Roy


> On Oct 14, 2015, at 5:10 AM, ic...@apache.org wrote:
> 
> Author: icing
> Date: Wed Oct 14 12:10:11 2015
> New Revision: 1708593
> 
> URL: http://svn.apache.org/viewvc?rev=1708593=rev
> Log:
> mod_http2: new directive H2Compliance on/off, checking TLS protocol and 
> cipher against RFC7540
> 
> Modified:
>httpd/httpd/trunk/docs/manual/mod/mod_http2.xml
>httpd/httpd/trunk/modules/http2/h2_config.c
>httpd/httpd/trunk/modules/http2/h2_config.h
>httpd/httpd/trunk/modules/http2/h2_conn.c
>httpd/httpd/trunk/modules/http2/h2_h2.c
>httpd/httpd/trunk/modules/http2/h2_h2.h
>httpd/httpd/trunk/modules/http2/h2_switch.c
> 
> Modified: httpd/httpd/trunk/docs/manual/mod/mod_http2.xml
> URL: 
> http://svn.apache.org/viewvc/httpd/httpd/trunk/docs/manual/mod/mod_http2.xml?rev=1708593=1708592=1708593=diff
> ==
> --- httpd/httpd/trunk/docs/manual/mod/mod_http2.xml (original)
> +++ httpd/httpd/trunk/docs/manual/mod/mod_http2.xml Wed Oct 14 12:10:11 2015
> @@ -74,11 +74,11 @@
> Direct communication means that if the first bytes received 
> by the 
> server on a connection match the HTTP/2 preamble, the HTTP/2
> protocol is switched to immediately without further 
> negotiation.
> -This mode falls outside the RFC 7540 but has become widely 
> implemented
> -on cleartext ports as it is very convenient for development 
> and testing. 
> +This mode is defined in RFC 7540 for the cleartext (h2c) 
> case. Its
> +use on TLS connections is not allowed by the standard.
> 
> 
> -Since this detection implies that the client will send data 
> on
> +Since this detection requires that the client will send data 
> on
> new connection immediately, direct HTTP/2 mode is disabled by
> default.
> 
> 
> Modified: httpd/httpd/trunk/modules/http2/h2_config.c
> URL: 
> http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/http2/h2_config.c?rev=1708593=1708592=1708593=diff
> ==
> --- httpd/httpd/trunk/modules/http2/h2_config.c (original)
> +++ httpd/httpd/trunk/modules/http2/h2_config.c Wed Oct 14 12:10:11 2015
> @@ -49,6 +49,7 @@ static h2_config defconf = {
> 0,/* serialize headers */
> 0,/* h2 direct mode */
> -1,   /* # session extra files */
> +1,/* rfc 7540 compliance */
> };
> 
> static int files_per_session = 0;
> @@ -100,6 +101,7 @@ static void *h2_config_create(apr_pool_t
> conf->serialize_headers= DEF_VAL;
> conf->h2_direct= DEF_VAL;
> conf->session_extra_files  = DEF_VAL;
> +conf->rfc_compliance   = DEF_VAL;
> return conf;
> }
> 
> @@ -138,6 +140,7 @@ void *h2_config_merge(apr_pool_t *pool,
> n->serialize_headers = H2_CONFIG_GET(add, base, serialize_headers);
> n->h2_direct  = H2_CONFIG_GET(add, base, h2_direct);
> n->session_extra_files = H2_CONFIG_GET(add, base, session_extra_files);
> +n->rfc_compliance = H2_CONFIG_GET(add, base, rfc_compliance);
> 
> return n;
> }
> @@ -162,6 +165,8 @@ int h2_config_geti(h2_config *conf, h2_c
> return H2_CONFIG_GET(conf, , alt_svc_max_age);
> case H2_CONF_SER_HEADERS:
> return H2_CONFIG_GET(conf, , serialize_headers);
> +case H2_CONF_COMPLIANCE:
> +return H2_CONFIG_GET(conf, , rfc_compliance);
> case H2_CONF_DIRECT:
> return H2_CONFIG_GET(conf, , h2_direct);
> case H2_CONF_SESSION_FILES:
> @@ -332,8 +337,25 @@ static const char *h2_conf_set_direct(cm
> return "value must be On or Off";
> }
> 
> -#define AP_END_CMD AP_INIT_TAKE1(NULL, NULL, NULL, RSRC_CONF, NULL)
> +static const char *h2_conf_set_compliance(cmd_parms *parms,
> +  void *arg, const char *value)
> +{
> +h2_config *cfg = h2_config_sget(parms->server);
> +if (!strcasecmp(value, "On")) {
> +cfg->rfc_compliance = 1;
> +return NULL;
> +}
> +else if (!strcasecmp(value, "Off")) {
> +cfg->rfc_compliance = 0;
> +return NULL;
> +}
> +
> +(void)arg;
> +return "value must be On or Off";
> +}
> +
> 
> +#define AP_END_CMD AP_INIT_TAKE1(NULL, NULL, NULL, RSRC_CONF, NULL)
> 
> const command_rec h2_cmds[] = {
> AP_INIT_TAKE1("H2MaxSessionStreams", h2_conf_set_max_streams, NULL,
> @@ -354,6 +376,8 @@ const command_rec h2_cmds[] = {
>   RSRC_CONF, "set the 

Re: HTTP_MISDIRECTED_REQUEST

[Roy T. Fielding]

 On Aug 26, 2015, at 3:15 PM, William A Rowe Jr wr...@rowe-clan.net 
 mailto:wr...@rowe-clan.net wrote:
 
 Should this exception have a protocol version guard for HTTP/2.0 requests, 
 and leave the response as HTTP_BAD_REQUEST for HTTP/1.1 and earlier?
 
 @@ -203,6 +204,9 @@
  ap_log_error(APLOG_MARK, APLOG_ERR, 0, r-server, 
 APLOGNO(02032)
  Hostname %s provided via SNI and hostname %s 
 provided
   via HTTP are different, servername, host);
 +if (r-connection-keepalives  0) {
 +return HTTP_MISDIRECTED_REQUEST;
 +}
  return HTTP_BAD_REQUEST;
  }
  }

IIRC, it is applicable to HTTP/1.1 as well. Think misdirected requests 
containing
an absolute request URI that points to some other server.  I don't think the 
conditional
is needed at all -- just return HTTP_MISDIRECTED_REQUEST.

Hmm, I wonder how this impacts Google's desire to allow multiple hosts to reuse
the same SPDY connection ... was that dropped for h2?

Roy

Re: TWS ; LWS permitted by RFC 7230 4.1.1? Apparently, no.

[Roy T. Fielding]

 On Jun 15, 2015, at 9:33 AM, William A Rowe Jr wr...@rowe-clan.net wrote:
 
 Reviewing the spec, I cannot find where Sambar server is permitted to insert 
 whitespace. I further reviewed the ABNF appendix, and it does not appear 
 there, either.

Right, this was a deliberate decision to reduce the number of infinite stream 
possibilities.
We can still read a few SP and discard for robustness, but it should be limited 
to the same
few characters as leading zeros.

Sambar Server has been EOL for 7 years with no available source code for 
review, so it's
behavior is no longer relevant to the standard.

Roy

Re: RFC 7540 (HTTP/2) wrt reusable connections and SNI

[Roy T. Fielding]

 On Jun 9, 2015, at 3:42 AM, Yann Ylavic ylavic@gmail.com wrote:
 
 It just needed to get out :)
 
 But I agree that since we are to implement the RFC, we must comply,
 and find a way to still comply with HTTP/1.
 Both checks on SNI and renegotiation occur in the post_read_request
 hook, so we should be able to deal with vhost's parameters (configured
 Protocols, ProtocolTransports...), and do the right thing.
 
 On Tue, Jun 9, 2015 at 12:09 PM, Stefan Eissing
 stefan.eiss...@greenbytes.de wrote:
 Yann, I am with you and feel at least unease about this mixing.
 
 But the RFC has been approved and browsers will adhere to it. So if we do 
 not enforce some policies in the server, connections will fail for 
 mysterious reasons. And tickets will be raised...

Well, don't be too hasty.  There are a number of requirements in the RFC that
have nothing to do with HTTP and should be summarily ignored in the core 
implementation.
There are other requirements in the RFC that might turn out to be wrong or 
unnecessary,
just as we found in RFC2068, and it is our task to implement what works and 
change
the RFCs later.

However, the server as a whole should be configurable to be compliant (by 
default)
in the relevant code.  All of the requirements around TLS, for example, need to 
be
available in the SSL configs, but it is not h2's responsibility to ensure that 
it
has an RFC7540-compliant TLS config.  That is the admin's responsibility/choice.

WRT renegotiation, it is fair to say that the WG punted on the idea due to lack 
of time.
If someone figures out a way to safely renegotiate an h2 connection (and all of 
its
streams), then go ahead and implement it, describe it in an I-D, and submit it 
to
the httpbis WG.  There is nothing wrong with Apache leading by example.

Cheers,

Roy

Re: ALPN patch comments

[Roy T. Fielding]

 On Jun 4, 2015, at 9:19 AM, Stefan Eissing stefan.eiss...@greenbytes.de 
 wrote:
 
 I think we need to clarify some things:
 
 1. ALPN is initiated by the client. When a client does not send ALPN as part 
 of client helo, the SSL alpn callbacks are not invoked and the server does 
 not send any ALPN information back. This is different from NPN.
 
 2. SSLAlpnPreference is intended as the final word to make a choice when 
 either one ALPN callback proposes many protocols or of several callbacks 
 propose protocols. So, when mod_spdy and mod_h2 are active *and* the client 
 claims to support spdy/3.1 and h2, the SSLAlpnPreference determines what gets 
 chosen and sent to the client. This was not necessary with NPN as in that SSL 
 extension the client was making the choice.
 
 3. Independent of the client proposal, as I read the spec, the server is free 
 to chose any protocol identifier it desires. This might result in the client 
 closing the connection. So, if the client uses ALPN and the server does not 
 want/cannot do/is configured not to support any of the clients proposals, 
 httpd can always send back „http/1.1“ since this is what it always supports.
 
 In this light, and correct me if I’m wrong, I see no benefit and only 
 potential harm by introducing a „SSLALpn on|off“ configuration directive. I 
 think the current implementation covers all use cases and if one is missing, 
 please point out the scenario.

Ultimately, what we need is a single configuration that defines how the host
will respond to connections.  I suggest that this should be done on a per-vhost 
basis
if SNI is present, or a per-server basis if not.  It should not depend on 
either ALPN
or TLS being present.  This needs to be defined by the server admin, not hard 
coded in
the h2 code.  We should also have a way for the end of a response to reset the 
connection
to a possibly different set of protocols (i.e., Upgrade), but that's an 
independent concern.

Hence, we might need a configurable way to ignore a client's ALPN, though I 
doubt that
SSLalpn off is the right way to express that.  Likewise, neither is 
SSLAlpnPreference.
The server protocol(s) preference should be independent of the 
session/connection protocol.
Our internal configuration and use of ALPN should be based on the overall 
configuration, not a
configuration specific to the SSL code.  Many configurations won't include ALPN.

 As with the register once or on every connection optimization, yes, there 
 might be some performance to gain. But I think it is not so straightforward 
 to implement this, as not only the address and port influences this but also 
 the SNI that gets send in the client helo. So, one would have at least to 
 verify that registering an ALPN callback *after* the connection is open and 
 SNI has been received has any effect. 

I would hope that SNI is received before our connection is established (our 
connection is the
virtual session over TLS, not the TCP connection).  There shouldn't be any need 
to mess with
SSL internals within mod_h2.  Otherwise, it will be difficult to support h2c 
and h2 over SSH
with the same code.

Roy

Re: svn commit: r1492395 - in /httpd/httpd/trunk: CHANGES modules/aaa/mod_auth_digest.c

[Roy T. Fielding]

On Jun 14, 2013, at 2:16 PM, Stefan Fritsch wrote:
 On Thursday 13 June 2013, Roy T. Fielding wrote:
 On Jun 12, 2013, at 12:34 PM, s...@apache.org wrote:
 Author: sf
 Date: Wed Jun 12 19:34:19 2013
 New Revision: 1492395
 
 URL: http://svn.apache.org/r1492395
 Log:
 Actually use the secret when generating nonces.
 
 This change may cause problems if used with round robin load
 balancers. Before it is backported, we should add a directive to
 use a user specified secret.
 
 PR: 54637
 
 FWIW, I don't think this code can be released as is.
 
 I agree. That's what I wanted to express in the commit message, sorry 
 if that did not come out correctly.
 
 
 Using a global pointer to an allocated pool variable is
 not even remotely safe when that pool gets deallocated.
 And a routine that gets called within .htaccess files is not an
 appropriate place to set a server-wide value.
 
 It's the process pool, and that won't get cleaned up before server 
 shutdown. And the secret will be initialized in post_config hook at 
 the latest, so there is no chance that it will be called from 
 .htaccess. But moving the whole thing to pre_config would be clearer 
 and better. I will do that when I have some cycles.

The original code did not need access to a pool -- it was
a static global array.  As near as I can tell, the reason you
changed that was just to check to see if it has already been
initialized, which can be done easily by making the first byte
a non-random, non-zero constant.  In spite of the name, this
value has almost no need to be secret, so reducing the number
of random bytes by one won't matter.

 because that is where the real fix should be.  The current secret
 should be replaced here by a configurable string that is set in
 the virtual host config.
 
 Ack. Though in absence of a configured value, the random secret is 
 fine.

I think it will cause horrible performance issues for any large
installation of Digest, such as WebDAV services load-balanced
across multiple machines, so I think the fix is worse than the bug.
However, I am not confidant that I know enough about this code
to object further.  The original code seems much more convoluted
than it should be, which means either the design is inherently
wrong or I simply don't know why it needs to be so complicated.

Roy

Re: svn commit: r1492395 - in /httpd/httpd/trunk: CHANGES modules/aaa/mod_auth_digest.c

[Roy T . Fielding]

On Jun 12, 2013, at 12:34 PM, s...@apache.org wrote:

 Author: sf
 Date: Wed Jun 12 19:34:19 2013
 New Revision: 1492395
 
 URL: http://svn.apache.org/r1492395
 Log:
 Actually use the secret when generating nonces.
 
 This change may cause problems if used with round robin load balancers.
 Before it is backported, we should add a directive to use a user specified
 secret.
 
 PR: 54637

FWIW, I don't think this code can be released as is.

Yes, the prior code is broken, in that it creates the
nonce using a supposedly random value before that value is
actually initialized, and thus ends up creating a nonce that
is effectively just a timestamp.

However, fixing that is an error in itself, since the nonce
should not be random, which is why the existing code actually
works the way the reporter wanted.

The change from static global array to a static global pointer
into an allocated module config pool (?) obscures the bug fix
change of

 @@ -453,6 +473,21 @@ static void *create_digest_dir_config(ap
 static const char *set_realm(cmd_parms *cmd, void *config, const char *realm)
 {
 digest_config_rec *conf = (digest_config_rec *) config;
 +int i;
 +
 +/* pass NULL because cmd-server may not have a valid log config yet */
 +if (secret == NULL
 + initialize_secret(cmd-server-process-pool, NULL) != 
 APR_SUCCESS)
 +return Could not get random numbers for secret;
 +
 +#ifdef AP_DEBUG
 +/* check that we got random numbers */
 +for (i = 0; i  SECRET_LEN; i++) {
 +if (secret[i] != 0)
 +break;
 +}
 +ap_assert(i  SECRET_LEN);
 +#endif

Using a global pointer to an allocated pool variable is
not even remotely safe when that pool gets deallocated.
And a routine that gets called within .htaccess files is not an
appropriate place to set a server-wide value.

The nonce is supposed to be window-valid and verifiable, not
random.  Hence, this part of the nonce should be a configurable
salt shared by all servers for a given hostname.  Likewise,
the request host:port needs to be in the nonce hash, but now
I see that the prior code for that was commented-out by Ronald.
The window seems to be handled elsewhere using a timestamp.

If we assume that something like the above is planned
as an actual fix, then this patch is only complicating matters.
So, I suggest that the new code is not an improvement.

I think a better solution would be to simply remove the secret
hash entirely and figure out why this code in gen_nonce_hash()

1074ronald  85341/*
1075dougm   88019apr_sha1_update_binary(ctx, (const unsigned char *) 
server-server_hostname,
1076aaron   92626strlen(server-server_hostname));
1077dougm   88019apr_sha1_update_binary(ctx, (const unsigned char *) 
server-port,
1078aaron   92626sizeof(server-port));
1079ronald  85341*/

was commented out by Ronald in

http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/aaa/mod_auth_digest.c?r1=85340r2=85341pathrev=1492395view=patch

because that is where the real fix should be.  The current secret
should be replaced here by a configurable string that is set in
the virtual host config.

However, I am no expert with this code, and have no convenient
means for testing whatever I'd screw up by making changes here.
I am hoping that others on the list can look at this discussion
and either confirm my worries or explain why the random secret
is needed.

Cheers,

Roy

Re: RFC: Handling abnormally large chunk sizes

[Roy T. Fielding]

On May 14, 2013, at 8:58 AM, Graham Leggett wrote:

 Hi all,
 
 I am currently getting to the bottom of a test case that checks httpd's 
 response to an abnormally large chunk extension from a reverse proxy server. 
 What httpd does now is trigger an error, causing both the upstream and 
 downstream connections to be terminated and truncated. Is this the correct 
 way to respond to this?

Is there any other way to handle it?  How long are we talking about?
I certainly wouldn't try to process more than a single input buffer.

 http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-22#section-4.1 
 says this:
 
 All recipients MUST be able to receive and decode the chunked transfer 
 coding and MUST ignore chunk-ext extensions they do not understand.
 
 Does the above they do not understand requirement include the requirement 
 to ignore chunk-ext extensions that are too long? (For some arbitrary 
 definition of long)

Probably, but we can fix the requirement if you think it might be
a security issue (or just a bit too silly).  Note that sending
chunk-ext has been deprecated.

Roy

Re: svn commit: r1480058 - in /httpd/httpd/trunk: CHANGES modules/proxy/mod_proxy_ftp.c modules/proxy/mod_proxy_http.c modules/proxy/proxy_util.c

[Roy T. Fielding]

On May 8, 2013, at 1:11 AM, Ruediger Pluem wrote:
 Graham Leggett wrote:
 On 08 May 2013, at 9:47 AM, Ruediger Pluem rpl...@apache.org wrote:
 
 I don't agree with this. The case you mention is only true if the client 
 sends Cache-Control: must-revalidate.
 If this is not the case IMHO 10.5.3 and 10.5.5 apply.
 And only a cache is required to respond with 504 in this case, not a 
 gateway or a proxy. So the cache should
 change a 502 to a 504 in case the revalidation fails. Imagine the case 
 where you have other backend modules
 as our proxy modules.
 So while changing the response to 504 for failed DNS lookups is always 
 correct it is not for other failures.
 10.5.3: The server, while acting as a gateway or proxy, received an invalid 
 response from the upstream server it
 accessed in attempting to fulfill the request.
 
 Arbitrarily changing a 502 to a 504 on the fly will confuse people, as this 
 server isn't the only server that could generate a 502, and 502 might be the 
 intended error to be sent to the client.
 
 The way I interpret the RFC is that received an invalid response described 
 by 10.5.3 occurs when the upstream has said something that the proxy doesn't 
 like, while the unfortunately named 10.5.5 504 Gateway Timed Out describes a 
 timely response which means at the appropriate time, meaning a network 
 error of some kind.
 
 The change above isolates network errors specifically and returns 504 in 
 those cases only, while leaving 502 for genuine protocol errors, mostly 
 affecting ftp.
 
 Ultimately Roy is the authority on this?
 
 Good idea. He also knows possible clarifications on that that are work in 
 progress and that we could already apply.
 Roy can you please help us here?

Unfortunately, I am at the tail end of a long standards meeting and
haven't slept much for three days.  Have you checked to see if the
descriptions changed in HTTPbis p6?  RFC2616 hasn't been relevant
for a while now.

http://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p6-cache.html

I'll look at it tomorrow when I have slept a bit.

Roy

Re: mod_cache with Cache-Control no-cache= or private=

[Roy T. Fielding]

On Mar 13, 2013, at 10:20 AM, Graham Leggett wrote:

 On 11 Mar 2013, at 12:50 PM, Yann Ylavic ylavic@gmail.com wrote:
 
 The way I read the spec, the specified field-name(s) MUST NOT be sent in 
 the response to a subsequent request without successful revalidation with 
 the origin server. What this means is that if the specified field names 
 are found to be present in the cached response, then the origin server 
 needs to be given the opportunity to update these fields through a 
 conditional request. In the current cache code, we return 0 meaning this 
 is stale, revalidate, and a conditional request is sent to the origin. We 
 hope the origin sends 304 Not Modified, with updated headers 
 corresponding to the fields.
 
 Ok, I see your point, and this is surely the right reading of the rfc,
 but there is necessarily a difference between no-cache and
 no-cache=header(s), particularly with the handling of that (stale)
 header(s).
 
 For what I understand now, if the origin does not send (one of) the
 header(s) in its 304 response, the stale header(s) MUST NOT be
 served.
 
 I don't read it that way from the spec, I think it all comes down to the 
 phrase without successful revalidation with the origin server. I read it as 
 without successful revalidation [of the whole request] with the origin 
 server. In other words, the origin server sent the original header, if the 
 origin server doesn't update the header in the 304 response then it means 
 I've had my opportunity to revalidate the entity, current cached value is 
 fine, send it.
 
 Roy ultimately would be able to answer this?

It means delete the header fields prior to storing them in the cache
for later reuse.  If the origin had wanted must-revalidate, it would
simply use that directive instead.  The successful revalidation bit
is saying that the cache should forward all of the fields for the response
to the original request and for any response that is revalidated
(i.e., forward the new fields received in 304), but not for the
requests that are entirely handled by the cache.

Roy

Re: [Vote] Overhaul modules.apache.org

[Roy T. Fielding]

+1

Roy

Re: [VOTE] accept mod_macro as standard module in httpd

[Roy T. Fielding]

+1

Roy

Re: mod_macro contributors?

[Roy T. Fielding]

On Dec 30, 2012, at 5:19 PM, Eric Covener wrote:

 On Mon, Dec 24, 2012 at 1:42 AM,  fab...@apache.org wrote:
 
 Thanks Fabien. I am striking out on a valid e-mail for Dirk. Do you
 recall if he even sent in a patch, and if so the size/scope?
 
 
 It was not a patch. It was a bug report about the Warning  Error
 directives which were coredumping on Apache 2.2 because of changes between
 2.0  2.2. It triggered version 1.1.9 of mod_macro in November 2007.
 
 
 The only unresolved contribution is:
 
 - Paul Mcilwaine paul dot mcilwaine at gmail dot com,
   Mike Papper michael at realgirlsmedia dot com
   (Mac OS X compiliation issue with boolean/false/true)
 
 Fabien -- can you review/describe it here?
 
 (I did send Mike a mail at his new work address and didn't CC dev@
 but no response.)

Please note that it is not necessary to obtain permission for
contributions that consisted of a bug report or repair to the
existing functionality of the product, particularly when they
were freely given to the maintainer.  Those contributions are
not eligible for separate copyright anyway, and we shouldn't
annoy folks who aren't active in the development community when
their permission isn't needed in the first place.

Roy

Re: svn commit: r1426877 - in /httpd/httpd/trunk: CHANGES include/ap_mmn.h include/http_core.h include/httpd.h modules/http/http_filters.c server/core.c server/protocol.c server/util.c server/vhost.c

[Roy T. Fielding]

Hi Stefan,

Thanks for this work, but I don't consider HTTP conformance to be
an option.  These are checks we should be making while parsing
the received message, not as a separate pass, and in many cases
they are required to result in a 400, 500, or 502 response.

I am trying to get HTTPbis ready for last call this week.
After that, I will be looking into making the changes in httpd,
and I won't be using a configurable option.  I suggest we just
remove that part and iterate on these checks as we go.

The current HTTP/1.1 drafts are at

  http://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/

and the message parsing requirements are in p1-messaging.html

BTW, the protocol version is now restricted to uppercase and
one digit per major and minor, so we can simplify that check
to a very specific HTTP/[0-9].[0-9].

And, yes, it is possible to have a valid empty Host because
HTTP can be used with a proxy for any URI (including URNs).

Roy

On Dec 29, 2012, at 5:23 PM, s...@apache.org wrote:

 Author: sf
 Date: Sun Dec 30 01:23:24 2012
 New Revision: 1426877
 
 URL: http://svn.apache.org/viewvc?rev=1426877view=rev
 Log:
 Add an option to enforce stricter HTTP conformance
 
 This is a first stab, the checks will likely have to be revised.
 For now, we check
 
 * if the request line contains control characters
 * if the request uri has fragment or username/password
 * that the request method is standard or registered with RegisterHttpMethod
 * that the request protocol is of the form HTTP/[1-9]+.[0-9]+,
   or missing for 0.9
 * if there is garbage in the request line after the protocol
 * if any request header contains control characters
 * if any request header has an empty name
 * for the host name in the URL or Host header:
   - if an IPv4 dotted decimal address: Reject octal or hex values, require
 exactly four parts
   - if a DNS host name: Reject non-alphanumeric characters besides '.' and
 '-'. As a side effect, this rejects multiple Host headers.
 * if any response header contains control characters
 * if any response header has an empty name
 * that the Location response header (if present) has a valid scheme and is
   absolute
 
 If we have a host name both from the URL and the Host header, we replace the
 Host header with the value from the URL to enforce RFC conformance.
 
 There is a log-only mode, but the loglevels of the logged messages need some
 thought/work. Currently, the  checks for incoming data log for 'core' and the
 checks for outgoing data log for 'http'. Maybe we need a way to configure the
 loglevels separately from the core/http loglevels.

Re: svn commit: r1406719 - in /httpd/httpd/trunk: CHANGES docs/log-message-tags/next-number include/http_core.h server/core.c server/protocol.c

[Roy T. Fielding]

On Nov 8, 2012, at 2:39 AM, Stefan Fritsch wrote:

 On Wed, 7 Nov 2012, Graham Leggett wrote:
 
 On 07 Nov 2012, at 8:12 PM, Stefan Fritsch s...@sfritsch.de wrote:
 
 Any suggestions for a syntax? Maybe:
 
 HttpProtocol 1.1# only 1.1
 HttpProtocol 1.0-   # 1.0 and above
 HttpProtocol 1.0-1.1# 1.0 and 1.1
 HttpProtocol -1.0   # 1.0 and below
 
 We could then still add additional flags like +/- strict.
 
 The - in front of something means switch this off in other directives, I 
 suspect it might cause confusion.
 
 I have already used a similar syntax in RequestReadTimeout. But do you think 
 it would be clearer to require two numbers:
 
 HttpProtocol 0.0-1.0  # 1.0 and below
 HttpProtocol 1.0-10.100   # 1.0 and above
 
 The upper limit would be rather arbitrary. Of course, I don't see HTTP/2.x 
 arriving any time soon.
 
 Would it make sense to try use globbing (apr_fnmatch)? Perhaps multiple 
 options separated by commas, or an ITERATE separated by spaces?
 
 HttpProtocol *   # any version
 HttpProtocol 1.1 # only 1.1
 HttpProtocol 1.* # 1.0 and above
 HttpProtocol 1.0,1.1 # 1.0 and 1.1
 HttpProtocol 0.*,1.0 # 1.0 and below
 
 A list of allowed versions would be somewhat more complex to implement, 
 because it would require space in core_server_config to store a list (e.g. 
 with an apr_array) and we would need to iterate that list when checking the 
 request. Only having a minimum and maximum value seems like a less 
 over-engineered solution.
 
 Alternatively, if we use a list, then we could limit ourselves to the three 
 versions actually in use and not support arbitrary values. Not sure if that 
 is a future-proof solution.
 
 RFC2616 defines the version as follows:
 
  HTTP-Version   = HTTP / 1*DIGIT . 1*DIGIT
 
 We could potentially also add a check to make sure that DIGIT part is 
 checked to be actual digits, and reject it otherwise.
 
 In the received request? Yes, I think I had that one on my list already.

FWIW, I don't think any of this should be configurable.
HTTP/0.9 is on the chopping block -- it cannot be reasonably
supported on networks today because routing is based on the
Host header field.  We should just delete all backasswards.

HTTP/1.x must be supported to be 1.1 compliant and there is no
point in allowing configuration of support for future protocols
when we have nothing capable of processing those protocols.
Only new protocol modules can determine what else is supported.

Roy

Re: [Bug 53219] mod_ssl should allow to disable ssl compression

[Roy T. Fielding]

On Oct 7, 2012, at 6:05 PM, Eric Covener wrote:

 Any opinions on the default change?  AIUI current maintenance of
 browsers have disabled TLS compression already, because they can be
 driven to generate arbitrary traffic that eventually reveals httpOnly
 session cookies.

Just disable it completely -- adaptive compression of headers is
inherently incompatible with the goals of TLS.

Roy

Re: DNT IE10 (was svn commit: r1371878 - /httpd/httpd/trunk/docs/conf/httpd.conf.in)

[Roy T. Fielding]

On Sep 13, 2012, at 4:48 AM, Eric Covener wrote:

 On Sat, Aug 11, 2012 at 3:51 AM,  field...@apache.org wrote:
 Author: fielding
 Date: Sat Aug 11 07:51:52 2012
 New Revision: 1371878
 
 URL: http://svn.apache.org/viewvc?rev=1371878view=rev
 Log:
 Apache does not tolerate deliberate abuse of open standards
 
 I've come around on this one over time.  While I appreciate the
 message/intent, I don't think this is reasonable for the default
 configuration because it errs on the side of ditching a privacy header
 and information loss for a (sensitive) header that we're not yet
 interpreting.

For those of you who haven't been following along, I'll include some
links at the bottom for background.  DNT is not a privacy header.
There is no magic pixy dust that sprinkles privacy bits on anyone
that receives it.  DNT is supposed to be an expression of user
preference so that recipients will respect that user's desires.

It really is a question of deployment.  Right now, nobody can comply
with DNT on the server because none of the response mechanisms have
been approved yet and the meaning of DNT is not agreed.  There are
a few sites that had been recognizing DNT as equivalent to their
prior cookie-based opt-out, but most of those have since removed
support of DNT (either for all UAs or only for IE 10.0) because
of the default issue.

OTOH, if we were to attempt an implementation of DNT, then we could
address it directly with the user instead of dropping the header
field. Unfortunately, the WG has not yet agreed on a mechanism for a
server to indicate that it supports DNT in general, but for your
specific user agent we need to ask again to confirm that it was
by choice. There is also a general problem that, because compliance
means long-term data controls and access restrictions are promised
by the service owner, we can't respond as DNT compliant even if we
have complied within our own server software.

 IMO it's enough even without this specific DNT text:
 
 An HTTP intermediary must not add, delete, or modify the DNT header
 field in requests forwarded through that intermediary unless that
 intermediary has been specifically installed or configured to do so by
 the user making the requests. For example, an Internet Service
 Provider must not inject DNT: 1 on behalf of all of their users who
 have not selected a choice.

Yes (I wrote that part too), but keep in mind that we don't comply
with DNT yet, nor are we likely to until the access log issues
are resolved.  I agree that we cannot have the config remain if
we intend to comply with the standard, but that simply doesn't
matter if IE 10.0 destroys DNT before we can even get there.

 I'd like to revert it, but this is not yet a veto.  I'd like to hear
 what others think and would appreciate an ACK from Roy/Greg/Jim who
 voted for the backport to avoid any churn.

Strictly speaking, I don't think it is possible to veto a change made
in a prior release, but I think this one should be reverted (or at least
modified) if any of our PMC members feel so strongly now that they
would have vetoed it last month.  Consensus is important here.

Given the pathetic way that the Tracking Protection working group
members have addressed this issue, both for and against the behavior
of IE 10.0, I have lost any energy I once had for defending Mozilla's
original definition.  It was the only issue of substance that the WG
had managed to record consensus, in over a year of deliberation.
I would prefer that the WG change the text, one way or the other,
before we make another change, but I also want anything we do to be
based on what we think is right, not what others think or fail to do.

Regardless, I am +0 to revert, for none of the above reasons.
I am not fond of the performance hit of checking for a browser
version and setting an environment variable, just to support a
standard that is not being backed by the standards group.
I'd rather focus on new standards that aren't being manipulated
by EC/DC politics and the trolls that they feed.  I would, however,
like to leave the three browsermatch lines in the config
(commented out) as an example.

With regard to open letters, I'm done with that after our experience
with Sun.  If I had thought there was any chance of a letter working,
it would have been the first action proposed.  I am not opposed to
the idea of sending official feedback through our friends at Microsoft,
but please understand that it won't be effective unless we can make
it in their own interest to stop abusing the standard.

Apache's original mission is still important to me, even if the
rest of the world has forgotten.

  http://oreilly.com/catalog/opensources/book/brian.html

Roy

The following links may help with background.

http://www.w3.org/2011/tracking-protection/drafts/tracking-dnt.html#determining

http://blogs.technet.com/b/microsoft_on_the_issues/archive/2012/08/07/do-not-track-in-the-windows-8-set-up-experience.aspx

Re: Ideas for an output filter for mod_lua

[Roy T. Fielding]

On Aug 23, 2012, at 2:49 PM, Nick Kew wrote:

 On Thu, 23 Aug 2012 22:32:20 +0100
 Tim Bannister is...@jellybaby.net wrote:
 
 
 That's quite neat, then. I will try to make an actual implementation in Lua.
 The part I found difficult was the interaction with the second 
 transfer-encoding, “chunked”. Using gzip Transfer-Encoding: implies using 
 chunked, because we want to shorten the response and this means that the 
 Content-Length definitely doesn't match the size of the HTTP response body.
 
 gzip is a content-encoding, not a transfer-encoding.

It is both.  Transfer-Encoding is the correct way to implement it.

 That makes it easy to implement: content filters and protocol
 filters work at different levels, so can coexist without trouble.

Except that they aren't standards-compliant because they don't
handle etags correctly.  Content-Encoding works with more clients
but is much harder to implement correctly.

Roy

Re: [VOTE] Release Apache httpd 2.4.3 as GA

[Roy T. Fielding]

signatures are good
source files checked against svn
license and notices in place
compiles and runs on Mac OS X 10.7.4

+1

Roy

undesired modules loading

[Roy T. Fielding]

I built 2.4.3 with the options

./configure \
--prefix=$tdir \
--with-apr=$adir \
--with-apr-util=$adir \
--without-ssl \
--without-crypto \
--disable-cache \
--without-distcache \
--enable-maintainer-mode

but then noticed in the error_log some garbage about

[Sat Aug 18 12:26:23.182875 2012] [ssl:warn] [pid 82399:tid 140735220779360] 
AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Sat Aug 18 12:26:23.183039 2012] [lbmethod_heartbeat:notice] [pid 82399:tid 
140735220779360] AH02282: No slotmem from mod_heartmonitor

which is undoubtedly because several modules are being built and loaded
that depend on certain options but do not get disabled by the lack
of those options in configure.

IMO, none of the following modules should be configured by default:

LoadModule file_cache_module modules/mod_file_cache.so
LoadModule dumpio_module modules/mod_dumpio.so
LoadModule firehose_module modules/mod_firehose.so
LoadModule buffer_module modules/mod_buffer.so
LoadModule unique_id_module modules/mod_unique_id.so
LoadModule remoteip_module modules/mod_remoteip.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
LoadModule proxy_scgi_module modules/mod_proxy_scgi.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_express_module modules/mod_proxy_express.so
LoadModule session_module modules/mod_session.so
LoadModule session_cookie_module modules/mod_session_cookie.so
LoadModule session_dbd_module modules/mod_session_dbd.so
LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so
LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so
LoadModule lbmethod_heartbeat_module modules/mod_lbmethod_heartbeat.so
LoadModule dav_fs_module modules/mod_dav_fs.so

The vast majority of installations do not want a f*(^ing proxy.
Building them is fine -- these should be commented out in the
installed httpd.conf unless --with-proxy is enabled.

At the very least, modules dependent on SSL must not be loaded if
--without-ssl is the configure option.

I don't consider this a showstopper for 2.4.3, but I do think they
are bugs in trunk and 2.4.x.

Roy

Re: undesired modules loading

[Roy T. Fielding]

On Aug 18, 2012, at 1:45 PM, Rainer Jung wrote:

 Yes, before 2.4.0 we introduced exactly this difference. All modules that 
 were build get a LoadModule line in the installed config, but most are 
 commented out. I don't have the list of modles active by default at hand 
 though.

Ah, okay.

 At the very least, modules dependent on SSL must not be loaded if
 --without-ssl is the configure option.
 
 I don't consider this a showstopper for 2.4.3, but I do think they
 are bugs in trunk and 2.4.x.
 
 Partial answer: the current semantics of maintainer-mode is (per configure 
 help):
 
 Turn on debugging and compile time warnings and load all compiled modules

Umm, WTF?  Why?

 In fact configure should output
 
 Maintainer mode setting LOAD_ALL_MODULES to yes
 
 It seems you can switch of this side effect of maintainer mode by explicitly 
 adding --enable-load-all-modules=no to your configure flags. This is the 
 default except for when building in maintainer mode.

So, basically what you are saying is that an incompatible change
was made to the existing configuration flags so that my build
scripts are now broken for no good reason.

  --enable-load-all-modules
  Load all modules
  --enable-maintainer-mode
  Turn on debugging and compile time warnings and load
  all compiled modules
  --enable-debugger-mode  Turn on debugging and compile time warnings and turn
  off optimization
  --enable-modules=MODULE-LIST
  Space-separated list of modules to enable | all |
  most | few | none | reallyall

Many of our modules are not cross-platform, yet we expect all of
our developers to test with --enable-maintainer-mode.  That smells
like a brain fart to me.

Roy

Re: Time for Apache httpd 2.4.3 ??

[Roy T. Fielding]

I don't know of any issues with 308, and Julian generally knows what
he is doing with regard to HTTP.  In general, we should consider
the IANA registry to be authoritative unless it is a known bug,
which means we should support everything in

http://www.iana.org/assignments/http-status-codes/http-status-codes.xml

Roy


On Jul 11, 2012, at 5:46 AM, Jim Jagielski wrote:

 Roy, as Main Dude for compliance, any issue with getting
 
   https://issues.apache.org/bugzilla/show_bug.cgi?id=53292
 
 into trunk (and 2.4.x)?
 
 On Jul 11, 2012, at 3:51 AM, Julian Reschke wrote:
 
 On 2012-07-11 05:09, Jim Jagielski wrote:
 Just how supported and standard is this? Chrome seems to use it
 for something else:
 
 http://code.google.com/p/gears/wiki/ResumableHttpRequestsProposal
 
 I was told by Google that they are phasing this out (this may already have 
 happened), and then will fix 
 http://code.google.com/p/chromium/issues/detail?id=109012.
 
 Not that it *is* implemented in Firefox 14, shipping next week: 
 https://bugzilla.mozilla.org/show_bug.cgi?id=714302.
 
 Best regards, Julian
 
 

Re: TRACE still enabled by default

[Roy T. Fielding]

On Mar 21, 2012, at 5:33 AM, Jim Jagielski wrote:
 On Mar 20, 2012, at 3:04 PM, Stefan Fritsch wrote:
 
 On Saturday 17 March 2012, Roy T. Fielding wrote:
 We still enable TRACE by default.
 
 
 
 Is this useful enough to justify making every other poor sap with
 a security scanner have to manually turn it off?
 
 Yes.
 
 I'm hoping 2.4.x is early enough in life where flipping this
 wouldn't be too astonishing.
 
 I don't change protocols based on fool security researchers and
 their failure to correctly direct security reports.  TRACE is not
 a vulnerability.
 
 That doesn't mean that it's a good idea to have it on by default. I 
 can't remember ever having needed it for debugging. While it may 
 actually be useful in reverse-proxy situations, it is usually 
 necessary to disable it there because one does not want to leak 
 internal information like the private IPs from X-Forwarded-For.
 
 It can also compound security issues in webapps. In general, one can 
 say that it increases the attack surface a web server presents to the 
 internet. I think it is a good idea to make it default to off.
 
 
 I agree w/ Roy that having our defaults be non-compliant is
 bad, and actions which seem to imply that the idea that TRACE
 is a vulnerability is valid should be avoided.

TRACE won't work at all if the most popular end-point doesn't support it.

If folks want to protect clients (including gateways) against their own
stupidity regarding what they choose to send in a TRACE request, then
do so by selectively removing some lines from the response and I will
try to update the standard accordingly.

Turning it off by default is not an option.  I will veto that.

Roy

Re: TRACE still enabled by default

[Roy T. Fielding]

On Mar 16, 2012, at 7:18 AM, Eric Covener wrote:

 We still enable TRACE by default.
 
 Is this useful enough to justify making every other poor sap with a
 security scanner have to manually turn it off?

Yes.

 I'm hoping 2.4.x is early enough in life where flipping this wouldn't
 be too astonishing.

I don't change protocols based on fool security researchers and their
failure to correctly direct security reports.  TRACE is not a vulnerability.

Roy

Re: svn commit: r1301867 - /httpd/httpd/trunk/docs/conf/mime.types

[Roy T. Fielding]

On Mar 17, 2012, at 1:12 AM, William A. Rowe Jr. wrote:

 On 3/17/2012 12:59 AM, field...@apache.org wrote:
 Author: fielding
 Date: Sat Mar 17 05:59:06 2012
 New Revision: 1301867
 
 URL: http://svn.apache.org/viewvc?rev=1301867view=rev
 Log:
 new IANA media types as of 2012 Mar 16 18:55 PDT
 
 I'd encourage a backport to branches/2.4.x, and branches/2.4.x alone.

Done for all branches, since it is applicable to old distros even if we
don't do any new releases.  I just had to remember how to do the svn merge
command (again).

Roy

Fwd: Working Group Last Call: httpbis p4 / p5 / p6 / p7

[Roy T. Fielding]

In case some folks want to check our compliance and fix the spec
(or the code) if we aren't ... now is the time.

Roy

Begin forwarded message:

 Resent-From: ietf-http...@w3.org
 From: Mark Nottingham m...@mnot.net
 Subject: Working Group Last Call: httpbis p4 / p5 / p6 / p7
 Date: March 15, 2012 12:20:50 AM PDT
 To: HTTP Working Group ietf-http...@w3.org
 
 After discussion with the editors and ADs, I believe the following documents 
 are ready for Working Group Last Call (WGLC):
 
 HTTP/1.1, part 4: Conditional Requests
 http://tools.ietf.org/html/draft-ietf-httpbis-p4-conditional-19
 
 HTTP/1.1, part 5: Range Requests and Partial Responses
 http://tools.ietf.org/html/draft-ietf-httpbis-p5-range-19
 
 HTTP/1.1, part 6: Caching
 http://tools.ietf.org/html/draft-ietf-httpbis-p6-cache-19
 
 HTTP/1.1, part 7: Authentication
 http://tools.ietf.org/html/draft-ietf-httpbis-p7-auth-19
 
 To encourage implementers to review and provide feedback on the documents, 
 we're doing a four-week WGLC, ending on April 12, 2012. However, early 
 reviews are appreciated, as that will give us the opportunity to discuss 
 potential changes at our Paris meeting.
 
 *** Providing Feedback
 
 Your input should be sent to this mailing list, clearly marked with WGLC 
 and the appropriate part. E.g., with Subject lines such as:
 
 Subject: WGLC review of p4-conditional
 Subject: WGLC issue: foo in p5
 
 Issues that you believe to be editorial in nature (e.g., typos, suggested 
 re-phrasing) can be grouped together in a single e-mail. Substantive issues 
 (what we call design issues) that may need discussion should be sent one 
 per e-mail, with a descriptive subject.
 
 If you disagree with the resolution of a previously discussed issue, you're 
 encouraged to note that at this time.
 
 *** What's Next
 
 The Working Group will discuss these issues, re-issuing drafts as necessary. 
 Tickets raised on these drafts will have a severity of In WG Last Call , 
 and once they are disposed of, we'll see if there's consensus on going to 
 IETF Last Call on them (as Chair, I'll judge this for each document except 
 p6, where I'll defer to Mark Baker, the document shepherd, since I've edited 
 a substantial part of that specification).
 
 p1, p2 and p3 should join them in WGLC at or shortly after our Paris meeting. 
 We anticipate sending all of the documents to IETF LC together.
 
 Thanks to the editors for their hard work in getting to this point.
 
 --
 Mark Nottingham
 http://www.mnot.net/



Begin forwarded message:

 Resent-From: ietf-http...@w3.org
 From: Julian Reschke julian.resc...@gmx.de
 Subject: Re: Working Group Last Call: httpbis p4 / p5 / p6 / p7
 Date: March 15, 2012 1:23:00 AM PDT
 To: HTTP Working Group ietf-http...@w3.org
 
 Hi there,
 
 please also note that we have HTML versions at
 
 https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/19/p4-conditional.html
 
 https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/19/p5-range.html
 
 https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/19/p6-cache.html
 
 https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/19/p7-auth.html
 
 which come with feedback links that should make it really simple to provide 
 quick comments.
 
 Also, we'll continue to work on the drafts as comments come in; you can 
 always find the latest and greatest at
 
 https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p4-conditional.html
 
 https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p5-range.html
 
 https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p6-cache.html
 
 https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p7-auth.html
 
 For trivial corrections, if you suspect that somebody else might already have 
 reported the bug, you may want to check here whether it's fixed already.
 
 Looking forward to review comments.
 
 Julian

Re: Technical reasons for -1 votes (?)

[Roy T. Fielding]

On Mar 1, 2012, at 9:20 AM, William A. Rowe Jr. wrote:

 On 2/29/2012 6:25 PM, Roy T. Fielding wrote:
 On Feb 29, 2012, at 9:42 AM, William A. Rowe Jr. wrote:
 
 Let's take Roy's position on the attached vote discussion, it's relevant.
 These new modules are certainly additions/deletions to httpd.
 
 Yes, but they are modules.  Hence, their mere existence in our tree
 is not a technical reason to exclude them.  We have a modular architecture
 so that people who don't want a module don't have to build it.  In fact,
 it was exactly this type of argument in 1995 that caused rst to focus
 on creating a modular architecture.
 
 Ok, so to clarify, you are reversing your answer to Joe?  The LDAP changes
 were a change to a module.  Not an httpd-wide API, just a module.  So you
 are saying that your comment to Joe, The API was moved to *this* project
 and all of the names were changed.  It is, effectively, a new public API
 for this project. was a misinformed statement (as it moved the ldap code
 into mod_ldap and not into the core API).  That veto was unjustified?
 
 Or rather, had we not exported a single symbol, then the veto was unjustified?

I don't remember the details, but it was presented by you as a new API.
It was described by you as the addition of new LDAP libraries to
httpd.  And it was most certainly a change to the existing code in
mod_ldap.

Had it been presented as a new module with a new name and optional
configuration, I would have said new modules only need one committer
to maintain and a majority vote to decide if there is any objections.

Quite frankly, I don't care either way for either change.  What I care
about is continual use of procedural bullshit to justify interference
with other people scratching their own itches.  If you don't like the
module, then don't use it.  If it has security holes or licensing issues,
then those are grounds for a veto.  If you don't like Grahams's attitude,
that's too frigging bad -- find another way to vent your frustration or
try to resolve it via the PMC, but don't mix it with a technical vote.

 If there were dependencies or license conditions brought in that somehow
 harmed the server without the module being active, then that would be a
 technical objection.
 
 There was no such case for the ldap change, so you are saying that specific
 veto was unjustified, right?  Anyways, all of this is distraction, let's
 drill in further to your analysis;
 
 Traditionally, we have allowed any module that has at least one willing
 volunteer committer to maintain it.  And I agree with Jim, none of the
 subprojects have been as successful as just placing the code in the
 main tree.
 
 Allowed it by... voting to accept it, right?  I accept that all of my
 colleagues have different rationals for their +/- votes.  Right now, just
 about 3% of the committee are willing to entertain these module additions
 to httpd trunk and 1.5% are against adding these to trunk.  I chalk this
 up to burnout, 2.4 was very complex to release and people know their +1
 votes are commitments and alter the httpd tarball for better or worse.

If a vote is required (i.e., anyone objects), then it is a majority vote
with minimal quorum (three +1s), as usual.

 If there was a vote to add this module to core, that would be lovely, we
 could vote on it.  Sadly, minfrin offered no such vote, and whatever your
 conclusion on the 'better course' - it was not voted on, and you hadn't
 expressed a vote to adopt it in either case.

I submit that there was confusion and the right thing to do would be to
ask politely for the confusion to be resolved by an actual vote.
It would help if you were not being an ass about it and randomly
accusing people of nefarious behavior just because they don't give
a damn about this particular difference of opinion.

And it isn't Graham's responsibility to run the vote.

 I have no idea why mod_fcgi is in a subproject.  mod_ftp is there
 because it isn't an HTTP server.  mod_aspdotnet had all sorts of
 licensing issues that I never quite figured out.
 
 Nonsequitor, if we want to launch into a discussion of subprojects vs.
 modules, that's great.  That wasn't the conversation minfrin started,
 and that's why I have a VETO on the commit.  Correct the mis-execution
 by conforming to the vote he held, or correct the vote instead.  I've
 VETOED a commit which is something other than what was voted on because
 the project did not consent to his action.

You are confused.  You don't need to veto a non-event -- just conduct
the vote properly and be done.

 If there are two other committers who will vote with Jim for this
 project to accept one or more of these modules into trunk (rather than
 subproject), and someone will finish the ip-clearance, I'm great with
 that.  If none of that happens I will revert this mess.

Fine with me, if you stop blathering about it and conduct the vote
properly.  You know how to conduct a vote.

 I see no reason not to commit mod_firehose, though I

Re: Technical reasons for -1 votes (?)

[Roy T. Fielding]

On Mar 1, 2012, at 12:28 PM, William A. Rowe Jr. wrote:

 On 3/1/2012 1:58 PM, Jim Jagielski wrote:
 
 On Mar 1, 2012, at 1:25 PM, William A. Rowe Jr. wrote:
 
 On 2/29/2012 6:25 PM, Roy T. Fielding wrote:
 
 Yes, but they are modules.  Hence, their mere existence in our tree
 is not a technical reason to exclude them.  We have a modular architecture
 so that people who don't want a module don't have to build it.
 
 Which explains mod_macro how, exactly?
 
 At this point, my head exploded Scanners-like...
 
 Funny, mine had exploded at the dis-congruity of;
 
 mod_ftp is there because it isn't an HTTP server.
 
 which is apropos of what?

It is a different product.  If it were changed so that it could
be meaningfully documented as part of the HTTP server, then I'd
suggest it be moved to trunk too.

  This isn't an ajp or scgi or fastcgi server
 either, it's an http server, which means we should pull out those interfaces?

Those are all gateway interfaces *used* by an HTTP server to
communicate with a backend.  If we had a better way to ship
modular code, like CPAN or gem, they wouldn't be part of our
core product.  But we don't, so they are (or should be).

Roy

Re: Technical reasons for -1 votes (?)

[Roy T. Fielding]

On Feb 29, 2012, at 9:42 AM, William A. Rowe Jr. wrote:

 On 2/29/2012 8:59 AM, André Malo wrote:
 On Wednesday 29 February 2012 04:11:35 William A. Rowe Jr. wrote:
 
 I withdraw this vote, reverting my position to -1, until collaboration and
 respect for options and insights of fellow committers as well as project
 decisions and votes can be consistently demonstrated.
 
 I always thought, you'd have to provide technical reasons for -1 votes (?).
 
 Let's take Roy's position on the attached vote discussion, it's relevant.
 These new modules are certainly additions/deletions to httpd.

Yes, but they are modules.  Hence, their mere existence in our tree
is not a technical reason to exclude them.  We have a modular architecture
so that people who don't want a module don't have to build it.  In fact,
it was exactly this type of argument in 1995 that caused rst to focus
on creating a modular architecture.

If there were dependencies or license conditions brought in that somehow
harmed the server without the module being active, then that would be a
technical objection.

Traditionally, we have allowed any module that has at least one willing
volunteer committer to maintain it.  And I agree with Jim, none of the
subprojects have been as successful as just placing the code in the
main tree.

I have no idea why mod_fcgi is in a subproject.  mod_ftp is there
because it isn't an HTTP server.  mod_aspdotnet had all sorts of
licensing issues that I never quite figured out.

I see no reason not to commit mod_firehose, though I haven't had a
chance to look at the code myself.  Nor am I willing to respect a
veto war based on the impact of past vetos.

Now, if you'll excuse me, I have at least two other walls to bang
my head on today ...

Roy

Re: Effective IP address / real IP address

[Roy T. Fielding]

On Dec 13, 2011, at 5:33 PM, Graham Leggett wrote:
 On 14 Dec 2011, at 12:50 AM, Graham Leggett wrote:
 On 12 Dec 2011, at 11:25 PM, William A. Rowe Jr. wrote:
 
 I have a frustrating update, which we need to take into consideration for
 the whole remote_ip-related resolution.  From the httpd-ng workgroup...
 
 This makes sense, we're an HTTP server, lets stick to RFC related terms.
 
 Done in r 1214022.

Huh, I was wondring what Bill was talking about ... I couldn't remember any
discussion that had reversed the meaning, and now I know why.

The IP address received by the server interface when we are acting as a
reverse proxy is not necessarily the IP address of the user agent.  It could
just as easily be the IP address of an ISP proxy, a corporate firewall,
or a dozen other client-side intermediaries that are not the user agent.
Hence, it is just a client.

I can hear Graham screaming now.

No worries -- I don't care what the variable is called in request_rec
as long as the intent is clear enough, so feel free to leave it as is.

Roy

trunk makefile echoing modules

[Roy T. Fielding]

I am getting the following on OS X Lion:

Installing configuration files
/bin/sh: ,authn_file,: command not found
/bin/sh: ,authn_dbm,: command not found
/bin/sh: ,authn_anon,: command not found
/bin/sh: ,authn_dbd,: command not found
/bin/sh: ,authn_socache,: command not found
/bin/sh: ,authn_core,: command not found
/bin/sh: ,authz_host,: command not found
/bin/sh: ,authz_groupfile,: command not found
/bin/sh: ,authz_user,: command not found
/bin/sh: ,authz_dbm,: command not found
...

I am guessing it has something to do with 


r1200064 | sf | 2011-11-09 16:41:20 -0800 (Wed, 09 Nov 2011) | 2 lines
Changed paths:
   M /httpd/httpd/trunk/Makefile.in
   M /httpd/httpd/trunk/acinclude.m4

Fix selection of loaded modules on AIX because its grep doesn't understand \ 
and \


Index: Makefile.in
===
--- Makefile.in (revision 1200063)
+++ Makefile.in (revision 1200064)
@@ -60,7 +60,7 @@
 $$i; \
for j in $(DSO_MODULES) ^EOL^; do \
if test $$j != ^EOL^; then \
-   if echo 
$(ENABLED_DSO_MODULES)|$(EGREP) \$$j\  /dev/null ; then \
+   if echo 
,$(ENABLED_DSO_MODULES),|$(EGREP) ,$$j,  /dev/null ; then \
loading_disabled=; \
else \
loading_disabled=#; \
Index: acinclude.m4
===
--- acinclude.m4(revision 1200063)
+++ acinclude.m4(revision 1200064)
@@ -257,7 +257,7 @@
 # add default MPM to LoadModule list
 if test $1 = $default_mpm; then
 DSO_MODULES=$DSO_MODULES mpm_$1
-ENABLED_DSO_MODULES=$ENABLED_DSO_MODULES mpm_$1
+ENABLED_DSO_MODULES=${ENABLED_DSO_MODULES},mpm_$1
 fi
 fi
 $4
@@ -360,7 +360,7 @@
   shared=yes
   DSO_MODULES=$DSO_MODULES $1
   if test $_apmod_required = yes ; then
-ENABLED_DSO_MODULES=$ENABLED_DSO_MODULES $1
+ENABLED_DSO_MODULES=${ENABLED_DSO_MODULES},$1
   fi
   ;;
 esac

Re: Who's at the Hackathon?

[Roy T. Fielding]

On Nov 7, 2011, at 9:22 AM, Sander Temme wrote:

 Folks, 
 
 The httpd table now has:
 
 Jeff Trawick
 Jean-Frederic Leclere
 Stefan Fritsch
 Rainer Jung
 and myself
 
 Who else is at the conference?  Anybody joining tomorrow?

I'll be around tomorrow (at the board meeting today).

…Roy

Re: svn commit: r1187992 - /httpd/httpd/trunk/modules/filters/mod_filter.c

[Roy T. Fielding]

On Oct 23, 2011, at 3:19 PM, s...@apache.org wrote:

 else if (r-content_type) {
 const char **type = provider-types;
 AP_DEBUG_ASSERT(type != NULL);
 while (*type) {
 -if (strcmp(*type, r-content_type) == 0) {
 +/* Handle 'content-type;charset=...' correctly */
 +size_t len = strcspn(r-content_type, ; \t);
 +if (strlen(*type) == len
 + strncmp(*type, r-content_type, len) == 0) {
 match = 1;
 break;
 }
 type++;
 }

Wouldn't this be faster as

else if (r-content_type) {
+   /* Handle 'content-type;charset=...' correctly */
+   size_t len = strcspn(r-content_type, ; \t);
const char **type = provider-types;
AP_DEBUG_ASSERT(type != NULL);
while (*type) {
+if ((strncmp(*type, r-content_type, len) == 0) 
+(*type[len] == '\0')) {
match = 1;
break;
}
type++;
}

or am I just trying to outwit the compiler?

Roy

Re: svn commit: r1187986 - in /httpd/httpd/trunk/docs/manual: custom-error.xml mod/core.xml

[Roy T . Fielding]

On Oct 23, 2011, at 3:09 PM, s...@apache.org wrote:

 --- httpd/httpd/trunk/docs/manual/mod/core.xml (original)
 +++ httpd/httpd/trunk/docs/manual/mod/core.xml Sun Oct 23 22:09:34 2011
 @@ -1165,6 +1165,7 @@ in case of an error/description
  ErrorDocument 404 /cgi-bin/bad_urls.plbr /
  ErrorDocument 401 /subscription_info.htmlbr /
  ErrorDocument 403 Sorry can't allow you access today
 +  ErrorDocument 403 Forbidden!
/example
 
pAdditionally, the special value codedefault/code can be used

I don't think that change was intended, right?  Looks like an error test.

Roy

Re: svn commit: r1172686 - in /httpd/httpd/trunk: ./ include/ modules/cache/ modules/examples/ modules/proxy/ modules/ssl/ server/ server/mpm/event/ server/mpm/worker/

[Roy T. Fielding]

I am pretty sure that this kind of change has been vetoed numerous
times in the past.  What has changed?

Roy

On Sep 19, 2011, at 9:25 AM, s...@apache.org wrote:

 Author: sf
 Date: Mon Sep 19 16:25:42 2011
 New Revision: 1172686
 
 URL: http://svn.apache.org/viewvc?rev=1172686view=rev
 Log:
 Add wrappers for malloc, calloc, realloc that check for out of memory
 situations.  Use them in most places where malloc, and friends are used.
 This results in clean error messages in an out of memory situation instead of
 segfaulting or silently malfunctioning. In some places, it just allows to
 remove some logging code.
 
 PR 51568, PR 51569, PR 51571.
 
 Modified:
httpd/httpd/trunk/CHANGES
httpd/httpd/trunk/include/ap_config.h
httpd/httpd/trunk/include/ap_mmn.h
httpd/httpd/trunk/include/httpd.h
httpd/httpd/trunk/modules/cache/cache_cache.c
httpd/httpd/trunk/modules/cache/cache_hash.c
httpd/httpd/trunk/modules/cache/cache_pqueue.c
httpd/httpd/trunk/modules/cache/mod_socache_dbm.c
httpd/httpd/trunk/modules/examples/mod_case_filter_in.c
httpd/httpd/trunk/modules/proxy/proxy_util.c
httpd/httpd/trunk/modules/ssl/ssl_util.c
httpd/httpd/trunk/server/config.c
httpd/httpd/trunk/server/main.c
httpd/httpd/trunk/server/mpm/event/event.c
httpd/httpd/trunk/server/mpm/worker/worker.c
httpd/httpd/trunk/server/mpm_unix.c
httpd/httpd/trunk/server/scoreboard.c
httpd/httpd/trunk/server/util.c
 
 Modified: httpd/httpd/trunk/CHANGES
 URL: 
 http://svn.apache.org/viewvc/httpd/httpd/trunk/CHANGES?rev=1172686r1=1172685r2=1172686view=diff
 ==
 --- httpd/httpd/trunk/CHANGES [utf-8] (original)
 +++ httpd/httpd/trunk/CHANGES [utf-8] Mon Sep 19 16:25:42 2011
 @@ -12,6 +12,10 @@ Changes with Apache 2.3.15
  PR 51714. [Stefan Fritsch, Jim Jagielski, Ruediger Pluem, Eric Covener,
  lowprio20 gmail.com]
 
 +  *) Add wrappers for malloc, calloc, realloc that check for out of memory
 + situations and use them in many places. PR 51568, PR 51569, PR 51571.
 + [Stefan Fritsch]
 +
   *) Fix cross-compilation of mod_cgi/mod_cgid when APR_HAVE_STRUCT_RLIMIT is 
  false but RLIMIT_* are defined.  PR51371. [Eric Covener]
 
 
 Modified: httpd/httpd/trunk/include/ap_config.h
 URL: 
 http://svn.apache.org/viewvc/httpd/httpd/trunk/include/ap_config.h?rev=1172686r1=1172685r2=1172686view=diff
 ==
 --- httpd/httpd/trunk/include/ap_config.h (original)
 +++ httpd/httpd/trunk/include/ap_config.h Mon Sep 19 16:25:42 2011
 @@ -182,4 +182,12 @@
 #define ap_func_attr_sentinel
 #endif
 
 +#if ( defined(__GNUC__) \
 +  (__GNUC__ = 4 || ( __GNUC__ == 3  __GNUC_MINOR__ = 4))) \
 +|| __has_attribute(warn_unused_result)
 +#define ap_func_attr_warn_unused_result   __attribute__((warn_unused_result))
 +#else
 +#define ap_func_attr_warn_unused_result
 +#endif
 +
 #endif /* AP_CONFIG_H */
 
 Modified: httpd/httpd/trunk/include/ap_mmn.h
 URL: 
 http://svn.apache.org/viewvc/httpd/httpd/trunk/include/ap_mmn.h?rev=1172686r1=1172685r2=1172686view=diff
 ==
 --- httpd/httpd/trunk/include/ap_mmn.h (original)
 +++ httpd/httpd/trunk/include/ap_mmn.h Mon Sep 19 16:25:42 2011
 @@ -352,6 +352,8 @@
  * 20110724.5 (2.3.15-dev) add ap_set_accept_ranges()
  * 20110724.6 (2.3.15-dev) add max_overlaps and max_reversals to 
 core_dir_config
  * 20110724.7 (2.3.15-dev) add ap_random_insecure_bytes(), ap_random_pick()
 + * 20110724.8 (2.3.15-dev) add ap_abort_on_oom(), ap_malloc(), ap_calloc(),
 + * ap_realloc()
  */
 
 #define MODULE_MAGIC_COOKIE 0x41503234UL /* AP24 */
 @@ -359,7 +361,7 @@
 #ifndef MODULE_MAGIC_NUMBER_MAJOR
 #define MODULE_MAGIC_NUMBER_MAJOR 20110724
 #endif
 -#define MODULE_MAGIC_NUMBER_MINOR 7/* 0...n */
 +#define MODULE_MAGIC_NUMBER_MINOR 8/* 0...n */
 
 /**
  * Determine if the server's current MODULE_MAGIC_NUMBER is at least a
 
 Modified: httpd/httpd/trunk/include/httpd.h
 URL: 
 http://svn.apache.org/viewvc/httpd/httpd/trunk/include/httpd.h?rev=1172686r1=1172685r2=1172686view=diff
 ==
 --- httpd/httpd/trunk/include/httpd.h (original)
 +++ httpd/httpd/trunk/include/httpd.h Mon Sep 19 16:25:42 2011
 @@ -2086,6 +2086,38 @@ AP_DECLARE(void) ap_random_insecure_byte
  */
 AP_DECLARE(apr_uint32_t) ap_random_pick(apr_uint32_t min, apr_uint32_t max);
 
 +/**
 + * Abort with a error message signifying out of memory
 + */
 +AP_DECLARE(void) ap_abort_on_oom(void) __attribute__((noreturn));
 +
 +/**
 + * Wrapper for malloc() that calls ap_abort_on_oom() if out of memory
 + * @param size size of the memory block
 + * @return pointer to the allocated memory
 + * @note ap_malloc may be 

Re: svn commit: r1163833 - /httpd/httpd/trunk/modules/http/byterange_filter.c

[Roy T. Fielding]

On Sep 1, 2011, at 1:11 AM, Tim Bannister wrote:
 On Wed, Aug 31, 2011 at 6:28 PM, Roy T. Fielding wrote:
 On Aug 31, 2011, at 6:10 PM, William A. Rowe Jr. wrote:
 The presumption here is that the client requests bytes=0- to begin the 
 transmission, and provided it sees a 206, restarting somewhere in the 
 stream results in aborting the connection and streaming bytes=n- from the 
 restart point.  Further testing should determine if this was the broken 
 assumption.
 Do we send the Accept-Ranges header field?
  http://tools.ietf.org/html/rfc2616#page-105
 
 Apache httpd 2.2.9 is sending this header in the Debian bug report at 
 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=639825

Then the client is broken and they also break intermediary
caches (most of which don't cache 206 responses) by performing
this stunt.  We could add a version-specific browsermatch to do
the 206, but I'd prefer to just tell the client developers to
fix their code.  

Roy

Re: svn commit: r1163833 - /httpd/httpd/trunk/modules/http/byterange_filter.c

[Roy T. Fielding]

On Aug 31, 2011, at 2:37 PM, s...@apache.org wrote:

 Author: sf
 Date: Wed Aug 31 21:37:38 2011
 New Revision: 1163833
 
 URL: http://svn.apache.org/viewvc?rev=1163833view=rev
 Log:
 Send a 206 response for a Range: bytes=0- request, even if 200 would be more
 efficient.

200 is a better response for caches as well.  Why do we want to support this?

Roy

Re: svn commit: r1163833 - /httpd/httpd/trunk/modules/http/byterange_filter.c

[Roy T. Fielding]

On Aug 31, 2011, at 6:10 PM, William A. Rowe Jr. wrote:

 On 8/31/2011 6:06 PM, Stefan Fritsch wrote:
 On Wednesday 31 August 2011, Roy T. Fielding wrote:
 Author: sf
 Date: Wed Aug 31 21:37:38 2011
 New Revision: 1163833
 
 URL: http://svn.apache.org/viewvc?rev=1163833view=rev
 Log:
 Send a 206 response for a Range: bytes=0- request, even if 200
 would be more efficient.
 
 200 is a better response for caches as well.  Why do we want to
 support this?
 
 As explained to me by Stefan on irc...
 
 There seems to be at least one streaming client that gets confused by 
 this behaviour.
 
 But maybe my commit was premature. If the change does not actually fix 
 the problem, I will revert.
 
 The presumption here is that the client requests bytes=0- to begin the
 transmission, and provided it sees a 206, restarting somewhere in the
 stream results in aborting the connection and streaming bytes=n- from
 the restart point.  Further testing should determine if this was the
 broken assumption.

Do we send the Accept-Ranges header field?

  http://tools.ietf.org/html/rfc2616#page-105

Roy

Re: Fixing Ranges

[Roy T. Fielding]

On Aug 25, 2011, at 2:02 PM, Jim Jagielski wrote:

 Using stef's byterange4 test, I'm seeing:
 
 apr_brigade_length (bb=0x7feb00a23200, read_all=1, length=0x7fff6e03e8b0) at 
 apr_brigade.c:201
 201   if (bkt-length == (apr_size_t)(-1)) {

apr_size_t is unsigned.  That's borked.

Roy

Re: DoS with mod_deflate range requests

[Roy T. Fielding]

On Aug 24, 2011, at 8:35 AM, Tim Bannister wrote:

 On Tue, Aug 23, 2011, Roy T. Fielding wrote:
 And the spec says ...
   When a client requests multiple ranges in one request, the
   server SHOULD return them in the order that they appeared in the
   request.
 My suggestion is to reject any request with overlapping ranges or more than 
 five ranges with a 416, and to send 200 for any request with 4-5 ranges.  
 There is simply no need to support random access in HTTP.
 
 Deshpande  Zeng in http://dx.doi.org/10.1145/500141.500197 describe a method 
 for streaming JPEG 2000 documents over HTTP, using many more than 5 ranges 
 in a single request.
 A client that knows about any server-side limit could make multiple requests 
 each with a small number of ranges, but discovering that limit will add 
 latency and take more code.

I have no interest in supporting such a use case over HTTP.
Consider how stupid it is to request ranges like their example

Range: bytes=120-168,175-200,205-300,345-346,400-500,555-666,
   667-800,900-1000,2500-2567,2890-3056,5678-9000,
   1-12004,12050-12060,15600-15605,17000-17001,
   17005-17010,17050-17060,17800-17905,2-20005

keeping in mind that between each one of those ranges will be
a multipart boundary of approximately 80 bytes!  Hence, any
range request that contains gaps of less than 80 bytes should
be considered a denial of service, or at least an idiot programmer
that deserves to be slapped by Apache.

To be clear, I am more than willing to rewrite the part on
Ranges such that the above is explicitly forbidden in HTTP.
I am not sure what the WG would agree to, but I am quite certain
that part of the reason we have an Apache server is to protect
the Internet from idiotic ideas like the above.

Roy

Re: DoS with mod_deflate range requests

[Roy T. Fielding]

On Aug 24, 2011, at 8:55 AM, Plüm, Rüdiger, VF-Group wrote:
 Hm. If I got it right what Roy says above about the spec sorting and merging 
 is
 not an option as we need to stick to the order and number of ranges the client
 requested. But we can deny overlapping with a 416.

We should implement whatever is needed to make the service
secure from this denial of service.  If that means changing the
spec, then so be it.

 Or we do a 416 as well if merging would change something.

Or 200 if folks are squeamish about making the developer feel
the pain.

Roy

Re: DoS with mod_deflate range requests

[Roy T. Fielding]

On Aug 24, 2011, at 1:56 PM, Roy T. Fielding wrote:
 To be clear, I am more than willing to rewrite the part on
 Ranges such that the above is explicitly forbidden in HTTP.
 I am not sure what the WG would agree to, but I am quite certain
 that part of the reason we have an Apache server is to protect
 the Internet from idiotic ideas like the above.

http://trac.tools.ietf.org/wg/httpbis/trac/ticket/311

Roy

Re: DoS with mod_deflate range requests

[Roy T. Fielding]

On Aug 24, 2011, at 4:39 PM, William A. Rowe Jr. wrote:

 On 8/24/2011 4:54 PM, Roy T. Fielding wrote:
 On Aug 24, 2011, at 1:56 PM, Roy T. Fielding wrote:
 To be clear, I am more than willing to rewrite the part on
 Ranges such that the above is explicitly forbidden in HTTP.
 I am not sure what the WG would agree to, but I am quite certain
 that part of the reason we have an Apache server is to protect
 the Internet from idiotic ideas like the above.
 
 http://trac.tools.ietf.org/wg/httpbis/trac/ticket/311
 
 Excellent, thanks.  Just curious, isn't this clarification outside of
 the remit of 2616bis?

Security repairs are never out of scope.

Roy

Re: DoS with mod_deflate range requests

[Roy T. Fielding]

On Aug 23, 2011, at 2:34 PM, William A. Rowe Jr. wrote:

 On 8/23/2011 4:00 PM, Greg Ames wrote:
 
 On Tue, Aug 23, 2011 at 3:32 PM, William A. Rowe Jr. wrote:
 
I suggest we should be parsing and reassembling the list before we
start the bucket logic. 
 
I propose we satisfy range requests in the only sensible manner, returning
the ranges in sequence,
 
 yeah, overlapping ranges should be merged up front. That ought to completely 
 fix the issue.
 
 So the only remaining question; are we free to reorder them into sequence?

And the spec says ...

   When a client requests multiple ranges in one request, the
   server SHOULD return them in the order that they appeared in the
   request.

My suggestion is to reject any request with overlapping ranges or more
than five ranges with a 416, and to send 200 for any request with 4-5
ranges.  There is simply no need to support random access in HTTP.

Roy

Re: [vote] mod_ldap

[Roy T. Fielding]

On Jul 12, 2011, at 8:20 AM, Joe Orton wrote:
 On Sun, Jul 10, 2011 at 03:34:10PM -0700, Roy T. Fielding wrote:
 Regardless of anyone else's opinion, the addition or deletion of a
 new API to our product is a technical change that can be vetoed.
 Likewise, the API being an incomplete abstraction that isn't
 needed in httpd is a valid technical reason to veto it even if
 it had once been in apr-util.
 
 Other than the convoluted history of this particular argument,
 I don't see any reason for further frustration.  Revert the commit.
 
 Yet again: if the objection is to extending the exported mod_ldap API, 
 that objection can be resolved without wholesale revert; most of the 
 stuff added does not need to be exposed in the API, it was just done for 
 consistency.

The objection can only be resolved by convincing the person who has
objected to change *their* opinion or by removing the thing being
objected to.  The time for convincing has expired -- let's move on.

 I do not understand using incomplete abstraction as motivation for 
 veto, because mod_ldap's API was already an incomplete abstraction.  If 
 this was OK before, it is not reason for veto now.

The API was moved to *this* project and all of the names were changed.
It is, effectively, a new public API for this project.

 We are doomed to revisit this argument time and again if we avoid 
 actually discussing the technical issues.

The whole point of having a set of voting guidelines is to avoid
having a discussion about process which is colored by the particular
issue being discussed and to avoid having discussions about
technical issues which become poisoned because of perceived unfairness
in the way people's opinions are being respected.

Remove the process issue first and then the technical issues can be
resolved one at a time as technical issues.

Roy

Re: [vote] mod_ldap

[Roy T. Fielding]

Regardless of anyone else's opinion, the addition or deletion of a
new API to our product is a technical change that can be vetoed.
Likewise, the API being an incomplete abstraction that isn't
needed in httpd is a valid technical reason to veto it even if
it had once been in apr-util.

Other than the convoluted history of this particular argument,
I don't see any reason for further frustration.  Revert the commit.

Whatever it is we need to add to httpd in order to support
compatibility with 2.2.x for some future version of APR,
we can do so as specific fixes that solve a specific problem.
If someone gets a brainstorm and comes up with a perfect LDAP API
in the future, then the discussion can be revisited at that time.

Roy

Re: MPM-Event, renaming MaxClients, etc.

[Roy T. Fielding]

On Jun 20, 2011, at 12:01 PM, Stefan Fritsch wrote:
 On Monday 20 June 2011, William A. Rowe Jr. wrote:
 On 6/20/2011 9:07 AM, Greg Ames wrote:
 On Sun, Jun 19, 2011 at 8:49 AM, Stefan Fritsch s...@sfritsch.de
 mailto:s...@sfritsch.de
 
 wrote:
Speaking about config options, I think that MaxClients should
be renamed to MaxWorkers, because it limits the number of
worker threads, not the number of clients. As with the
MaxRequestsPerChild - MaxConnectionsPerChild rename, we
would still accept the old name with a warning. Thoughts?
 
 +1.  That's more accurate for sure, and more important when you
 loose the 1:1 thread:connection ratio.
 
 Can we call this MaxRequestWorkers, now that we have different
 sorts of workers living in the same process?
 
 Good point. Committed as r1137744

That kind of last-minute change is going to kill people trying to
upgrade from 2.2 to 2.4 with a shared config.  We should make
MaxRequestWorkers an alias for MaxClients in a released 2.2.x

Roy

Re: MPM-Event, renaming MaxClients, etc.

[Roy T. Fielding]

On Jun 20, 2011, at 2:48 PM, William A. Rowe Jr. wrote:
 On 6/20/2011 4:36 PM, Stefan Fritsch wrote:
 On Monday 20 June 2011, Roy T. Fielding wrote:
 On Jun 20, 2011, at 12:01 PM, Stefan Fritsch wrote:
 On Monday 20 June 2011, William A. Rowe Jr. wrote:
 On 6/20/2011 9:07 AM, Greg Ames wrote:
 On Sun, Jun 19, 2011 at 8:49 AM, Stefan Fritsch s...@sfritsch.de
 mailto:s...@sfritsch.de
 
 wrote:
   Speaking about config options, I think that MaxClients
   should be renamed to MaxWorkers, because it limits the
   number of worker threads, not the number of clients. As
   with the MaxRequestsPerChild - MaxConnectionsPerChild
   rename, we would still accept the old name with a warning.
   Thoughts?
 
 +1.  That's more accurate for sure, and more important when you
 loose the 1:1 thread:connection ratio.
 
 Can we call this MaxRequestWorkers, now that we have different
 sorts of workers living in the same process?
 
 Good point. Committed as r1137744
 
 That kind of last-minute change is going to kill people trying to
 upgrade from 2.2 to 2.4 with a shared config.  We should make
 MaxRequestWorkers an alias for MaxClients in a released 2.2.x
 
 I think you missed that MaxClients still works in 2.4 (albeit with a 
 warning on startup). So I don't think it's such a big deal.
 
 Perhaps we should lower the severity from warning to info?  Not every
 admin needs constant reminders while they are running 2.2 and 2.4 from
 a single config.

My point wasn't the warning, actually, but rather the fact that a config
that uses MaxRequestWorkers (instead of MaxClients) will abort an instance
of httpd 2.2.x.  Hence, a person upgrading to 2.4.x will get tripped up
if they try to do so incrementally with shared config files across
many web servers, unless they happen to notice that this change is
merely cosmetic and they keep MaxClients instead.

For a trivial improvement like this, we should make it easier on admins
by backporting the alias to 2.2.x (even if we do not use it on 2.2.x).

Roy

Re: blocking Upgrade

[Roy T. Fielding]

On Mar 29, 2011, at 11:16 PM, Greg Stein wrote:

 Do you have an internet draft spec for some context here? Is there a
 proposal for HTTP/2.0?

websockets

 I might also argue that a directive is not the right answer here.
 Instead, I'd suggest that modules advertise their ability to consume
 protocols. If an Upgrade arrives, and a relevant module is found, then
 the request is handled. If no such module is found, then the Upgrade
 header is ignored, and nothing happens.˛˛
 
 If a module or CGI sees the Upgrade header, then what is the problem?
 It cannot truly alter the protocol in effect for the connection. That
 seems to be something only possible to handle within the core (to
 change the connection handler).
 
 And back to the AllowUpgrade directive. What the heck should it do
 if something is present *besides* none. It isn't like that can be
 handled. Some code must be written to provide other protocols, and
 *that* code should manage the tokens recognized. Not a directive.

Right, I woke up thinking the same thing -- that the http filter
is the only place that an Upgrade could actually be implemented
because it would have to switch itself off.  What I was trying
to avoid is a script thinking that it can switch without writing
a new protocol filter, or a CGI that deliberately spoofs an upgrade
for some nefarious purpose I haven't thought of yet.

We can just delete the Upgrade header in the http_filter, for now.

Roy

Re: blocking Upgrade

[Roy T. Fielding]

On Mar 30, 2011, at 12:32 PM, Graham Leggett wrote:
 On 30 Mar 2011, at 10:49 AM, Roy T. Fielding wrote:
 On Mar 29, 2011, at 11:16 PM, Greg Stein wrote:
 
 Do you have an internet draft spec for some context here? Is there a
 proposal for HTTP/2.0?
 
 websockets
 
 In theory, over and above natively supporting websockets, it may be useful to 
 teach mod_proxy how to reverse proxy the ws scheme to another websockets 
 server. Would a module like mod_proxy_websockets or mod_proxy_ws make sense?

No, websockets is not designed to work with intermediaries.
There is no standard behavior beyond opening the connection,
so connections through proxies should use CONNECT.

I am not planning to implement websockets -- I just want to
protect the rest of the server from kids playing with toys.

Roy

Re: blocking Upgrade

[Roy T. Fielding]

On Mar 30, 2011, at 4:11 PM, Graham Leggett wrote:
 On 30 Mar 2011, at 3:53 PM, Roy T. Fielding wrote:
 
 No, websockets is not designed to work with intermediaries.
 There is no standard behavior beyond opening the connection,
 so connections through proxies should use CONNECT.
 
 Does a websocket client have a way of knowing it should use CONNECT?

My guess is that it would if it were told to use a proxy for ws.

Keep in mind that when I say proxy, I do not mean to include reverse proxy.
A reverse proxy of websockets is just an implementation of websockets or
a tunnel.  I consider both to be pretty dangerous and better done in
a special-purpose server rather than as a child of Apache httpd.

Roy

blocking Upgrade

[Roy T. Fielding]

Does anyone with a working install want a quick project?

We need to block the Upgrade header field by default.  What this
will require is a new configuration command, like

   AllowUpgrade None | word ...

where word is any protocol name, like HTTP/2.0, waka, websocket, etc.
The config command must only be allowed in rsrc_conf.

We then need a check somewhere in the http filter for an incoming
request header field called Upgrade.  If present and the config option
is set to None (or default), then remove the Upgrade field before it
is seen by the request handler (i.e., before it might be used by some
module or CGI script to send the server down a rat hole).
If the config option is set and not None, then set the Upgrade
header field-value to be the intersection of what was sent by the
client and what is allowed by the config.

Likewise, perform the same filtering on outbound responses.

In other words, only allow a handler to upgrade the connection
if it has been explicitly configured by the main server config
to be an okay thing to do.

Any takers?  If not, I'll give it a try next week when I am back
from the IETF.

Roy

Re: svn commit: r1070179 - in /httpd/httpd/trunk: CHANGES docs/manual/mod/mod_cache.xml modules/cache/cache_storage.c modules/cache/cache_storage.h modules/cache/mod_cache.c modules/cache/mod_cache.h

[Roy T. Fielding]

On Feb 13, 2011, at 5:03 AM, Graham Leggett wrote:

 On 13 Feb 2011, at 9:59 AM, Roy T. Fielding wrote:
 
 URL: http://svn.apache.org/viewvc?rev=1070179view=rev
 Log:
 mod_cache: When a request other than GET or HEAD arrives, we must
 invalidate existing cache entities as per RFC2616 13.10. PR 15868.
 
 Cache entries should not be invalidated unless the response comes
 back as a success (2xx).  Likewise, it only applies to methods with
 known write effects (not including TRACE, OPTIONS, PROPFIND, etc.).
 
 This has already been updated in httpbis p6.
 
 Hmmm...
 
 So is it right to say that POST, PUT and DELETE should invalidate, while all 
 others would be left alone?
 
 Or do we need some kind of mechanism for modules that respond to new methods 
 to declare those methods as needing to be invalidated or not?

Well, I thought I could answer that with a simple link to the
drafts, but they just got changed in the wrong direction.
It should be implemented as invalidating when the response
is 2xx and the method is *not* one of the safe ones
(GET, HEAD, OPTIONS, TRACE, PROPFIND, and maybe REPORT).
That way, a new extension method would invalidate on success.

 Should the cache try and cache any of the other methods, or is cacheability 
 limited to GET only?

It can cache other methods if it knows how to.  PROPFIND is
probably the only other method worth caching, but its key
depends on a bunch of webdav stuff outside the URI.

Roy