Re: svn commit: r1210378 - /httpd/httpd/trunk/server/util_expr_eval.c
Am 13.12.2011 06:57, schrieb Stefan Fritsch: I think r1213567/r1213570 should fix it. Would be nice if you (or someone else) could check it, though. works. Thanks Stefan! Gün.
Proposal: adoption of mod_firehose subproject
Hi all, I have concluded negotiation with the BBC to open source some httpd modules that I wrote under the AL, and the BBC have very kindly agreed to donate the code to the ASF[1], which I believe would fit well as subprojects of httpd, and would like to know whether httpd would accept these. To be clear, this isn't a code dump, my intention is to continue to develop and support this moving forward, and hopefully expand the community around them. - mod_firehose: tcpdump for httpd Based originally on mod_dumpio.c, mod_firehose is an httpd filter that writes the contents of a request and/or a response to a file or pipe in such a way that the requests can be reconstructed later using a second dedicated tool called firehose. It was initially developed to help debug restful services that were secured with client certificates and therefore opaque to other tools like tcpdump or tcpflow, but was then subsequently used to record dirty traffic for subsequent replay for the purposes of testing. The module and the corresponding firehose demultiplexer was used to uncover some of the more tricky bugs in mod_cache, as well as protocol inconsistencies in backend services, and would prove very useful to anyone deploying restful services. We have also intended it to be used to create a dark live environment, where live traffic can be split off and diverted to a staging environment to test whether a software update works correctly. The code is currently packaged as an RPM, wrapped in autotools, and a snapshot is available here: http://people.apache.org/~minfrin/bbc-donated/mod_firehose/ http://people.apache.org/~minfrin/bbc-donated/firehose/ The corresponding README documenting in more detail is here: http://people.apache.org/~minfrin/bbc-donated/mod_firehose/README The code itself is here: http://people.apache.org/~minfrin/bbc-donated/mod_firehose/mod_firehose.c http://people.apache.org/~minfrin/bbc-donated/firehose/firehose.c Obviously the expectation is for the documentation to be completed and fleshed out. [1] https://issues.apache.org/bugzilla/show_bug.cgi?id=52322 Regards, Graham --
Re: Proposal: adoption of mod_firehose subproject
+1 on adopting On 13/12/2011 17:19, Graham Leggett wrote: Hi all, I have concluded negotiation with the BBC to open source some httpd modules that I wrote under the AL, and the BBC have very kindly agreed to donate the code to the ASF[1], which I believe would fit well as subprojects of httpd, and would like to know whether httpd would accept these. To be clear, this isn't a code dump, my intention is to continue to develop and support this moving forward, and hopefully expand the community around them. - mod_firehose: tcpdump for httpd Based originally on mod_dumpio.c, mod_firehose is an httpd filter that writes the contents of a request and/or a response to a file or pipe in such a way that the requests can be reconstructed later using a second dedicated tool called firehose. It was initially developed to help debug restful services that were secured with client certificates and therefore opaque to other tools like tcpdump or tcpflow, but was then subsequently used to record dirty traffic for subsequent replay for the purposes of testing. The module and the corresponding firehose demultiplexer was used to uncover some of the more tricky bugs in mod_cache, as well as protocol inconsistencies in backend services, and would prove very useful to anyone deploying restful services. We have also intended it to be used to create a dark live environment, where live traffic can be split off and diverted to a staging environment to test whether a software update works correctly. The code is currently packaged as an RPM, wrapped in autotools, and a snapshot is available here: http://people.apache.org/~minfrin/bbc-donated/mod_firehose/ http://people.apache.org/~minfrin/bbc-donated/firehose/ The corresponding README documenting in more detail is here: http://people.apache.org/~minfrin/bbc-donated/mod_firehose/README The code itself is here: http://people.apache.org/~minfrin/bbc-donated/mod_firehose/mod_firehose.c http://people.apache.org/~minfrin/bbc-donated/firehose/firehose.c Obviously the expectation is for the documentation to be completed and fleshed out. [1] https://issues.apache.org/bugzilla/show_bug.cgi?id=52322 Regards, Graham --
Re: Proposal: adoption of mod_firehose subproject
It is a wonderful tool, my vote does not count, but please +1 !! On Tue, Dec 13, 2011 at 12:46 PM, Issac Goldstand mar...@beamartyr.net wrote: +1 on adopting On 13/12/2011 17:19, Graham Leggett wrote: Hi all, I have concluded negotiation with the BBC to open source some httpd modules that I wrote under the AL, and the BBC have very kindly agreed to donate the code to the ASF[1], which I believe would fit well as subprojects of httpd, and would like to know whether httpd would accept these. To be clear, this isn't a code dump, my intention is to continue to develop and support this moving forward, and hopefully expand the community around them. - mod_firehose: tcpdump for httpd Based originally on mod_dumpio.c, mod_firehose is an httpd filter that writes the contents of a request and/or a response to a file or pipe in such a way that the requests can be reconstructed later using a second dedicated tool called firehose. It was initially developed to help debug restful services that were secured with client certificates and therefore opaque to other tools like tcpdump or tcpflow, but was then subsequently used to record dirty traffic for subsequent replay for the purposes of testing. The module and the corresponding firehose demultiplexer was used to uncover some of the more tricky bugs in mod_cache, as well as protocol inconsistencies in backend services, and would prove very useful to anyone deploying restful services. We have also intended it to be used to create a dark live environment, where live traffic can be split off and diverted to a staging environment to test whether a software update works correctly. The code is currently packaged as an RPM, wrapped in autotools, and a snapshot is available here: http://people.apache.org/~minfrin/bbc-donated/mod_firehose/ http://people.apache.org/~minfrin/bbc-donated/firehose/ The corresponding README documenting in more detail is here: http://people.apache.org/~minfrin/bbc-donated/mod_firehose/README The code itself is here: http://people.apache.org/~minfrin/bbc-donated/mod_firehose/mod_firehose.c http://people.apache.org/~minfrin/bbc-donated/firehose/firehose.c Obviously the expectation is for the documentation to be completed and fleshed out. [1] https://issues.apache.org/bugzilla/show_bug.cgi?id=52322 Regards, Graham --
Proposal: adoption of mod_policy subproject
Hi all, As with mod_firehose, I have concluded negotiation with the BBC to open source some httpd modules that I wrote under the AL, and the BBC have very kindly agreed to donate the code to the ASF[1], which I believe would fit well as a subproject of httpd, and would like to know whether httpd would accept these. To be clear, this isn't a code dump, my intention is to continue to develop and support this moving forward, and hopefully expand the community around them. - mod_policy: HTTP protocol police One of the curses of developing restful services is that clients are liberal in what they accept. This leads many developers of restful services to be liberal in what they send, resulting in a service that works for the developer, but fails under load or other real world conditions. mod_policy is a set of httpd filters that detect and implement a set of HTTP protocol checks, the idea being you declare a policy for your development and testing environments, and requests/responses that violate the policy will either log a warning to the error_log or explicitly fail with a suitable error message, clearly telling the developer what they have done wrong, with the expectation that the developer fixes this before the code sees production. The set of policies to apply is as follows, but is expected to change with time: o Content-Type: check that it's present and valid o Content-Length: check that it is present and valid (used to ensure that keepalive requests between httpd and load balancers aren't prematurely terminated by a Connection: close) o Keepalive: more detailed keepalive checks o Vary: headers like User-Agent represent a potential caching DoS, if specified header is present in Vary, fail o Validation: if ETag/Last-Modified not present, fail o Conditional: if a conditional request doesn't comes back with a properly compliant conditional response, fail o No-cache: if the response is declared no cache, fail o Max-age: if the response has a max-age less than a given threshold, fail o Version: if the request was less than a given version ( HTTP1/1, for example) fail These are an initial set of policies that were created to meet current needs at the time of development, however it is expected this list will grow with time. mod_policy would benefit greatly from the experience of the authorities on HTTP that exist here, with the above policies being tightened up and improved. With the proliferation of restful services out there in various states of dubious protocol compliance, this set of filters can be a huge help to stop developers doing non compliant things, while not getting in the way of production code. The filters also help enforce that content remains cacheable, which for sites that endure high loads or thundering herds is important. The code is currently packaged as an RPM, wrapped in autotools, and a snapshot is available here: http://people.apache.org/~minfrin/bbc-donated/mod_policy/ The corresponding README documenting in more detail is here: http://people.apache.org/~minfrin/bbc-donated/mod_policy/README The code itself is here: http://people.apache.org/~minfrin/bbc-donated/mod_policy/mod_policy.c Obviously the expectation is for the documentation to be completed and fleshed out. [1] https://issues.apache.org/bugzilla/show_bug.cgi?id=52322 Regards, Graham --
Proposal: adoption of mod_combine subproject
Hi all, As with mod_firehose and mod_policy, I have concluded negotiation with the BBC to open source some httpd modules that I wrote under the AL, and the BBC have very kindly agreed to donate the code to the ASF[1], which I believe would fit well as a subproject of httpd, and would like to know whether httpd would accept these. To be clear, this isn't a code dump, my intention is to continue to develop and support this moving forward, and hopefully expand the community around them. - mod_combine: Response concatenation As a page gets more complex, and eventually parts of the page like the header and footer become maintained by separate teams, the elements that make up a page can become fragmented. In the process, you can end up with a page that takes ages to load, because lots of fragments of javascript or fragments of CSS files are being downloaded separately by the browser. mod_combine is a handler that allows multiple URLs hosted by the server to be concatenated together and delivered as a single response, cutting down on the number of requests, and in turn the page loading time. At the same time, mod_combine attempts to behave sensibly when one or more of the files is missing, so as not to amplify a failure. The handler also properly supports conditional requests, creating a super ETag, and then reversing it to apply conditional requests on each element being concatenated. The code is currently packaged as an RPM, wrapped in autotools, and a snapshot is available here: http://people.apache.org/~minfrin/bbc-donated/mod_combine/ The corresponding README documenting in more detail is here: http://people.apache.org/~minfrin/bbc-donated/mod_combine/README The code itself is here: http://people.apache.org/~minfrin/bbc-donated/mod_combine/mod_combine.c Obviously the expectation is for the documentation to be completed and fleshed out. [1] https://issues.apache.org/bugzilla/show_bug.cgi?id=52322 Regards, Graham --
Re: Proposal: adoption of mod_firehose subproject
On 12/13/2011 9:19 AM, Graham Leggett wrote: - mod_firehose: tcpdump for httpd Based originally on mod_dumpio.c, mod_firehose is an httpd filter that writes the contents of a request and/or a response to a file or pipe in such a way that the requests can be reconstructed later using a second dedicated tool called firehose. It was initially developed to help debug restful services that were secured with client certificates and therefore opaque to other tools like tcpdump or tcpflow, but was then subsequently used to record dirty traffic for subsequent replay for the purposes of testing. The module and the corresponding firehose demultiplexer was used to uncover some of the more tricky bugs in mod_cache, as well as protocol inconsistencies in backend services, and would prove very useful to anyone deploying restful services. We have also intended it to be used to create a dark live environment, where live traffic can be split off and diverted to a staging environment to test whether a software update works correctly. A silly question perhaps, but was the tcpdump/wireshark file format considered? If not, was there a reason to invent a new representational format? It seems like the functionality you describe, at the httpd-internals visibility, emitted in a tcpdump-compatible representation, would be a godsend. All the GUI inspection tools already exist.
Re: Proposal: adoption of mod_policy subproject
On 12/13/2011 10:22 AM, Graham Leggett wrote: - mod_policy: HTTP protocol police mod_policy is a set of httpd filters that detect and implement a set of HTTP protocol checks, the idea being you declare a policy for your development and testing environments, and requests/responses that violate the policy will either log a warning to the error_log or explicitly fail with a suitable error message, clearly telling the developer what they have done wrong, with the expectation that the developer fixes this before the code sees production. Very interesting, sounds generally useful to developers, and rounds out that large gap in modules/test/ that we essentially offer very little for 'testing' in terms of functional test modules. The set of policies to apply is as follows, but is expected to change with time: o Content-Type: check that it's present and valid o Content-Length: check that it is present and valid (used to ensure that keepalive requests between httpd and load balancers aren't prematurely terminated by a Connection: close) Presume this means C-L and/or T-E chunked?
Re: Proposal: adoption of mod_firehose subproject
On 13 Dec 2011, at 8:35 PM, William A. Rowe Jr. wrote: A silly question perhaps, but was the tcpdump/wireshark file format considered? If not, was there a reason to invent a new representational format? It seems like the functionality you describe, at the httpd-internals visibility, emitted in a tcpdump-compatible representation, would be a godsend. All the GUI inspection tools already exist. It wasn't considered no, we were after something that was very simple, was text based so we could open up a capture file with a text viewer and look at the result without necessarily running it through firehose to demux it. It does sound like a very useful thing to do though. Having looked at the packet format, it seems like all the format types are different layer 2 packets of various kinds, when I'm after something a lot simpler (just request fragments muxed together). Something definitely worth looking into in more depth. Regards, Graham --
Re: Proposal: adoption of mod_policy subproject
On 13 Dec 2011, at 8:39 PM, William A. Rowe Jr. wrote: The set of policies to apply is as follows, but is expected to change with time: o Content-Type: check that it's present and valid o Content-Length: check that it is present and valid (used to ensure that keepalive requests between httpd and load balancers aren't prematurely terminated by a Connection: close) Presume this means C-L and/or T-E chunked? The content length filter just does a simple check for content length, while the keepalive filter takes into account T/E chunked in far more detail. What we've discovered is that our restful service layer was receiving a lot of HTTP/1.0 requests from client libraries (not entirely sure why, seems HTTP/1.0 is the default request version for some well used HTTP client libraries out there). If your service leaves off content length, and the client is HTTP/1.0, then the server indicates the end of the request by terminating the connection, which in turn makes the load balancers less effective than they should have been. We started with complaining about keepalive, and then switching to complaining about content-length so that HTTP/1.0 requests were taken into account. Regards, Graham --
Re: Proposal: adoption of mod_firehose subproject
Graham, Mod_firehose sounds very helpful. I like the record/replay options. It would be great if you could convince the developers. It is possible to do similar stuff with mod_security, though not in a very easy way, but mod_security still helps for debugging purposes. One thing you can not do with mod_security, though, is the following: To log encrypted connections between a reverse proxy and the backend applications. So far it is very hard to prove the Apache proxy sends the right stuff if you can not get hold of the backend application's logs. Now I wonder if mod_firehose could solve this problem too. Regards, Christian Folini -- First you make it, then it works, then you invite people to make it better. -- Eben Moglen, Free Software Foundation
Re: Proposal: adoption of mod_firehose subproject
On 13 Dec 2011, at 10:37 PM, Christian Folini wrote: One thing you can not do with mod_security, though, is the following: To log encrypted connections between a reverse proxy and the backend applications. So far it is very hard to prove the Apache proxy sends the right stuff if you can not get hold of the backend application's logs. Now I wonder if mod_firehose could solve this problem too. Yes, it can using the FirehoseProxyConnectionInput and FirehoseProxyConnectionOutput options described in http://people.apache.org/~minfrin/bbc-donated/mod_firehose/README. Regards, Graham --
Re: Proposal: adoption of mod_firehose subproject
On Dec 13, 2011, at 7:19 AM, Graham Leggett wrote: - mod_firehose: tcpdump for httpd +1 on adopting. S. -- scte...@apache.orghttp://www.temme.net/sander/ PGP FP: FC5A 6FC6 2E25 2DFD 8007 EE23 9BB8 63B0 F51B B88A View my availability: http://tungle.me/sctemme
Re: Time for httpd 2.4.0-RC1 ??
On Dec 11, 2011, at 6:01 AM, Jim Jagielski wrote: Now that apu-1.4.1 is close to release, it looks like we are close to being able to have our 1st RC for 2.4.0... My plan is to TR sometime this week... +1, let's do it. S. -- scte...@apache.orghttp://www.temme.net/sander/ PGP FP: FC5A 6FC6 2E25 2DFD 8007 EE23 9BB8 63B0 F51B B88A View my availability: http://tungle.me/sctemme
Re: Effective IP address / real IP address
On 12 Dec 2011, at 11:25 PM, William A. Rowe Jr. wrote: I have a frustrating update, which we need to take into consideration for the whole remote_ip-related resolution. From the httpd-ng workgroup... This makes sense, we're an HTTP server, lets stick to RFC related terms. Mark Nottingham m...@mnot.net response to my observation below; That's exactly backwards from how we have always used the terms in HTTP - 1945: client An application program that establishes connections for the purpose of sending requests. user agent The client which initiates a request. These are often browsers, editors, spiders (web-traversing robots), or other end user tools. 2068: client A program that establishes connections for the purpose of sending requests. user agent The client which initiates a request. These are often browsers, editors, spiders (web-traversing robots), or other end user tools. 2616: client A program that establishes connections for the purpose of sending requests. user agent The client which initiates a request. These are often browsers, editors, spiders (web-traversing robots), or other end user tools. +1. Regards, Graham --
Re: Effective IP address / real IP address
On 14 Dec 2011, at 12:50 AM, Graham Leggett wrote: On 12 Dec 2011, at 11:25 PM, William A. Rowe Jr. wrote: I have a frustrating update, which we need to take into consideration for the whole remote_ip-related resolution. From the httpd-ng workgroup... This makes sense, we're an HTTP server, lets stick to RFC related terms. Done in r 1214022. Regards, Graham --
Re: Effective IP address / real IP address
On Dec 13, 2011, at 5:33 PM, Graham Leggett wrote: On 14 Dec 2011, at 12:50 AM, Graham Leggett wrote: On 12 Dec 2011, at 11:25 PM, William A. Rowe Jr. wrote: I have a frustrating update, which we need to take into consideration for the whole remote_ip-related resolution. From the httpd-ng workgroup... This makes sense, we're an HTTP server, lets stick to RFC related terms. Done in r 1214022. Huh, I was wondring what Bill was talking about ... I couldn't remember any discussion that had reversed the meaning, and now I know why. The IP address received by the server interface when we are acting as a reverse proxy is not necessarily the IP address of the user agent. It could just as easily be the IP address of an ISP proxy, a corporate firewall, or a dozen other client-side intermediaries that are not the user agent. Hence, it is just a client. I can hear Graham screaming now. No worries -- I don't care what the variable is called in request_rec as long as the intent is clear enough, so feel free to leave it as is. Roy
Re: Effective IP address / real IP address
On 12/13/2011 9:06 PM, Roy T. Fielding wrote: On Dec 13, 2011, at 5:33 PM, Graham Leggett wrote: On 14 Dec 2011, at 12:50 AM, Graham Leggett wrote: On 12 Dec 2011, at 11:25 PM, William A. Rowe Jr. wrote: I have a frustrating update, which we need to take into consideration for the whole remote_ip-related resolution. From the httpd-ng workgroup... This makes sense, we're an HTTP server, lets stick to RFC related terms. Done in r 1214022. Huh, I was wondring what Bill was talking about ... I couldn't remember any discussion that had reversed the meaning, and now I know why. The IP address received by the server interface when we are acting as a reverse proxy is not necessarily the IP address of the user agent. It could just as easily be the IP address of an ISP proxy, a corporate firewall, or a dozen other client-side intermediaries that are not the user agent. Hence, it is just a client. That is EXACTLY our understanding. The immediate connection comes from a client. Not the end node asking for the data, but our immediate client which is a BigIP balancer or a telco cell browser proxy or anything else which you mention. the user agent is not just a client, but the originator of the request. Are we on the same page now?
