Re: [W3af-develop] checking http CONNECT method
OK thanks a lot for your response. I'll experiment with extended_urllib.py and also try to fix the duplicates in the output. Thanks On Wed, Aug 6, 2014 at 12:15 AM, Andres Riancho andres.rian...@gmail.com wrote: Sergio, On Tue, Aug 5, 2014 at 5:42 PM, Sergio A foobarm...@gmail.com wrote: Hi guys, Yesterday, while playing with w3af I saw something (detailed below) with the allowed methods plugin related to checking if the the http CONNECT method is available in a server or not and I'd like to know if you think it could be a bug or not. In the case you think it is a bug, I'd be very happy if I can fix it and, if possible, would like some advice or guidelines on what next steps you think I should do for implement the fix. The problem is that I'm not a python guru like you at all, currently I just know some python for my day to day stuff. And if it is not a bug, I'd like as well to contribute. So guidelines, advice, etc on how to do it is welcome :) Here is what I did: 1. git pull the last w3af version. 2. setup an apache server as a forward proxy Setup an apache web server that has enabled the CONNECT method for proxying clients (forward proxy not reverse proxy) so them can navigate setting the apache box as their proxy (the only client I´m testing now is on the same host hence Allow from 127.0.0.1). This is the relevant apache config I have (apart of the enabled mod_proxy modules, etc): VirtualHost *:80 ProxyRequests On ProxyVia On AllowCONNECT 80 443 563 /VirtualHost Proxy * Order deny,allow Deny from all Allow from 127.0.0.1 /Proxy 3. Manually check CONNECT method is working (look at the format on the CONNECT line): bla@ubuntu:~$ telnet localhost 80 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. CONNECT google.com:80 HTTP/1.1 HOST: google.com HTTP/1.0 200 Connection Established --- it worked Proxy-agent: Apache/2.4.7 (Ubuntu) blablablabla HTTP/1.0 400 Bad Request this is normal because blablabla is a bad request Content-Type: text/html; charset=UTF-8 Content-Length: 1419 Date: Mon, 04 Aug 2014 18:13:32 GMT Server: GFE/2.0 snip Ok, so far so good :) I understood the setup. 4. Setup a w3af profile for checking CONNECT method I setup a profile enabling only allowed methods plugin and this is what I see as the output of that plugin: The URL http://localhost/; has the following enabled HTTP methods: CONNECT, GET, GET, HEAD, HEAD, OPTIONS, OPTIONS, POST, POST. This information was found in the requests with ids 32, 39, 47, 52, 55 and 71. Strange to see that there are duplicated methods! GET/GET, HEAD/HEAD, etc. ugly at least. Note that, apart of all the methods but CONNECT been duplicated, what I see when going into request/response navigator for the CONNECT request (which is the one with id 55) is (look on the CONNECT line format): Request: CONNECT http://localhost/ HTTP/1.1 Host: localhost Accept-encoding: gzip, deflate Accept: */* User-agent: w3af.org Response: HTTP/1.1 400 Bad Request date: Mon, 04 Aug 2014 16:57:55 GMT content-length: 300 content-type: text/html; charset=iso-8859-1 connection: close server: Apache/2.4.7 (Ubuntu) But, if I run again the same w3af profile and look with wireshark what I see on the wire (look here as well in the CONNECT line) is: Request: CONNECT / HTTP/1.1 Host: localhost Accept-encoding: gzip, deflate Accept: */* User-agent: w3af.org Ah, yes, that's a bug in the representation of the HTTP requests. It is known (and ugly) that w3af sends: CONNECT / HTTP/1.1 Host: localhost Accept-encoding: gzip, deflate Accept: */* User-agent: w3af.org And this is recorded in the log/shown in the GUI: CONNECT http://localhost/ HTTP/1.1 Host: localhost Accept-encoding: gzip, deflate Accept: */* User-agent: w3af.org This was done to avoid adding host/port/protocol fields in the GUI and text files. Does that make sense? Not sure. But now we just show: CONNECT http://localhost/ HTTP/1.1 And all the information is there. In an enhanced version we could have a different GUI design with more widgets showing: Protocol: http Port: 80 Host: localhost Request: CONNECT / HTTP/1.1 That would be the ideal case. For your tests with the CONNECT method this is specially confusing. Response: HTTP/1.1 400 Bad Request Date: Mon, 04 Aug 2014 17:06:57 GMT Server: Apache/2.4.7 (Ubuntu) Content-Length: 300 Connection: close Content-Type: text/html; charset=iso-8859-1 So in summary: (A) Manual request (which works): CONNECT google.com:80 HTTP/1.1 (B) w3af Request/Response navigator reported request: CONNECT http://localhost/ HTTP/1.1 (C) w3af on the wire request: CONNECT / HTTP/1.1 I think that according to the RFC it looks like a valid request should have just a hostname, a colon and a port, like (A) and not like (B) neither (C):
[W3af-develop] checking http CONNECT method
Hi guys, Yesterday, while playing with w3af I saw something (detailed below) with the allowed methods plugin related to checking if the the http CONNECT method is available in a server or not and I'd like to know if you think it could be a bug or not. In the case you think it is a bug, I'd be very happy if I can fix it and, if possible, would like some advice or guidelines on what next steps you think I should do for implement the fix. The problem is that I'm not a python guru like you at all, currently I just know some python for my day to day stuff. And if it is not a bug, I'd like as well to contribute. So guidelines, advice, etc on how to do it is welcome :) Here is what I did: 1. git pull the last w3af version. 2. setup an apache server as a forward proxy Setup an apache web server that has enabled the CONNECT method for proxying clients (forward proxy not reverse proxy) so them can navigate setting the apache box as their proxy (the only client I´m testing now is on the same host hence Allow from 127.0.0.1). This is the relevant apache config I have (apart of the enabled mod_proxy modules, etc): VirtualHost *:80 ProxyRequests On ProxyVia On AllowCONNECT 80 443 563 /VirtualHost Proxy * Order deny,allow Deny from all Allow from 127.0.0.1 /Proxy 3. Manually check CONNECT method is working (look at the format on the CONNECT line): bla@ubuntu:~$ telnet localhost 80 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. CONNECT google.com:80 HTTP/1.1 HOST: google.com HTTP/1.0 200 Connection Established --- it worked Proxy-agent: Apache/2.4.7 (Ubuntu) blablablabla HTTP/1.0 400 Bad Request this is normal because blablabla is a bad request Content-Type: text/html; charset=UTF-8 Content-Length: 1419 Date: Mon, 04 Aug 2014 18:13:32 GMT Server: GFE/2.0 snip 4. Setup a w3af profile for checking CONNECT method I setup a profile enabling only allowed methods plugin and this is what I see as the output of that plugin: The URL http://localhost/; has the following enabled HTTP methods: CONNECT, GET, GET, HEAD, HEAD, OPTIONS, OPTIONS, POST, POST. This information was found in the requests with ids 32, 39, 47, 52, 55 and 71. Note that, apart of all the methods but CONNECT been duplicated, what I see when going into request/response navigator for the CONNECT request (which is the one with id 55) is (look on the CONNECT line format): Request: CONNECT http://localhost/ HTTP/1.1 Host: localhost Accept-encoding: gzip, deflate Accept: */* User-agent: w3af.org Response: HTTP/1.1 400 Bad Request date: Mon, 04 Aug 2014 16:57:55 GMT content-length: 300 content-type: text/html; charset=iso-8859-1 connection: close server: Apache/2.4.7 (Ubuntu) But, if I run again the same w3af profile and look with wireshark what I see on the wire (look here as well in the CONNECT line) is: Request: CONNECT / HTTP/1.1 Host: localhost Accept-encoding: gzip, deflate Accept: */* User-agent: w3af.org Response: HTTP/1.1 400 Bad Request Date: Mon, 04 Aug 2014 17:06:57 GMT Server: Apache/2.4.7 (Ubuntu) Content-Length: 300 Connection: close Content-Type: text/html; charset=iso-8859-1 So in summary: (A) Manual request (which works): CONNECT google.com:80 HTTP/1.1 (B) w3af Request/Response navigator reported request: CONNECT http://localhost/ HTTP/1.1 (C) w3af on the wire request: CONNECT / HTTP/1.1 I think that according to the RFC it looks like a valid request should have just a hostname, a colon and a port, like (A) and not like (B) neither (C): http://tools.ietf.org/html/rfc7231#section-4.3.6 Do you think this could be a bug or issue on how w3af generates a CONNECT request ? Also, how about the duplicates in the plugin output (I mean method names appearing here twice on all but CONNECT): The URL http://localhost/; has the following enabled HTTP methods: CONNECT, GET, GET, HEAD, HEAD, OPTIONS, OPTIONS, POST, POST. This information was found in the requests with ids 32, 39, 47, 52, 55 and 71. 5. Digging a bit I went through the code trying to read it, and I think the connect request is generated in: w3af.core.data.url.extended_urllib.AnyMethod But I'm not sure on where you think it could be a good place to fix it and how (of course in the case you thinking there´s a bug). Regards -- Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk ___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
Re: [W3af-develop] checking http CONNECT method
Sergio, On Tue, Aug 5, 2014 at 5:42 PM, Sergio A foobarm...@gmail.com wrote: Hi guys, Yesterday, while playing with w3af I saw something (detailed below) with the allowed methods plugin related to checking if the the http CONNECT method is available in a server or not and I'd like to know if you think it could be a bug or not. In the case you think it is a bug, I'd be very happy if I can fix it and, if possible, would like some advice or guidelines on what next steps you think I should do for implement the fix. The problem is that I'm not a python guru like you at all, currently I just know some python for my day to day stuff. And if it is not a bug, I'd like as well to contribute. So guidelines, advice, etc on how to do it is welcome :) Here is what I did: 1. git pull the last w3af version. 2. setup an apache server as a forward proxy Setup an apache web server that has enabled the CONNECT method for proxying clients (forward proxy not reverse proxy) so them can navigate setting the apache box as their proxy (the only client I´m testing now is on the same host hence Allow from 127.0.0.1). This is the relevant apache config I have (apart of the enabled mod_proxy modules, etc): VirtualHost *:80 ProxyRequests On ProxyVia On AllowCONNECT 80 443 563 /VirtualHost Proxy * Order deny,allow Deny from all Allow from 127.0.0.1 /Proxy 3. Manually check CONNECT method is working (look at the format on the CONNECT line): bla@ubuntu:~$ telnet localhost 80 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. CONNECT google.com:80 HTTP/1.1 HOST: google.com HTTP/1.0 200 Connection Established --- it worked Proxy-agent: Apache/2.4.7 (Ubuntu) blablablabla HTTP/1.0 400 Bad Request this is normal because blablabla is a bad request Content-Type: text/html; charset=UTF-8 Content-Length: 1419 Date: Mon, 04 Aug 2014 18:13:32 GMT Server: GFE/2.0 snip Ok, so far so good :) I understood the setup. 4. Setup a w3af profile for checking CONNECT method I setup a profile enabling only allowed methods plugin and this is what I see as the output of that plugin: The URL http://localhost/; has the following enabled HTTP methods: CONNECT, GET, GET, HEAD, HEAD, OPTIONS, OPTIONS, POST, POST. This information was found in the requests with ids 32, 39, 47, 52, 55 and 71. Strange to see that there are duplicated methods! GET/GET, HEAD/HEAD, etc. ugly at least. Note that, apart of all the methods but CONNECT been duplicated, what I see when going into request/response navigator for the CONNECT request (which is the one with id 55) is (look on the CONNECT line format): Request: CONNECT http://localhost/ HTTP/1.1 Host: localhost Accept-encoding: gzip, deflate Accept: */* User-agent: w3af.org Response: HTTP/1.1 400 Bad Request date: Mon, 04 Aug 2014 16:57:55 GMT content-length: 300 content-type: text/html; charset=iso-8859-1 connection: close server: Apache/2.4.7 (Ubuntu) But, if I run again the same w3af profile and look with wireshark what I see on the wire (look here as well in the CONNECT line) is: Request: CONNECT / HTTP/1.1 Host: localhost Accept-encoding: gzip, deflate Accept: */* User-agent: w3af.org Ah, yes, that's a bug in the representation of the HTTP requests. It is known (and ugly) that w3af sends: CONNECT / HTTP/1.1 Host: localhost Accept-encoding: gzip, deflate Accept: */* User-agent: w3af.org And this is recorded in the log/shown in the GUI: CONNECT http://localhost/ HTTP/1.1 Host: localhost Accept-encoding: gzip, deflate Accept: */* User-agent: w3af.org This was done to avoid adding host/port/protocol fields in the GUI and text files. Does that make sense? Not sure. But now we just show: CONNECT http://localhost/ HTTP/1.1 And all the information is there. In an enhanced version we could have a different GUI design with more widgets showing: Protocol: http Port: 80 Host: localhost Request: CONNECT / HTTP/1.1 That would be the ideal case. For your tests with the CONNECT method this is specially confusing. Response: HTTP/1.1 400 Bad Request Date: Mon, 04 Aug 2014 17:06:57 GMT Server: Apache/2.4.7 (Ubuntu) Content-Length: 300 Connection: close Content-Type: text/html; charset=iso-8859-1 So in summary: (A) Manual request (which works): CONNECT google.com:80 HTTP/1.1 (B) w3af Request/Response navigator reported request: CONNECT http://localhost/ HTTP/1.1 (C) w3af on the wire request: CONNECT / HTTP/1.1 I think that according to the RFC it looks like a valid request should have just a hostname, a colon and a port, like (A) and not like (B) neither (C): http://tools.ietf.org/html/rfc7231#section-4.3.6 Do you think this could be a bug or issue on how w3af generates a CONNECT request ? * There is room for improvement in the way we show the request in the GUI and store it in the log files. * In the
