Another idea I've been playing with during the last days is to split w3af into two different processes: * Main: plugins run here * HTTPClient and response parser
Potentially use multiprocessing to connect both using a multiprocesing.Queue. The reasons to split w3af's architecture in two are: * The plugins have a moderate CPU usage, the parsers (html, pdf, etc.) use most of the CPU. Most workstations have more than one core, and we're only using one. Our HTTP request/response throughput is today limited by the CPU (parsing). If we move parsing to a different process we'll benefit from other core(s). To start with I believe the best is to have only one process doing HTTP+parsing; but the code should be written in such a way that we can have multiple processes for that. * We use threads to send/receive HTTP requests/responses, which is not the best way to do it. They consume resources (memory) and are not as fast as other options. Threads and gevent (to name one) don't play well together, so it would bring many issues to have gevent and threads in the same process. That's why I'll split in two processes and use threads for plugins and gevent for sending http requests. NOT going to be working on this during the following months, but wanted to hear your input and experiences with architectures like the one proposed. Regards, On Tue, Jun 5, 2012 at 10:03 AM, Andres Riancho <andres.rian...@gmail.com> wrote: > Taras, > > On Mon, Jun 4, 2012 at 5:00 PM, Taras <ox...@oxdef.info> wrote: >> Andres, >> geventhttpclient looks very fast HTTP client! >> Did you also try Twisted? Can you make simple comparison in req/s for: >> >> 1. currently used in w3af solution (urllib+threads) >> 2. geventhttpclient >> 3. Twisted > > I tried Twisted, not good compared with what geventhttpclient has to offer. > >> >>> During the last hours I've been trying to find a faster HTTP >>> client to integrate into w3af, and also performed some experiments >>> [0]. After testing some implementations, clients, programming >>> methodologies, etc. It seems that I've found the winner: >>> geventhttpclient [1]. >>> >>> With my tests with different methods I was only able to achieve >>> ~650 req/s , but according to geventhttpclient's home page it can >>> achieve ~4000 req/s (when tested in my environment it was around ~3500 >>> req/s). This is VERY impressive. >>> >>> There are some bad things about this library, like the C code used >>> for parsing the HTTP response which could bring some issues to Windows >>> users; and its dependency on gevent which adds one more dependency to >>> w3af; but with such a huge perf enhancement... I don't care ;) >> >> geventhttpclient hasn't package even in Debian/Ubuntu :( > > Yep, it's a very new library, only released a couple of months ago, > >> Only gevent bindings for Python: >> >> $ aptitude search gevent >> p python-gevent > > With that + geventhttpclient's code (which can be used without the > HTTP response parser and thus making it a pure-python library) we > should be ok. I'll finish the error handling stuff and then I'm > starting with some experiments with this library to see what we can > get. > >> >>> >>> Has someone researched on the topic of fast HTTP clients? Opinions? >>> Ideas? >>> >>> [0] >>> http://sourceforge.net/apps/trac/w3af/browser/extras/measure_http?rev=5041 >>> [1] https://github.com/gwik/geventhttpclient >>> >>> Regards, >> >> >> >> -- >> Taras >> http://oxdef.info > > > > -- > Andrés Riancho > Project Leader at w3af - http://w3af.org/ > Web Application Attack and Audit Framework > Twitter: @w3af > GPG: 0x93C344F3 -- Andrés Riancho Project Leader at w3af - http://w3af.org/ Web Application Attack and Audit Framework Twitter: @w3af GPG: 0x93C344F3 ------------------------------------------------------------------------------ Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce. With Perforce, you get hassle-free workflows. Merge that actually works. Faster operations. Version large binaries. Built-in WAN optimization and the freedom to use Git, Perforce or both. Make the move to Perforce. http://pubads.g.doubleclick.net/gampad/clk?id=122218951&iu=/4140/ostg.clktrk _______________________________________________ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
