Andres, Unfortunately I'm not expert in multiprocessing in Python but the idea (use threads for plugins and gevent for processing http transactions) looks pretty good!
06.03.2014 21:12, Andres Riancho пишет: > Another idea I've been playing with during the last days is to split > w3af into two different processes: > * Main: plugins run here > * HTTPClient and response parser > > Potentially use multiprocessing to connect both using a multiprocesing.Queue. > > The reasons to split w3af's architecture in two are: > * The plugins have a moderate CPU usage, the parsers (html, pdf, > etc.) use most of the CPU. Most workstations have more than one core, > and we're only using one. Our HTTP request/response throughput is > today limited by the CPU (parsing). If we move parsing to a different > process we'll benefit from other core(s). To start with I believe the > best is to have only one process doing HTTP+parsing; but the code > should be written in such a way that we can have multiple processes > for that. > > * We use threads to send/receive HTTP requests/responses, which is > not the best way to do it. They consume resources (memory) and are not > as fast as other options. Threads and gevent (to name one) don't play > well together, so it would bring many issues to have gevent and > threads in the same process. That's why I'll split in two processes > and use threads for plugins and gevent for sending http requests. > > NOT going to be working on this during the following months, but > wanted to hear your input and experiences with architectures like the > one proposed. > > Regards, > > On Tue, Jun 5, 2012 at 10:03 AM, Andres Riancho > <andres.rian...@gmail.com> wrote: >> Taras, >> >> On Mon, Jun 4, 2012 at 5:00 PM, Taras <ox...@oxdef.info> wrote: >>> Andres, >>> geventhttpclient looks very fast HTTP client! >>> Did you also try Twisted? Can you make simple comparison in req/s for: >>> >>> 1. currently used in w3af solution (urllib+threads) >>> 2. geventhttpclient >>> 3. Twisted >> >> I tried Twisted, not good compared with what geventhttpclient has to offer. >> >>> >>>> During the last hours I've been trying to find a faster HTTP >>>> client to integrate into w3af, and also performed some experiments >>>> [0]. After testing some implementations, clients, programming >>>> methodologies, etc. It seems that I've found the winner: >>>> geventhttpclient [1]. >>>> >>>> With my tests with different methods I was only able to achieve >>>> ~650 req/s , but according to geventhttpclient's home page it can >>>> achieve ~4000 req/s (when tested in my environment it was around ~3500 >>>> req/s). This is VERY impressive. >>>> >>>> There are some bad things about this library, like the C code used >>>> for parsing the HTTP response which could bring some issues to Windows >>>> users; and its dependency on gevent which adds one more dependency to >>>> w3af; but with such a huge perf enhancement... I don't care ;) >>> >>> geventhttpclient hasn't package even in Debian/Ubuntu :( >> >> Yep, it's a very new library, only released a couple of months ago, >> >>> Only gevent bindings for Python: >>> >>> $ aptitude search gevent >>> p python-gevent >> >> With that + geventhttpclient's code (which can be used without the >> HTTP response parser and thus making it a pure-python library) we >> should be ok. I'll finish the error handling stuff and then I'm >> starting with some experiments with this library to see what we can >> get. >> >>> >>>> >>>> Has someone researched on the topic of fast HTTP clients? Opinions? >>>> Ideas? >>>> >>>> [0] >>>> http://sourceforge.net/apps/trac/w3af/browser/extras/measure_http?rev=5041 >>>> [1] https://github.com/gwik/geventhttpclient >>>> >>>> Regards, >>> >>> >>> >>> -- >>> Taras >>> http://oxdef.info >> >> >> >> -- >> Andrés Riancho >> Project Leader at w3af - http://w3af.org/ >> Web Application Attack and Audit Framework >> Twitter: @w3af >> GPG: 0x93C344F3 > > > -- Taras https://www.oxdef.info ------------------------------------------------------------------------------ Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/13534_NeoTech _______________________________________________ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
