Andres,
>Floyd, > >On Fri, Jan 8, 2010 at 7:47 AM, Floyd Fuh <floyd_...@yahoo.de> wrote: >> Hi everybody >> >> I wrote a new plugin, that takes every response and >> "cluster" them together by comparing their bodies. At the end >> you get a summary that looks like this: >> >> After all, there were 38 group(s) of responses. The smallest group has 1 >> response(s) >> (the representative of this group is response no. 21), the biggest group has >> 36 members >> (representative is response no 1). >> Response group 1: 21 >> Response group 2: 24 >> Response group 3: 25 >> ... >> Response group 38: 1 3 19-20 30 33 37 41-42 45-46 52 55 66 65 68 >> 73-74 117 120-127 130 129 128 132 162-163 165-166 184 >> . This information was found in the requests with ids 1, 21 to 29, 31 to 32, >> 34 to 35, 38 to 40, 43 to 44, 47, 49 to 50, 53, 57, 60, 64, 80 to 82, 98, >> 101 to 102, 107, 133 to 134, 148, 152 and 210. > > I like the idea, could be useful in some cases. What were your >motivations behind coding this plugin? > First of all, for small webpages it's a very easy possibility to get a first overview which pages there are (I really like to run it with the gui and then open results tab - KB Browser tab - clusterResponseBodies - Clustering Test - Summary - Response tab - rendered tab and then klick through all pages). The other thing is that there are only a few possibilities to find new vulnerabilities that are not known in the IT-Security world. I mean especially the errorPages grep plugin is great, but if there is no error message (or only a programmers customized error message!) how is it possible to detect a new vulnerability? I think this is one possibility, but of course this means some manual work. And of course I made this because you told me to separate my fuzzerPlugin into seperate plugins (this will be a plugin dependency when I'm finished ;) What do you think? >> I implemented it as a grep plugin, but it isn't a classical grep plugin. >> And I think the plugin should get all responses, not only those for >> the grep plugins. > > I don't know what you mean by this, actually, grep plugins get all >the request/responses. > I meant the grepResult=True option inside of the send method in xUrllib. What was the idea behind that? I thought in this case it should get "all" responses, even if the send method is invoked with grepResult=False. But maybe I'm wrong. I think I'm fine with how it is right now. >> And of course I get a lot of these error messages: >> The "clusterResponseBodies" plugin took more than 5 seconds to run. >> For a plugin that should only perform pattern matching, this is too much, >> please review its source code. > > Yes, it makes sense to see these messages in plugins that use >relative_distance a lot. I'll have to find better ways of implementing >that algorithm. At first I used difflib.quick_ratio() , then I tried >levenshtein, and now I'm with difflib again... I'll try to improve it >because its used framework-wide and its sloooow. > Yes I saw that ;) ... Don't know what to do about that... >> Please try it/read the code and tell me if you like it.. > > I think that there is a big problem with this plugin: memory >consumption. The class attribute that holds all the different >responses is going to grow in size at a rate that will fill 50MB of >memory easily, and 200mb of memory if the website has some particular >design. Do you agree? > Ok, my test webpage was too small, haven't seen that. Is there a better possibility than to hold all the different responses? I made two improvements: - lowered default values - Responses won't be saved if they're not going to be outputed Could you try it again? If it still needs too much memory, we can decrease the maxOutputedGroups again. And another thing: Would it be ok if I modify the grep plugin errorPages? In my test application I get >40'000 info messages, which is too much. I think we could remove some duplicates cheers floyd __________________________________________________ Do You Yahoo!? Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz gegen Massenmails. http://mail.yahoo.com
clusterResponseBodies.py
Description: Binary data
------------------------------------------------------------------------------ This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev
_______________________________________________ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
