First of all, thanks for the feedback, I really appreciate it. On Mon, Mar 30, 2009 at 3:08 AM, Adriano Marques <[email protected]> wrote: > Hello João, > >> I'm pretty sure about time issues. Perform tests on a big host is a >> test that takes time. The first obvious idea is to use parallel >> operations. We can perform a number of tests simultaneously and, since >> getting a response from the server takes some time, it won't be a >> problem having some threads to do the job. > > We can't deal with too much parallelism with PyGTK and they say that > threads and PyGTK it's like alcohol and driving... It can occour, but > no one knows what is actually going to happen in the end. We can use > some event-driven tasks, but that isn't actually parallelism.
Being honest to you, I didn't know about this issue between PyGTK and threads. Even if I do not touch GTK related code from the threads I might have problems? I mean, I just wanted to create threads to deal with network requests. Data collected by these requests should be shared with the main loop, that handles GTK related code... doing this way, no GTK would never be touched from the threads... it would not even be visible. I agree that having non-deterministic code is harder in many aspects. Threads should be used carefully, and if they can be avoided without lowering performance, they should. Sockets that can perform http 1.1 pipelining might also be helpful. Crawling websites is something very common. I've already seem and written some code in python to perform such tasks. Of course that (mainly) when you deal with big targets, you need to improve your performance the best, but I don't see it being prohibitive. If we are able to write code that can perform good checkings, with conclusive results, our code will be already very useful, even if it takes some time to run. > >> There are also some things we can do, like developing smart tests, >> that can change itself dinamically depending on the results that it >> already get... I mean, there is no use to search for directories of a >> webapp if the webapp is not running on the server. >> >> Another good idea I just had would be developing distributed tests... >> Have you ever think about including this functionality to Umit? I >> honestly think that including it to the web app scanner may turn the >> gsoc project too huge... but it could be a good feature that could be >> exploited not only by the app scanner. > > I think I didn't get the idea of tests. Could you explain it better? The idea is simple. If you have a large amount of tests, you can split it and distribute it through different computers. Each different computer runs part of the test. Something similar to what is done by DDoS's attacks. Anyway, I'm not really sure if building such infrastructure is a good idea. Sometimes I just go too far thinking about something =]. > >> (maybe I can submit another proposal about it... lol :) > > Be my guest ;-) Do you think it is a good idea? If so, I might start planning it better... > >> About the DNS, I'm not pretty sure. Maybe twisted can give us some >> functionalities, and if it is not enough, we can extend it to deal >> with our needs. The second idea is a little bit uncertain yet, but >> maybe we could use a Nmap nse script. Anyway, its something that I >> need to research more for better efficiency. > > Twisted might not be an option in our case, because of two things: > 1 - It is a big dependency, and we may face issues if we want to keep > portability and easy installation procedures > 2 - We're using PyGTK's mainloop. Therefore, we can't have twisted > mainloop running aside. And if we decide to use GTK on twisted's > mainloop, I think we would need a refactoring of the whole code. Which > is not something that we're willing to do in a near feature. > I agree that Twisted isn't the best option here. Dependency for itself is a reason big enough to give up on it. I can figure two options. The first one would be using nmap's nse scripts. The second one is crafting a python script, using python's sockets, to deal with it. Perhaps we can use some functionalities provided by UMPA. Anyway, the DNS stuff needed isn't that heavy. >> Another thing that wasn't on the previous e-mail is the reports. The >> web app scanner needs to create good reports. The web app should >> output a lot of information to files (maybe xml files), and it can be >> parsed to generate good reports. The idea of having html allows the >> user to change the reports... like click to minimize not interesting >> information, click for including more information (like the requests >> and responses that leaded to the event discovered), etc. Of course it >> all will need the use of some javascript wizardry. But the idea is >> creating some really good, beautiful, interactive and easy to read >> reports. > > Indeed. That is a good idea, but I would start with a simple xhtml > report, and if we have time in the end of GSoC, we could improve it. > Reports are important, but they have lower priority. That's cool that > you mentioned that, but keep the priority thing in mind, ok? > You are certainly right. As said before, sometimes I start thinking about things, having many ideas, and just can't stop! =]. > > Cheers! > > -- > Adriano Monteiro Marques > > http://adriano-marques.blogspot.com > http://www.umitproject.org > http://www.pythonbenelux.org > > "Don't stay in bed, unless you can make money in bed." - George Burns > Thank you very much again for your feedback Adriano. ------------------------------------------------------------------------------ _______________________________________________ Umit-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/umit-devel
