Few quick answers:
1) pypy - no as we are focused mainly on an official Python runtime
2) "__functions__[hash_regex]" outside the for loop - will do, but the
benefits are really infinitesimally small (when comparing to the main
processing parts of that loop)
3) I'll for sure try something along that multiprocessing.Queue.
4) "I meant that if there are 100 threads sending stuff using that
method..." - this seems comprehensible. I believe that we'll just put a
constraint that --threads and --delay are not usable (it doesn't make much
sense to use e.g. --threads=10 --delay=10 <- oxymoron)
Kind regards,
Miroslav Stampar
On Tue, Jun 5, 2012 at 5:21 PM, Andres Riancho <andres.rian...@gmail.com>wrote:
> Miroslav,
>
> On Tue, Jun 5, 2012 at 6:40 AM, Miroslav Stampar
> <miroslav.stam...@gmail.com> wrote:
> > Hi Andres.
> >
> > On Sat, Jun 2, 2012 at 11:19 PM, Andres Riancho <
> andres.rian...@gmail.com>
> > wrote:
> >>
> >> List,
> >>
> >> During PHDays we had a really good idea with Miroslav: "I review
> >> sqlmap's code and send you some comments about it, and Miroslav will
> >> review some w3af code and do the same". So, while I had some spare
> >> minutes at the airport I performed some initial review:
> >>
> >> * I like the idea of using psyco, what's your experience with it? Do
> >> you guys recommend it?
> >
> > It was a good thing till it lasted (Python <= v2.6) but as it's official
> > page says "12 March 2012 Psyco is unmaintained and dead"
> > (http://psyco.sourceforge.net/), so it's advised to not use it.
>
> Oops! Have you tested pypy? Do you recommend it? :)
>
> >>
> >>
> >> * Liked the concept in "def smokeTest():", which sounds interesting to
> >> have also in w3af
> >
> > Ok
>
> While implementing this into w3af I realized that "nosetests" is
> actually doing all of that for me already, even the syntax error
> checks.
>
> dz0@dz0-laptop:~/workspace/w3af$ nosetests -s
> --doctest-result-variable=_test_res_ --rednose --with-doctest
> --doctest-tests plugins/discovery/netcraft.py
> X
>
> -----------------------------------------------------------------------------
> 1) ERROR: Failure: SyntaxError (invalid syntax (netcraft.py, line 107))
>
> Traceback (most recent call last):
>
> /usr/local/lib/python2.6/dist-packages/nose-0.11.4-py2.6.egg/nose/loader.py
> line 382 in loadTestsFromName
> addr.filename, addr.module)
>
> /usr/local/lib/python2.6/dist-packages/nose-0.11.4-py2.6.egg/nose/importer.py
> line 39 in importFromPath
> return self.importFromDir(dir_path, fqname)
>
> /usr/local/lib/python2.6/dist-packages/nose-0.11.4-py2.6.egg/nose/importer.py
> line 86 in importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
> SyntaxError: invalid syntax (netcraft.py, line 107)
>
>
> >>
> >>
> >> * lib/core/testing.py : shouldn't most/all of this be migrated to
> >> unit-tests and run using "nosetests" or some other tool like that?
> >
> > In majority of cases it's impossible to use any of those python-based
> > testing tools if you need to run a testing program as an standalone
> > executable (not as a same program same module). We need to run it as a
> > standalone against testing environment (xml/livetests.xml) and parse the
> > output to see if it went ok. Look into this 'testing.py' as our way how
> to
> > deal with that problem (without using any 3rd party tools).
>
> Hmmm, ok, understood. In our case we have a w3afCore object that we
> can manipulate in unittests in order to run scans and check the
> results, this has been a great step forward since we now can run
> "nosetests" and it will tell us if the scan results are the expected
> ones AND if the unit-tests all passed. This will also fit very well in
> an environment that (if possible) we'll have that will be a continuous
> integration system for building / testing every night.
>
> >>
> >>
> >> * As Miroslav mentioned, we're using the same keepalive.py module,
> >> I'll have to run a diff between w3af's and sqlmap's and see what we
> >> changed; since we both made modifications to "make it work".
> >
> > Ok
> >>
> >>
> >> * Using rangehandler.py is a great idea for speeding up (A LOT) the
> >> extraction of information, it seems that you guys add it to the
> >> urlopener but don't use it?
> >
> > We use it in --null-connection (and implicitly in -o) for boolean-based
> > blind cases. If you take a look into lib/core/option.py you'll see that
> in
> > def __urllib2Opener() it's installed among other handlers. Also, if you
> take
> > a look into rangehandler.py you'll see that it's sole purpose is to
> properly
> > handle 206 and 416 HTTP codes related to those range-cases. Grep for
> > "kb.nullConnection" and you'll see how "Range" (or we call it "null
> > connection") method is used (extremely fast if available for
> boolean-based
> > blind cases)
> >>
> >>
> >> * Could you please explain me the first part of this if? "if
> >> conf.hostname in ('localhost', '127.0.0.1') or conf.ignoreProxy:" does
> >> it really make sense? Aren't you ignoring the user's wish?
> >
> > Python, as you know, uses an automatic extraction of proxy information
> from
> > current environment (e.g. http_proxy env variable). Now, in 99% of cases
> you
> > don't want your automatic proxy settings to affect your access to the
> > localhost (be real, in most of browser settings first thing on the ignore
> > proxy list are localhost/127.0.0.1). That way we are just dealing with
> major
> > number of users who would complain about accessing localhost web server
> and
> > not reaching it (because corporate proxy settings were used
> automatically)
> >>
> >>
> >> * heh, I also use gprof2dot for profiling, but instead of having it
> >> inside w3af, I simply call it from the command line and have it
> >> generate a PNG. Note, where is "start()" defined for this line?
> >> cProfile.run("start()", profileOutputFile)
> >
> > that start() is defined inside the lib/controller/controller.py (it
> > represents the first sqlmap call that starts setting up everything and
> runs
> > the tool's functionality). string "start()" represents an eval-like
> python
> > call that will be called from the main() perspective. if you take a look
> > into the main() you'll see that start() is reachable from there.
> >>
> >>
> >> * Read this comment:
> >> """
> >> # Set kb.partRun in case "common prediction" feature (a.k.a. "good
> >> # samaritan") is used
> >> """
> >>
> >> Good samaritan was a feature I added many years ago to w3af's sqlmap,
> >> and the name came from the idea that the user could help the blind sql
> >> injection process by completing the word that was being extracted.
> >> Example: "If sqlmap extracted -hello w- the user could type -orld- in
> >> the console and have it checked with a SELECT statement". According to
> >> the pieces of code I was able to find, that was replaced by a more
> >> automatic idea where a file feeds common strings to the process,
> >> correct? The idea sounds good, but maybe users still want to
> >> contribute to the process?
> >
> > I am interested how you managed to get the user's input while outputting
> the
> > results in the same time?
>
> Yeah, I had problems with that too. At this moment that's working in
> w3af for our console. Here is the code:
>
> def _cmd_start(self, params):
> '''
> Start the core in a different thread, monitor keystrokes in
> the main thread.
> @return: None
> '''
>
>
> http://sourceforge.net/apps/trac/w3af/browser/trunk/core/ui/consoleUi/rootMenu.py
>
> > We've always had a problem where you have to
> > provide an user with that "raw_input" functionality and in the same time
> do
> > the output. Problem with Python is that it doesn't give you something
> like
> > "Keyboard Hooking" that would easify this all problem.
> >>
> >>
> >> * dataToStdout() is a handy function, but I think that you should
> >> consider migrating to something more generic like python's logging
> >> module. If in the future you want to provide options to storing the
> >> data in a file, or similar, it might come handy. In w3af we have the
> >> outputManag
> >
> > We are using both logging module and dataToStdout. dataToStdout can be
> > called from anywhere at any time and it will always output (in thread
> safe
> > manner) just the thing you've given to it. Logger as the other approach
> does
> > the output of the given text in an line manner (!) and that would be a
> very
> > bad thing especially when you want to output character by character.
> Also,
> > logger outputs everything in a message like structure (prepending e.g.
> > [CRITICAL]) and in lots of cases we don't want that. So, those two are
> > synergetic in a way and we need them both for a proper sqlmap run.
>
> Interesting needs you have indeed. (not sure why but that sentence
> came our in the way Yoda talks)
>
> >>
> >>
> >> - From our talks I understood that sqlmap used multiprocessing for
> >> cracking hashes (or something like that) but I can't find any
> >> reference to the multiprocessing module in the latest version. Could
> >> you point me in the right direction so I can analyze that code?
> >
> > lib/utils/hash.py
>
> I didn't had the latest version!
>
> Maybe you want to put the if before the hash calculation?
>
> current = __functions__[hash_regex](password = word,
> uppercase = False, **kwargs)
> count += 1
>
> if not isinstance(word, basestring):
> continue
>
> Also, the code could be sped up a little bit by taking this
> "__functions__[hash_regex]" outside the for loop
>
> Not sure about what I'm going to say next... but... given that all
> processes are going to read from the same wordlist object (which is
> fine), the wordlist has a "def next(self):" that is locked (which is
> required for multiprocessing to work) , don't you think that it might
> be the case in workstations with 4 cores where there is too much time
> spent waiting for the file lock to be released?
>
> Maybe you could have a multiprocessing.Queue inside wordlist that is
> loaded with 1000 values from the wordlist each time its size is 0?
>
> If you experiment with this, let me know the results.
>
> >>
> >>
> >> - Not sure how usable it is for you guys, but in some cases the
> >> charset is set in a meta tag; you're ignoring that here:
> >> if contentType and (contentType.find('charset=') != -1):
> >> charset = checkCharEncoding(contentType.split('charset=')[-1])
> >>
> >> if charset:
> >> page = getUnicode(page, charset)
> >
> > I am not sure if you are using the latest revision from our repository
> (go
> > to www.sqlmap.org for proper "svn checkout" line).
>
> Oops!
>
> > Those few lines go like this (in latest v1.0-dev):
> > ...
> > if contentType and (contentType.find('charset=') != -1):
> > httpCharset =
> > checkCharEncoding(contentType.split('charset=')[-1])
> >
> > metaCharset =
> > checkCharEncoding(extractRegexResult(META_CHARSET_REGEX, page, re.DOTALL
> |
> > re.IGNORECASE))
> > ...
> > We are not ignoring the metaCharset. We are using them both (while
> > httpCharset has the higher priority) in following code.
> >>
> >>
> >> See w3af's httpResponse.py for an example on how we're doing it.
> >>
> >> - Not thread safe?
> >>
> >> if conf.delay is not None and isinstance(conf.delay, (int,
> >> float)) and conf.delay > 0:
> >> time.sleep(conf.delay)
> >
> > But those few lines are IMHO irrelevant for any "thread-safe" manner.
> Thread
> > safe means that you have to be careful to prevent situations where
> something
> > critical could be changed in the same time as other thread is reading it
> (or
> > vice versa/similar) and this is really something of no interest in that
> > field.
>
> I meant that if there are 100 threads sending stuff using that method,
> all 100 requests will be sent to the wire "at the same time"; not
> respecting the users's delay configuration. Then, it will wait for
> conf.delay and send 100 more requests.
>
> > If you though that time.sleep() blocks the whole process, that's not the
> > case. It blocks only the current thread
> > (
> http://stackoverflow.com/questions/92928/time-sleep-sleeps-thread-or-process
> ),
> > so nothing to be worried in this field too.
> >>
> >>
> >> Maybe move the "kb.locks.reqLock.acquire()" some lines before?
> >
> > No need
> >>
> >>
> >> - Doesn't this kill the keepalive.py handler? Should try to capture
> >> packets.
> >>
> >> if not req.has_header("Connection"):
> >> requestHeaders += "\nConnection: close"
> >
> > Those requestHeaders is just a "log entry" and it doesn't kill the
> > "keep-alive" functionality.
>
> Oh, I feel stupid, read too fast.
>
> >This was just a dirty hack where everything has
> > been declared (in log/traffic files) as connection close (to appear like
> to
> > the end user) as in that point you can't know if something is really
> > keep-alive or not (you can take a look into the header content and you
> won't
> > see a thing - it's handled by a keepalive handler in a low-level manner).
> > Now, we could do some dirty hacks to signal from keepalive handler if
> > something is really Keep-Alive (I am saying that because there are lots
> of
> > cases where Keep-Alive is just not possible or dropped in some point) and
> > properly do the logging stuff but this is of low priority this moment.
> >>
> >>
> >> I know that many of these are questions, but I hope they trigger some
> >> good ideas :)
> >
> > Thank you for your observations :)
> >>
> >>
> >> PS: I only used 2h for reading code. 2h left.
> >
> > :)
> >
> > I'll try to do mine this week. Prepare yourself.
>
> hehehe, ok :)
>
> >>
> >>
> >> Regards,
> >>
> >> --
> >> Andrés Riancho
> >> Project Leader at w3af - http://w3af.org/
> >> Web Application Attack and Audit Framework
> >> Twitter: @w3af
> >> GPG: 0x93C344F3
> >>
> > Kind regards,
> > Miroslav Stampar
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> Live Security Virtual Conference
> >> Exclusive live event will cover all the ways today's security and
> >> threat landscape has changed and how IT managers can respond.
> Discussions
> >> will include endpoint security, mobile security and the latest in
> malware
> >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> >> _______________________________________________
> >> sqlmap-users mailing list
> >> sqlmap-users@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/sqlmap-users
> >
> >
> >
> >
> > --
> > Miroslav Stampar
> > http://about.me/stamparm
>
>
>
> --
> Andrés Riancho
> Project Leader at w3af - http://w3af.org/
> Web Application Attack and Audit Framework
> Twitter: @w3af
> GPG: 0x93C344F3
>
--
Miroslav Stampar
http://about.me/stamparm
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
sqlmap-users mailing list
sqlmap-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlmap-users
