Thats where thread pooling comes into play. It gives the best of both worlds. You set a maximum amount of threads you think I/O wise you can handle, let it be 2 or 10. Then each task gets added to the que. Lets use an example of 5 threads and 10 requests. It would put the first 5 requests out. As a request free's up it assigns it to the open thread until the que is empty. This way you don't crush your machine. I usually don't work with more then 10 threads.
S On Thu, Jan 10, 2013 at 10:57 AM, allison nixon <[email protected]> wrote: > Yeah that's why I said I accidentally wrote it in python. I just want to > know if I can salvage this since I already wrote it. > > I also just realized something that I might have been doing wrong that is > hogging most of the resources... if I can get 200 requests per second with > python i'll let you guys know... > > > > On Thu, Jan 10, 2013 at 10:37 AM, Martín <[email protected]> wrote: > >> Hi! >> >> be also advised that because of the GlL [1] of the cpython interpreter, >> you may need to switch to an alternative interpreter implementation when >> performance of high concurrency applications is a must. >> >> In other words, Python may not be the best choice when trying to go to >> such performance limits. >> >> [1] http://en.wikipedia.org/wiki/Global_Interpreter_Lock >> >> >> On Thu, Jan 10, 2013 at 4:22 PM, Scott Kragen <[email protected]> wrote: >> >>> Allison, >>> >>> Have you looked into threading for python? I have used this library in >>> several of projects because writing a thread pool from scratch started to >>> give me a headache. >>> >>> >>> http://code.activestate.com/recipes/577105-synchronization-decorator-for-class-methods/ >>> >>> The advantage of a thread pool is it can que based on the max amount of >>> threads you define. >>> >>> Scott >>> >>> On Thu, Jan 10, 2013 at 10:06 AM, allison nixon <[email protected]>wrote: >>> >>>> Say I'm writing a broadscanner for a pet project and I accidentally >>>> wrote the entire thing in Python. >>>> >>>> Using asyncore and sockets, the best I can get is 3 http requests per >>>> second, maybe 10 per second at the very max. For a scanner this is of >>>> course, very lame. >>>> >>>> My goal is to of course make as many http request as a desktop computer >>>> will handle, so perhaps 200 per second with whatever number of sockets >>>> waiting in the background for a response(which I check periodically for a >>>> response and parse when I get it) >>>> >>>> Is there any way I can acheive 200 requests per second without learning >>>> another programming language? >>>> >>>> _______________________________________________ >>>> Pauldotcom mailing list >>>> [email protected] >>>> http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom >>>> Main Web Site: http://pauldotcom.com >>>> >>> >>> >>> >>> -- >>> >>> ---------------------------------------------------------------------------------------------------------------------- >>> There is only one metric in security that can be truly measured and that >>> is failure! >>> >>> --- Jack Daniels >>> ---------------------------------------------------------------------------------------------------------------------- >>> >>> _______________________________________________ >>> Pauldotcom mailing list >>> [email protected] >>> http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom >>> Main Web Site: http://pauldotcom.com >>> >> >> >> _______________________________________________ >> Pauldotcom mailing list >> [email protected] >> http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom >> Main Web Site: http://pauldotcom.com >> > > > > -- > _________________________________ > Note to self: Pillage BEFORE burning. > > _______________________________________________ > Pauldotcom mailing list > [email protected] > http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom > Main Web Site: http://pauldotcom.com > -- ---------------------------------------------------------------------------------------------------------------------- There is only one metric in security that can be truly measured and that is failure! --- Jack Daniels ----------------------------------------------------------------------------------------------------------------------
_______________________________________________ Pauldotcom mailing list [email protected] http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom Main Web Site: http://pauldotcom.com
