Hi, I am having a requirement on to send async some 300,000 or more rest api url calls..
When I work with the code for 150,000 it works fine .It takes around 8 minutes.Is it possible to improve the performance When the count increased to some 200,000 ,I am getting the error. ------------------------------------------------------------------------------------------------- Traceback (most recent call last): Failure: twisted.internet.error.TimeoutError: User timeout caused connection failure. Unhandled error in Deferred: Unhandled Error Traceback (most recent call last): ----------------------------------------------------------------------------------------------- As my server is 8 CPU quadcore box, 2.40 Ghz, with 96G of RAM *Can you please suggest me how can I overcome this error * * * *Make the performance better* This is my piece of code where I have implemented twisted ------------------------------------------------------------------------------------------------ from twisted.internet import defer, reactor, task from twisted.web.client import getPage import sys, argparse, csv, collections, time, datetime maxRun =32 urls = "" def pageCallback(result): print len(result) return result def doWork(): for url in urls.split(','): d = getPage(url) d.addCallback(pageCallback) yield d def finish(ign): reactor.stop() def test(): deferreds = [] coop = task.Cooperator() work = doWork() for i in xrange(maxRun): d = coop.coiterate(work) deferreds.append(d) dl = defer.DeferredList(deferreds) dl.addCallback(finish) if __name__ == '__main__': filename = sys.argv[1] #I open the file ,read the files and create the url's test() reactor.run()
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python