On 08:35 pm, ashapatt...@gmail.com wrote:
Hi,
I am having a requirement on to send async some 300,000 or more rest
api
url calls..
Are your HTTP requests all the same server? Or are at least some of the
calls to the same server? If so, if you use persistent connections
(`twisted.web.client.Agent` supports these) you'll remove a bunch of the
latency and overhead of those requests.
Jean-Paul
When I work with the code for 150,000 it works fine .It takes around 8
minutes.Is it possible to improve the performance
When the count increased to some 200,000 ,I am getting the error.
-------------------------------------------------------------------------------------------------
Traceback (most recent call last):
Failure: twisted.internet.error.TimeoutError: User timeout caused
connection failure.
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
-----------------------------------------------------------------------------------------------
As my server is 8 CPU quadcore box, 2.40 Ghz, with 96G of RAM
*Can you please suggest me how can I overcome this error *
*
*
*Make the performance better*
This is my piece of code where I have implemented twisted
------------------------------------------------------------------------------------------------
from twisted.internet import defer, reactor, task
from twisted.web.client import getPage
import sys, argparse, csv, collections, time, datetime
maxRun =32
urls = ""
def pageCallback(result):
print len(result)
return result
def doWork():
for url in urls.split(','):
d = getPage(url)
d.addCallback(pageCallback)
yield d
def finish(ign):
reactor.stop()
def test():
deferreds = []
coop = task.Cooperator()
work = doWork()
for i in xrange(maxRun):
d = coop.coiterate(work)
deferreds.append(d)
dl = defer.DeferredList(deferreds)
dl.addCallback(finish)
if __name__ == '__main__':
filename = sys.argv[1]
#I open the file ,read the files and create the url's
test()
reactor.run()
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
