Hi Jean,
Thanks a lot for the quick response.
All my requests are on the same server. or to the same domain.
In that case do you suggest to use `twisted.web.client.Agent`
I really don't require any responses from url that I send, it is fire and
forget.
thanks and regards,
Asha
On Mon, Jul 1, 2013 at 11:42 PM, <exar...@twistedmatrix.com> wrote:
> On 08:35 pm, ashapatt...@gmail.com wrote:
>
>> Hi,
>>
>> I am having a requirement on to send async some 300,000 or more rest api
>> url calls..
>>
>
> Are your HTTP requests all the same server? Or are at least some of the
> calls to the same server? If so, if you use persistent connections
> (`twisted.web.client.Agent` supports these) you'll remove a bunch of the
> latency and overhead of those requests.
>
> Jean-Paul
>
>> When I work with the code for 150,000 it works fine .It takes around 8
>> minutes.Is it possible to improve the performance
>>
>> When the count increased to some 200,000 ,I am getting the error.
>>
>>
>>
>> ------------------------------**------------------------------**
>> ------------------------------**-------
>>
>> Traceback (most recent call last):
>> Failure: twisted.internet.error.**TimeoutError: User timeout caused
>> connection failure.
>> Unhandled error in Deferred:
>> Unhandled Error
>> Traceback (most recent call last):
>>
>> ------------------------------**------------------------------**
>> ------------------------------**-----
>>
>> As my server is 8 CPU quadcore box, 2.40 Ghz, with 96G of RAM
>>
>>
>> *Can you please suggest me how can I overcome this error *
>> *
>> *
>> *Make the performance better*
>>
>>
>> This is my piece of code where I have implemented twisted
>> ------------------------------**------------------------------**
>> ------------------------------**------
>> from twisted.internet import defer, reactor, task
>> from twisted.web.client import getPage
>> import sys, argparse, csv, collections, time, datetime
>>
>> maxRun =32
>> urls = ""
>>
>> def pageCallback(result):
>> print len(result)
>> return result
>>
>> def doWork():
>> for url in urls.split(','):
>> d = getPage(url)
>> d.addCallback(pageCallback)
>> yield d
>>
>> def finish(ign):
>> reactor.stop()
>>
>> def test():
>> deferreds = []
>> coop = task.Cooperator()
>> work = doWork()
>> for i in xrange(maxRun):
>> d = coop.coiterate(work)
>> deferreds.append(d)
>> dl = defer.DeferredList(deferreds)
>> dl.addCallback(finish)
>>
>>
>> if __name__ == '__main__':
>>
>> filename = sys.argv[1]
>> #I open the file ,read the files and create the url's
>>
>>
>> test()
>> reactor.run()
>>
>
> ______________________________**_________________
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.**com <Twisted-Python@twistedmatrix.com>
> http://twistedmatrix.com/cgi-**bin/mailman/listinfo/twisted-**python<http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
>
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
