Hi ,
In my case the rest api url calls looks like this
http://example.com/event/user_id=12345
So here how to I implement the agent
for me every url has different different user_id's to be passes
Thanks and regards,
Asha
On Tue, Jul 2, 2013 at 8:59 AM, Asha Joseph <ashapatt...@gmail.com> wrote:
> Hi Jean,
> Thanks a lot for the quick response.
>
> All my requests are on the same server. or to the same domain.
>
> In that case do you suggest to use `twisted.web.client.Agent`
>
> I really don't require any responses from url that I send, it is fire and
> forget.
>
> thanks and regards,
> Asha
>
>
> On Mon, Jul 1, 2013 at 11:42 PM, <exar...@twistedmatrix.com> wrote:
>
>> On 08:35 pm, ashapatt...@gmail.com wrote:
>>
>>> Hi,
>>>
>>> I am having a requirement on to send async some 300,000 or more rest api
>>> url calls..
>>>
>>
>> Are your HTTP requests all the same server? Or are at least some of the
>> calls to the same server? If so, if you use persistent connections
>> (`twisted.web.client.Agent` supports these) you'll remove a bunch of the
>> latency and overhead of those requests.
>>
>> Jean-Paul
>>
>>> When I work with the code for 150,000 it works fine .It takes around 8
>>> minutes.Is it possible to improve the performance
>>>
>>> When the count increased to some 200,000 ,I am getting the error.
>>>
>>>
>>>
>>> ------------------------------**------------------------------**
>>> ------------------------------**-------
>>>
>>> Traceback (most recent call last):
>>> Failure: twisted.internet.error.**TimeoutError: User timeout caused
>>> connection failure.
>>> Unhandled error in Deferred:
>>> Unhandled Error
>>> Traceback (most recent call last):
>>>
>>> ------------------------------**------------------------------**
>>> ------------------------------**-----
>>>
>>> As my server is 8 CPU quadcore box, 2.40 Ghz, with 96G of RAM
>>>
>>>
>>> *Can you please suggest me how can I overcome this error *
>>> *
>>> *
>>> *Make the performance better*
>>>
>>>
>>> This is my piece of code where I have implemented twisted
>>> ------------------------------**------------------------------**
>>> ------------------------------**------
>>> from twisted.internet import defer, reactor, task
>>> from twisted.web.client import getPage
>>> import sys, argparse, csv, collections, time, datetime
>>>
>>> maxRun =32
>>> urls = ""
>>>
>>> def pageCallback(result):
>>> print len(result)
>>> return result
>>>
>>> def doWork():
>>> for url in urls.split(','):
>>> d = getPage(url)
>>> d.addCallback(pageCallback)
>>> yield d
>>>
>>> def finish(ign):
>>> reactor.stop()
>>>
>>> def test():
>>> deferreds = []
>>> coop = task.Cooperator()
>>> work = doWork()
>>> for i in xrange(maxRun):
>>> d = coop.coiterate(work)
>>> deferreds.append(d)
>>> dl = defer.DeferredList(deferreds)
>>> dl.addCallback(finish)
>>>
>>>
>>> if __name__ == '__main__':
>>>
>>> filename = sys.argv[1]
>>> #I open the file ,read the files and create the url's
>>>
>>>
>>> test()
>>> reactor.run()
>>>
>>
>> ______________________________**_________________
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.**com <Twisted-Python@twistedmatrix.com>
>> http://twistedmatrix.com/cgi-**bin/mailman/listinfo/twisted-**python<http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
>>
>
>
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
