Using the web-server to make such a request to open 10,000 odd URLs seems unrealistic approach. Rather make a separate process to do the same.
Thanks, Subhranath Chunder. On Mon, May 30, 2011 at 11:29 PM, rahul jain <[email protected]>wrote: > All, > > I would like to know how to do this? > > For example, in my views I have to visit 10,000 websites (make url > connection). Each of those connections are independent connections. > I would like to hit them as quickly as possible. Then return an > http-response with some results. > > I have few questions now > > Obviously, I cannot hit these sites sequentially, that will take forever. > Also, I believe its not possible, Django, put response time-out for each of > those view responses! > > Creating 10,000 threads also is not a good idea.Therefore, I created queue > based threads. > I created 100 working threads and infinite size queue (i am not sure if > this is a good idea) > But receiving max URL connection error! > > How to fix this "max URL connection error" ? In addition, any tips for > correctly implementing threads in django will be very helpful. > Also, how to set the max timeout for http-responses (django views) ? > > Thanks. > > Rahul > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
