Re: python threads and django views

[Subhranath Chunder]

Using the web-server to make such a request to open 10,000 odd URLs seems
unrealistic approach. Rather make a separate process to do the same.

Thanks,
Subhranath Chunder.

On Mon, May 30, 2011 at 11:29 PM, rahul jain wrote:

> All,
>
> I would like to know how to do this?
>
> For example, in my views I have to visit 10,000 websites (make url
> connection).   Each of those connections are independent connections.
> I would like to hit them as quickly as possible. Then return an
> http-response with some results.
>
> I have few questions now
>
> Obviously, I cannot hit these sites sequentially, that will take forever.
> Also, I believe its not possible, Django, put response time-out for each of
> those view responses!
>
> Creating 10,000 threads also is not a good idea.Therefore, I created queue
> based threads.
> I created 100 working threads and infinite size queue (i am not sure if
> this is a good idea)
> But receiving max URL connection error!
>
> How to fix this "max URL connection error" ? In addition, any tips for
> correctly implementing threads in django will be very helpful.
> Also, how to set the max timeout for http-responses  (django views) ?
>
> Thanks.
>
> Rahul
>
>
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

python threads and django views

[rahul jain]

All,

I would like to know how to do this?

For example, in my views I have to visit 10,000 websites (make url
connection).   Each of those connections are independent connections.
I would like to hit them as quickly as possible. Then return an
http-response with some results.

I have few questions now

Obviously, I cannot hit these sites sequentially, that will take forever.
Also, I believe its not possible, Django, put response time-out for each of
those view responses!

Creating 10,000 threads also is not a good idea.Therefore, I created queue
based threads.
I created 100 working threads and infinite size queue (i am not sure if this
is a good idea)
But receiving max URL connection error!

How to fix this "max URL connection error" ? In addition, any tips for
correctly implementing threads in django will be very helpful.
Also, how to set the max timeout for http-responses  (django views) ?

Thanks.

Rahul

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.