Re: Speeding up network access: threading?

2010-01-05 Thread Steve Holden
Jens Müller wrote: > Hi and sorry for double posting - had mailer problems, > >> Terry said "queue". not "list". Use the Queue class (it's thread-safe) >> in the "Queue" module (assuming you're using Python 2.x; in Python 3.x >> it's called the "queue" module). > > Yes yes, I know. I use a queue

Re: Speeding up network access: threading?

2010-01-05 Thread Jens Müller
Hi and sorry for double posting - had mailer problems, Terry said "queue". not "list". Use the Queue class (it's thread-safe) in the "Queue" module (assuming you're using Python 2.x; in Python 3.x it's called the "queue" module). Yes yes, I know. I use a queue to realize the thread pool queue,

Re: Speeding up network access: threading?

2010-01-05 Thread Antoine Pitrou
Le Tue, 05 Jan 2010 15:04:56 +0100, Jens Müller a écrit : > > Is a list thrad-safe or do I need to lock when adding the results of my > worker threads to a list? The order of the elements in the list does not > matter. The built-in list type is thread-safe, but is doesn't provide the waiting fea

Re: Speeding up network access: threading?

2010-01-05 Thread MRAB
Jens Müller wrote: Hello, The fairly obvious thing to do is use a queue.queue for tasks and another for results and a pool of threads that read, fetch, and write. Thanks, indeed. Is a list thrad-safe or do I need to lock when adding the results of my worker threads to a list? The order of th

Re: Speeding up network access: threading?

2010-01-05 Thread Jens Müller
Hello, The fairly obvious thing to do is use a queue.queue for tasks and another for results and a pool of threads that read, fetch, and write. Thanks, indeed. Is a list thrad-safe or do I need to lock when adding the results of my worker threads to a list? The order of the elements in the l

Re: Speeding up network access: threading?

2010-01-05 Thread Jens Müller
Hello, The fairly obvious thing to do is use a queue.queue for tasks and another for results and a pool of threads that read, fetch, and write. Thanks, indeed. Is a list thrad-safe or do I need to lock when adding the results of my worker threads to a list? The order of the elements in the li

Re: Speeding up network access: threading?

2010-01-04 Thread exarkun
On 04:22 pm, m...@privacy.net wrote: Hello, what would be best practise for speeding up a larger number of http-get requests done via urllib? Until now they are made in sequence, each request taking up to one second. The results must be merged into a list, while the original sequence needs no

Re: Speeding up network access: threading?

2010-01-04 Thread Terry Reedy
On 1/4/2010 11:22 AM, Jens Müller wrote: Hello, what would be best practise for speeding up a larger number of http-get requests done via urllib? Until now they are made in sequence, each request taking up to one second. The results must be merged into a list, while the original sequence needs n

Speeding up network access: threading?

2010-01-04 Thread Jens Müller
Hello, what would be best practise for speeding up a larger number of http-get requests done via urllib? Until now they are made in sequence, each request taking up to one second. The results must be merged into a list, while the original sequence needs not to be kept. I think speed could be