Gerhard Fiedler wrote: > Rather than downloading and zipping in the same thread, you could run > multiple threads like you're doing that only download files, and one > zip-it-all-up thread. After downloading a file, the download threads place > a message in a queue that indicates the file they have downloaded, and the > zip-it-all-up thread takes messages out of that queue, one at a time, and > zips the files. >
I was using this approach earlier. The problem with this approach is too much temporary disk usage. Say I'm downloading 2 GB of data which is a combination of, say 600 files. Now following this approach, I'll have to make sure that I have 4 GB of disk space available on my hard drive. Where as downloading in pieces, adding them to the archive, and then unlinking the downloaded file helps proper utilization of disk resource. Thanks, Ritesh -- http://mail.python.org/mailman/listinfo/python-list