On 2009-01-12, Sion Arrowsmith <si...@chiark.greenend.org.uk> wrote:
> Grant Edwards <inva...@invalid> wrote:
>>On 2009-01-09, Sion Arrowsmith <si...@chiark.greenend.org.uk> wrote:
>>> Grant Edwards <inva...@invalid> wrote:
>>>>If I were you, I'd try mmap()ing the file instead of reading it
>>>>into string objects one chunk at a time.
>>> You've snipped the bit further on in that sentence where the
>>> OP says that the file of interest is 2GB. Do you still want to
>>> try mmap'ing it?
>>Sure. The larger the file, the more you gain from mmap'ing it.
>>2GB should easily fit within the process's virtual memory
>>space.
>
> Assuming you're in a 64bit world. Me, I've only got 2GB of address
> space available to play in -- mmap'ing all of it out of the question.
Oh. I assumed that decent 32-bit OSes would provide at least 3-4GB
of address space to user processes. What OS are you using?
> But I supposed that mmap'ing it chunk at a time instead of
> reading chunk at a time might be worth considering.
I'd try mmap'ing it in large chunks (512MB maybe).
--
Grant Edwards grante Yow! I feel like a wet
at parking meter on Darvon!
visi.com
--
http://mail.python.org/mailman/listinfo/python-list
