Re: write a 20GB file

2010-05-15 Thread Nobody
On Fri, 14 May 2010 18:38:55 -0400, J wrote: someone smarter than me can correct me, but file.write() will write when it's buffer is filled, or close() or flush() are called. And, in all probability, seek() will either flush it immediately or cause the next write() to flush it before writing

Re: write a 20GB file

2010-05-15 Thread Nathan Rice
This is precisely the situation mmap was made for :) It has almost the same methods as a file so it should be an easy replacement. On Sat, May 15, 2010 at 10:05 AM, Nobody nob...@nowhere.com wrote: On Fri, 14 May 2010 18:38:55 -0400, J wrote: someone smarter than me can correct me, but

Re: write a 20GB file

2010-05-15 Thread Dave Angel
Nathan Rice wrote: This is precisely the situation mmap was made for :) It has almost the same methods as a file so it should be an easy replacement. snip Only on a 64bit system, and I'm not sure it's even possible there in every case. On a 32bit system, it would be impossible to mmap a

Re: write a 20GB file

2010-05-15 Thread Patrick Maupin
On May 15, 7:09 pm, Dave Angel da...@ieee.org wrote: Nathan Rice wrote: This is precisely the situation mmap was made for :)  It has almost the same methods as a file so it should be an easy replacement. snip Only on a 64bit system, and I'm not sure it's even possible there in every

Re: write a 20GB file

2010-05-15 Thread Patrick Maupin
On May 15, 7:09 pm, Dave Angel da...@ieee.org wrote: Nathan Rice wrote: This is precisely the situation mmap was made for :)  It has almost the same methods as a file so it should be an easy replacement. snip Only on a 64bit system, and I'm not sure it's even possible there in every

write a 20GB file

2010-05-14 Thread Jackie Lee
Hello there, I have a 22 GB binary file, a want to change values of specific positions. Because of the volume of the file, I doubt my code a efficient one: #! /usr/bin/env python #coding=utf-8 import sys import struct try: f=open(sys.argv[1],'rb+') except (IOError,Exception): print

Re: write a 20GB file

2010-05-14 Thread Dave Angel
Jackie Lee wrote: Hello there, I have a 22 GB binary file, a want to change values of specific positions. Because of the volume of the file, I doubt my code a efficient one: #! /usr/bin/env python #coding=utf-8 import sys import struct try: f=open(sys.argv[1],'rb+') except

Re: write a 20GB file

2010-05-14 Thread Jackie Lee
Thx, Dave, The code works fine. I just don't know how f.write works. It says that file.write won't write the file until file.close or file.flush. So I don't know if the following one is more efficient (sorry I forget to add condition to break the loop): #! /usr/bin/env python #coding=utf-8

Re: write a 20GB file

2010-05-14 Thread J
On Fri, May 14, 2010 at 07:32, Jackie Lee jackie.sp...@gmail.com wrote: Thx, Dave, The code works fine. I just don't know how f.write works. It says that file.write won't write the file until file.close or file.flush. So I don't know if the following one is more efficient (sorry I forget to

Re: write a 20GB file

2010-05-14 Thread Martin v. Loewis
The code works fine. I just don't know how f.write works. It says that file.write won't write the file until file.close or file.flush. You are misinterpreting the documentation. It certainly won't keep the entire file in memory. Instead, it has a fixed-size buffer (something like 8kiB or 32kiB)

Re: write a 20GB file

2010-05-14 Thread Nobody
On Fri, 14 May 2010 10:50:49 -0400, J wrote: someone smarter than me can correct me, but file.write() will write when it's buffer is filled, or close() or flush() are called. And, in all probability, seek() will either flush it immediately or cause the next write() to flush it before writing

Re: write a 20GB file

2010-05-14 Thread J
On Fri, May 14, 2010 at 15:23, Nobody nob...@nowhere.com wrote: On Fri, 14 May 2010 10:50:49 -0400, J wrote: someone smarter than me can correct me, but file.write() will write when it's buffer is filled, or close() or flush() are called. And, in all probability, seek() will either flush it

Re: write a 20GB file

2010-05-14 Thread Jackie Lee
Thanks to y'all. I should have be more careful reading the documentation. Cheers On Fri, May 14, 2010 at 10:07 PM, Martin v. Loewis mar...@v.loewis.de wrote: The code works fine. I just don't know how f.write works. It says that file.write won't write the file until file.close or file.flush.