"Dinesh B Vadhia" <[EMAIL PROTECTED]> wrote
fw = open(newLine.txt, 'w')
for i in xrange(0, 700,000, 1):
read a file fname from folder
for line in open(fname, 'r'):
do some simple string processing on line
fw.write(newline)
fw.close()
From 550,000 to 580,000 takes an additional 4 hours.
Sounds like a memory problem,. Can you look in Task
manager (assuming Windows) or top(Linux) to see what
the memory usage looks like?
In theory the read files should get closed automatically but
being extra cautious I might try the inner loop as:
fr = open(fname, 'r')
for line in fr:
do some simple string processing on line
fw.write(newline)
fr.close()
Just to be sure.
I might also try adding an fw.flush() after the write to
ensure it goes to disk. If that isn't enough I might try
opening and closing the write file each time using
append mode. In theiry that shouldn't make any
difference but it might be that its trying to hold the
entire output file (which must be huge!) in memory...
Not certain any of those will help but its something to try!
Alan G.
_______________________________________________
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
