On Mar 7, 3:06 pm, per <[email protected]> wrote:
> hi all,
>
> i have a program that essentially loops through a textfile file thats
> about 800 MB in size containing tab separated data... my program
> parses this file and stores its fields in a dictionary of lists.
When building a very large structure like you're doing, the cyclic
garbage collector can be a bottleneck. Try disabling the cyclic
garbage collector before building the large dictionary, and re-
enabling it afterwards.
import gc
gc.disable()
try:
for line in file:
split_values = line.strip().split('\t')
# do stuff with split_values
finally:
gc.enable()
Carl Banks
--
http://mail.python.org/mailman/listinfo/python-list
