bruno at modulix wrote: > Joachim Worringen wrote: >> I need to process large lists (in my real application, this is to parse >> the content of a file). > > Then you probably want to use generators instead of lists. The problem > with large lists is that they eat a lot of memory - which can result in > swapping .
The effect also shows up in tiny examples (as the one posted) which surely don't swap on a 512MB machine. Also, I only read parts of the file into memory to avoid that memory becomes exhausted. Of course, using less memory is always a good idea - do you have a pointer on how to use generators for this application (basically, buffering file content in memory for faster access)? BTW, the effect also shows up with the linecache module. >> I noticed that the performance to access the >> individual list elements degrades over runtime. > > I leave this point to gurus, but it may have to do with swapping. Also, > this is not real-time, so variations may have to do with your OS tasks > scheduler. See above for the swapping. And the OS scheduler may create variations in runtime, but not monotone degradation. I don't think these two effect come into play here. -- Joachim - reply to joachim at domain ccrl-nece dot de Opinion expressed is personal and does not constitute an opinion or statement of NEC Laboratories. -- http://mail.python.org/mailman/listinfo/python-list
