On Sat, Oct 20, 2012 at 9:22 AM, Pradipto Banerjee <pradipto.baner...@adainvestments.com> wrote: > Dennis, > > 1. Yes, .readlines() work where .read() fails. Thanks for the suggestion - > this has really given a big boost to the size of the data I can read.
If at all possible, consider reading the file iteratively and retaining only the current line. You say you need to "compare one line versus another" - are you able, for instance, to sort the file (externally; I think every OS these days will have a sort utility that can handle files >RAM) and then compare the current line against a small amount of retained state? This sort (pun intended) of trick can not only get you past the limit, it will often improve running time enormously too. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
