On Mar 15, 2010, at 6:58 PM, Adam Heath wrote: > BJ Freeman wrote: >> one of my smaller import files (8mb) is taking forever to be read in and >> there is no output. >> I am seeing the memory rail against the max setting. >> got this error. > > The datafile set of classes is very broken for large files. It has a > List<Record>, which means it will copy the entire file into memory > before doing anything with it. The datafile code is not designed to > handle large files. > > I consider this a bug that needs to be fixed. Checking...
This should be possible, but may require API changes. The problem is that, like XML, data files can be hierarchical and a "node" can have header and footer lines in the file. I wouldn't consider this a bug, just like XML DOM parsing is not a "bug". Of course, you're certainly entitled to your opinion. -David
