David E Jones wrote: > On Mar 15, 2010, at 6:58 PM, Adam Heath wrote: > >> BJ Freeman wrote: >>> one of my smaller import files (8mb) is taking forever to be read in and >>> there is no output. >>> I am seeing the memory rail against the max setting. >>> got this error. >> The datafile set of classes is very broken for large files. It has a >> List<Record>, which means it will copy the entire file into memory >> before doing anything with it. The datafile code is not designed to >> handle large files. >> >> I consider this a bug that needs to be fixed. Checking... > > This should be possible, but may require API changes. The problem is that, > like XML, data files can be hierarchical and a "node" can have header and > footer lines in the file. > > I wouldn't consider this a bug, just like XML DOM parsing is not a "bug". Of > course, you're certainly entitled to your opinion.
Well, then, what would you suggest be done in this situation? Saying "don't do that" is not acceptable, as BJ has a real-world situation he is trying to solve. The datafile subsystem is close to providing this feature. It has a RecordIterator class, which is used by DataFile itself; modifying the calling code to use the former should be possible.
