On Mar 15, 2010, at 8:47 PM, Adam Heath wrote: > David E Jones wrote: >> On Mar 15, 2010, at 6:58 PM, Adam Heath wrote: >> >>> BJ Freeman wrote: >>>> one of my smaller import files (8mb) is taking forever to be read in and >>>> there is no output. >>>> I am seeing the memory rail against the max setting. >>>> got this error. >>> The datafile set of classes is very broken for large files. It has a >>> List<Record>, which means it will copy the entire file into memory >>> before doing anything with it. The datafile code is not designed to >>> handle large files. >>> >>> I consider this a bug that needs to be fixed. Checking... >> >> This should be possible, but may require API changes. The problem is that, >> like XML, data files can be hierarchical and a "node" can have header and >> footer lines in the file. >> >> I wouldn't consider this a bug, just like XML DOM parsing is not a "bug". Of >> course, you're certainly entitled to your opinion. > > Well, then, what would you suggest be done in this situation? Saying > "don't do that" is not acceptable, as BJ has a real-world situation > he is trying to solve. > > The datafile subsystem is close to providing this feature. It has a > RecordIterator class, which is used by DataFile itself; modifying the > calling code to use the former should be possible.
Yep, exactly, it's a "feature" as you say here. As I said it could certainly be implemented, but might require some API changes. -David
