here is my solution feel free to comment. check the files size if in acceptable limits (TBD) use the code as is. if above the limits send records to a datafile on disk, instead of the list. then add a new routine in the datafile2entityxml.java that reads that disk file to create the xml.
======================== BJ Freeman http://bjfreeman.elance.com Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93> Specialtymarket.com <http://www.specialtymarket.com/> Systems Integrator-- Glad to Assist Chat Y! messenger: bjfr33man Linkedin <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro> BJ Freeman sent the following on 3/15/2010 6:16 PM: > when code will not handle real world data, it is broken. > you and I discussed this when importing xml file and I choose to parse > them manually instead of using DOM. > So what term is there besides bug for broken design. > > ========================= > BJ Freeman > http://bjfreeman.elance.com > Strategic Power Office with Supplier Automation > <http://www.businessesnetwork.com/automation/viewforum.php?f=93> > Specialtymarket.com <http://www.specialtymarket.com/> > > Systems Integrator-- Glad to Assist > > Chat Y! messenger: bjfr33man > Linkedin > <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro> > > > David E Jones sent the following on 3/15/2010 6:05 PM: >> On Mar 15, 2010, at 6:58 PM, Adam Heath wrote: >> >>> BJ Freeman wrote: >>>> one of my smaller import files (8mb) is taking forever to be read in and >>>> there is no output. >>>> I am seeing the memory rail against the max setting. >>>> got this error. >>> The datafile set of classes is very broken for large files. It has a >>> List<Record>, which means it will copy the entire file into memory >>> before doing anything with it. The datafile code is not designed to >>> handle large files. >>> >>> I consider this a bug that needs to be fixed. Checking... >> This should be possible, but may require API changes. The problem is that, >> like XML, data files can be hierarchical and a "node" can have header and >> footer lines in the file. >> >> I wouldn't consider this a bug, just like XML DOM parsing is not a "bug". Of >> course, you're certainly entitled to your opinion. >> >> -David >> >> > > >
