Gosh what a response. Here's what we've found: - doing it on the coldfusion server timesout the server. this rules out cffile, cfx tags and such. - breaking it into chucks timed-out the server as well. performance didn't really alter when we changed the size of the chunks; sooner or later the server ran out of memory. - using the csv as the datasource was too slow.
We're going to get someone to develop a custom app in java to read the csv and populate the db. Costly, but solves the problem. Thanks for everybody's feedback! Grant On 14/06/07, David Harris <[EMAIL PROTECTED]> wrote: > > > that's all good stuff! > > working with files has been an issue with CF for a while now... > > Dale Fraser wrote: > > FileRead also has a buffer size so you can get bits at a time. > > > > Regards > > Dale Fraser > > > > http://dalefraser.blogspot.com > > > > > > -----Original Message----- > > From: cfaussie@googlegroups.com [mailto:[EMAIL PROTECTED] On > Behalf > > Of Andrew Scott > > Sent: Thursday, 14 June 2007 11:17 AM > > To: cfaussie@googlegroups.com > > Subject: [cfaussie] Re: Large CSV's and server timeouts > > > > > > Doh, > > > > I thought you were talking about cfloop:-) Anyway, I am assuming that it > > loads it into memory to process. But really thinking about it, that > would be > > a very hungry way of doing that new feature. So I hope I am wrong.. > > > > > > > > Andrew Scott > > Senior Coldfusion Developer > > Aegeon Pty. Ltd. > > www.aegeon.com.au > > Phone: +613 8676 4223 > > Mobile: 0404 998 273 > > > > > > > > -----Original Message----- > > From: cfaussie@googlegroups.com [mailto:[EMAIL PROTECTED] On > Behalf > > Of David Harris > > Sent: Thursday, 14 June 2007 11:11 AM > > To: cfaussie > > Subject: [cfaussie] Re: Large CSV's and server timeouts > > > > > > After a quick check, I am thinking of CFLOOP. > > > > CFLOOP now has a "file" attritbute. > > > > In CF8 Docs: > > ====================================== > > Attribute: "file" (Optional) > > The absolute path and filename of the text file to read, one > > line at a time. This is helpful when reading large text files, > > because you can reuse the value of the index variable, > > which contains the current line of the file. When the loop > > completes, ColdFusion closes the file. > > ====================================== > > > > Usage: > > =================================== > > <cfloop file="c:\temp\simplefile.txt" index="line"> > > <cfoutput>#line#</cfoutput><br> > > </cfloop> > > =================================== > > > > > > roll on CF8! > > > > > > Andrew Scott wrote: > > > Hmm... That might be a good question.. > > > > > > You could be right on that, never tried that new feature as of yet. > > > > > > > > > Andrew Scott > > > Senior Coldfusion Developer > > > Aegeon Pty. Ltd. > > > www.aegeon.com.au > > > Phone: +613 8676 4223 > > > Mobile: 0404 998 273 > > > > > > > > > > > > -----Original Message----- > > > From: cfaussie@googlegroups.com [mailto:[EMAIL PROTECTED] On > > Behalf > > > Of David Harris > > > Sent: Thursday, 14 June 2007 10:54 AM > > > To: cfaussie > > > Subject: [cfaussie] Re: Large CSV's and server timeouts > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "cfaussie" group. To post to this group, send email to cfaussie@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cfaussie?hl=en -~----------~----~----~----~------~----~------~--~---
