Eek! A file of that size and complexity needs to be handled by the database, putting it into an array would be a nightmare. It would be fine for smaller files, but something that size would probably push the server over.
Les, what database are you using? *Crosses fingers he says something decent :-D I would import into the database and then do the manipulation there with stored procs for sure. Rob -----Original Message----- From: Phillip M. Vector [mailto:[EMAIL PROTECTED] Sent: 12 June 2007 15:06 To: CF-Talk Subject: Re: Large CSV File Is it all brand new records? or will most of the data after the first be already in there and need to be updated? I learned that the best way to parse it is to set up a upload once (using cffile) and then every week, load the csv file into an array and then compare it with the data already there. Of course, if it's always new data, see if you can convince them to do uploads every night so that you can handle the data without to much hit on your server. Les Mizzell wrote: > I've got a vendor that's going to FTP product data files directly into a > folder on a site. From there, I need to be able to run several different > queries on this to come up with a list of PIN numbers which then get > used elsewhere. > > I got my first glimpse at one of the 'files' yesterday. It's a 75 to 80 > meg CSV text file! They FTP a new one once a week. > > What would be considered "best practice" in handling this file so i can > run queries on it? Import the needed bits into a database (still a big > chunk of data)? Something else? > > You can't directly query a CSV file, correct? > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| CF 8 â Scorpio beta now available, easily build great internet experiences â Try it now on Labs http://www.adobe.com/cfusion/entitlement/index.cfm?e=labs_adobecf8_beta Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:280767 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
