For that much data I am not sure I'd want it all loaded into CF vars, nor passed back and forth using lots and lots of queries.
You may be better off loading the entire thing into an import table in your database, and then using process on thee DBMS to do the synchronization and/or analysis work (a scheduled stored procedure, a trigger on INSERT on that import table, or something like that). --- Ben -----Original Message----- From: Phillip M. Vector [mailto:[EMAIL PROTECTED] Sent: Tuesday, June 12, 2007 10:06 AM To: CF-Talk Subject: Re: Large CSV File Is it all brand new records? or will most of the data after the first be already in there and need to be updated? I learned that the best way to parse it is to set up a upload once (using cffile) and then every week, load the csv file into an array and then compare it with the data already there. Of course, if it's always new data, see if you can convince them to do uploads every night so that you can handle the data without to much hit on your server. Les Mizzell wrote: > I've got a vendor that's going to FTP product data files directly into a > folder on a site. From there, I need to be able to run several different > queries on this to come up with a list of PIN numbers which then get > used elsewhere. > > I got my first glimpse at one of the 'files' yesterday. It's a 75 to 80 > meg CSV text file! They FTP a new one once a week. > > What would be considered "best practice" in handling this file so i can > run queries on it? Import the needed bits into a database (still a big > chunk of data)? Something else? > > You can't directly query a CSV file, correct? > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Deploy Web Applications Quickly across the enterprise with ColdFusion MX7 & Flex 2 Free Trial http://www.adobe.com/products/coldfusion/flex2/?sdid=RVJU Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:280761 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
