Hi Andrew, some of the smaller files we handle are in the order of 5 meg and larger files have been in the order of 12 meg ...we have had major issues with servers timing out on large files - CSV and the dreadful XLS files - depending on who created the files even became a factor i.e errors within the file (at least the Fed. Govt sent us XML files - in some cases 3 XML files to run in sequence each XML file in the order of 5-8 meg each without the server timing out) ... in our case a manual import wasn't a option - a daily or in some case twice daily import of data that updated several tables Rob Smartnumbers Auction site 2004 - 2006 Auctiontrader - 1999 - current Directwholesale - 2001 - current
_____ From: cfaussie@googlegroups.com [mailto:[EMAIL PROTECTED] On Behalf Of Andrew Scott Sent: Thursday, 14 June 2007 10:29 AM To: cfaussie@googlegroups.com Subject: [cfaussie] Re: Large CSV's and server timeouts Rob, I doubt CF will be able to handle a query of this magnitude that grant is looking at, but to split the file up into 1mg chunks could be viable. Andrew Scott Senior Coldfusion Developer Aegeon Pty. Ltd. www.aegeon.com.au Phone: +613 8676 4223 Mobile: 0404 998 273 From: cfaussie@googlegroups.com [mailto:[EMAIL PROTECTED] On Behalf Of Rob Wilson Sent: Thursday, 14 June 2007 10:16 AM To: cfaussie@googlegroups.com Subject: [cfaussie] Re: Large CSV's and server timeouts Hi Grant, we have used http://www.emerle.net/programming/display.cfm/t/cfx_text2query with success Cheers Rob _____ From: cfaussie@googlegroups.com [mailto:[EMAIL PROTECTED] On Behalf Of grant Sent: Thursday, 14 June 2007 9:54 AM To: cfaussie@googlegroups.com Subject: [cfaussie] Large CSV's and server timeouts Hi y'All Here's the scenario: we have a csv file that contains 1.5 million rows and is provided by an external vendor. we need to either (a) parse the csv and insert into a DB, or (b) do lookups directly on the csv. We do not and cannot have SQLloader. The DB is ORA10g. The servers are old and under fairly high load all the time. We've tried splitting the csv into smaller chunks and inserting into the DB and also trying a SQL link straight into the CSV. Anything we try time's out and basically brings down the server. Who can shed light on the most efficient way to solve this problem? Thanks Grant <BR __________ NOD32 2328 (20070613) Information __________ This message was checked by NOD32 antivirus system. http://www.eset.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "cfaussie" group. To post to this group, send email to cfaussie@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cfaussie?hl=en -~----------~----~----~----~------~----~------~--~---
