SQL Server 2005. I'm open to suggestion. This is part of an application that allows users to generate CSV files of their own based on their own criteria, so though I'm open to "non-CF" solutions, I'm not sure there really would be anyway except maybe a homegrown java class to handle the work and be more strict with memory consumption....
Rick On Mon, Jun 2, 2008 at 10:13 AM, Mark Kruger <[EMAIL PROTECTED]> wrote: > Rick, > > What's your DB platform? Are you sure there is not a better "non-cf" way > to > do it? > > > Mark A. Kruger, CFG, MCSE > (402) 408-3733 ext 105 > www.cfwebtools.com > www.coldfusionmuse.com > www.necfug.com > > -----Original Message----- > From: Rick Root [mailto:[EMAIL PROTECTED] > Sent: Monday, June 02, 2008 9:04 AM > To: CF-Talk > Subject: CSV Generation MEMORY SUCK > > So I've got a problem with generating large csv files.. it's a memory suck. > > I do this in an event gateway so that these file drops are generated "in > the > background"... here's the gateway code: > > http://cfm.pastebin.org/40043 > > The larger the file drop, the worse the memory suck is. A relatively small > drop of about 7200 rows and 138 columns (just over 1 million pieces of > data) > took 68 seconds. In my production environment, I've estimated that I can > generate between 15,000 and 20,000 pieces of data per second using the code > above. > > The problem is this drop (which only generates a 5MB file) causes a memory > suck of about 100MB... > > Take a look at this output from the server monitor: > www.it.dev.duke.edu/public/temp.rtf > > It shows the memory graph generated from two file drops, at 9:38 and 9:45 > am... the first one spiked the memory from 70MB to 170MB...the second one > dropped the memory back to about 90MB and then spiked it to 140MB. > > Of course, this size drop is not what causes my concern, it's when people > are dropping 10x that amount.. say 80,000 rows at 130 columns. Over 10 > million pieces of data, would take nearly 9 minutes ASSUMING you had no > memory issues, which I would. Such a drop would basically crash the > instance. > > -- > Rick Root > New Brian Vander Ark Album, songs in the music player and cool behind the > scenes video at www.myspace.com/brianvanderark > > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;192386516;25150098;k Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:306567 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
