We had a similar issue with an internal application and we made good
improvements using the standard java classes PrintStream,
FileOutputStream, and BufferedOutputStream to handle the writing to your
file.
Something like this (shown in java, so you'd have to wrap it properly
with cfscript):
PrintStream out = new PrintStream(new BufferedOutputStream(new
FileOutputStream("filename", true), bufferSize)); // pick a
decent-sized buffer.. Maybe 100k to start
This will let you do "out.print(...)" and "out.println(...)" from CF
that I think will be much more efficient than what CF can do. Be sure
to do "out.close()" within the page (so use try/catch to be sure that
the close happens).
Also, test you page with the user's query but with the output part
(actually writing the file) commented out... If the page is still slow
and a huge memory hog then the file stuff above won't help much and
you'll have to look at running the query in java too, but I but you'll
get something by handling the file better.
Thanks
Mark
-----Original Message-----
From: Rick Root [mailto:[EMAIL PROTECTED]
Sent: Monday, June 02, 2008 10:33 AM
To: CF-Talk
Subject: Re: CSV Generation MEMORY SUCK
SQL Server 2005.
I'm open to suggestion. This is part of an application that allows
users to generate CSV files of their own based on their own criteria, so
though I'm open to "non-CF" solutions, I'm not sure there really would
be anyway except maybe a homegrown java class to handle the work and be
more strict with memory consumption....
Rick
On Mon, Jun 2, 2008 at 10:13 AM, Mark Kruger <[EMAIL PROTECTED]>
wrote:
> Rick,
>
> What's your DB platform? Are you sure there is not a better "non-cf"
> way to do it?
>
>
> Mark A. Kruger, CFG, MCSE
> (402) 408-3733 ext 105
> www.cfwebtools.com
> www.coldfusionmuse.com
> www.necfug.com
>
> -----Original Message-----
> From: Rick Root [mailto:[EMAIL PROTECTED]
> Sent: Monday, June 02, 2008 9:04 AM
> To: CF-Talk
> Subject: CSV Generation MEMORY SUCK
>
> So I've got a problem with generating large csv files.. it's a memory
suck.
>
> I do this in an event gateway so that these file drops are generated
> "in the background"... here's the gateway code:
>
> http://cfm.pastebin.org/40043
>
> The larger the file drop, the worse the memory suck is. A relatively
> small drop of about 7200 rows and 138 columns (just over 1 million
> pieces of
> data)
> took 68 seconds. In my production environment, I've estimated that I
> can generate between 15,000 and 20,000 pieces of data per second using
> the code above.
>
> The problem is this drop (which only generates a 5MB file) causes a
> memory suck of about 100MB...
>
> Take a look at this output from the server monitor:
> www.it.dev.duke.edu/public/temp.rtf
>
> It shows the memory graph generated from two file drops, at 9:38 and
> 9:45 am... the first one spiked the memory from 70MB to 170MB...the
> second one dropped the memory back to about 90MB and then spiked it to
140MB.
>
> Of course, this size drop is not what causes my concern, it's when
> people are dropping 10x that amount.. say 80,000 rows at 130 columns.
> Over 10 million pieces of data, would take nearly 9 minutes ASSUMING
> you had no memory issues, which I would. Such a drop would basically
> crash the instance.
>
> --
> Rick Root
> New Brian Vander Ark Album, songs in the music player and cool behind
> the scenes video at www.myspace.com/brianvanderark
>
>
>
>
>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to
date
Get the Free Trial
http://ad.doubleclick.net/clk;192386516;25150098;k
Archive:
http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:306570
Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
