I don't know if that's the best subject line.. I posted some time ago
about how I had a script that ran for two days building a flat file... I
originally thought it might be the query, but a little <CFFLUSH> told me
that wasn't the case. So the file was taking two days to write! Two days!
It's only 10mb, why was it taking so long?
Something made me think about how I was writing the file. I was
concatenating to the same variable and once the loop was done, I would
write out the single file. I thought why don't I try appending to the file
for each iteration instead of holding the variable in memory.
Here's the code of my test:
<CFSET filestring = "The quick brown fox jumped over the lazy dog. Over
and over again.">
<CFSET iteration = 20000>
<P>Testing Variable->File Loop</P>
<CFFLUSH>
<CFSET start1 = GetTickCount()>
<CFSET filecontent = "First Line Header Content Fieldnames, etc.">
<CFLOOP FROM="1" TO="#iteration#" INDEX="i">
<CFSET filecontent = filecontent & filestring & CHR(13) & CHR(10)>
</CFLOOP>
<CFFILE ACTION="WRITE" FILE="d:\temp\looptest1.txt" OUTPUT="#filecontent#"
ADDNEWLINE="No">
<CFSET end1 = GetTickCount()>
<CFSET loop1 = end1-start1>
<CFOUTPUT><P>Done. #DecimalFormat(loop1)# ticks.</P></CFOUTPUT>
<P>Testing File Loop</P>
<CFFLUSH>
<CFSET start2 = GetTickCount()>
<CFSET filecontent = "First Line Header Content Fieldnames, etc.">
<CFFILE ACTION="WRITE" FILE="d:\temp\looptest2.txt" OUTPUT="#filecontent#"
ADDNEWLINE="Yes">
<CFLOOP FROM="1" TO="#iteration#" INDEX="i">
<CFFILE ACTION="APPEND" FILE="d:\temp\looptest2.txt"
OUTPUT="#filestring#" ADDNEWLINE="Yes">
</CFLOOP>
<CFSET end2 = GetTickCount()>
<CFSET loop2 = end2-start2>
<CFOUTPUT><P>Done. #DecimalFormat(loop2)# ticks.</P></CFOUTPUT>
The results:
Iteration Loop1 Loop2
1000 900 400
10000 172k 5k
20000 764k 10k
Wow. I was quite surprised. At first, I thought writing the file everytime
would be slower (all the disk i/o), but I guess that's not a big deal
compared to holding the every growing variable in memory and adding to it
each time. It would seem that for each iteration of the loop it takes
longer and longer to add something to the end of the variable.
Hope that helps someone. Sure taught me a thing or two!
In my real script which last took 46 hours (!), rewriting it to write one
line at time, it nows take only 1 hour. Yay!
Tony Schreiber, Senior Partner Man and Machine, Limited
mailto:[EMAIL PROTECTED] http://www.technocraft.com
http://www.simplemessageboard.com ___Free Forum Software for Cold Fusion
http://www.is300.net ___________The Enthusiast's Home of the Lexus IS300
http://www.digitacamera.com ______________DigitA Camera Scripts and Tips
http://www.linklabexchange.com _____________Miata Link ECU Data Exchange
______________________________________________________________________
Structure your ColdFusion code with Fusebox. Get the official book at
http://www.fusionauthority.com/bkinfo.cfm
FAQ: http://www.thenetprofits.co.uk/coldfusion/faq
Archives: http://www.mail-archive.com/[email protected]/
Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
