Along those same lines, it seems save to assume that THIS also creates
a new string isntance every iteration:
<cfloop list="#fields#" index="ThisCol">
Since I know list processing is slow (at least it's slower in CF7)...
and I don't want to be creating 10 million strings... I tried
converting the list to an array and using this functionality instead:
<cfloop from="1" to="#numFields#" index="i" step="1">
and then instead of reference the variable ThisCol,
Didn't help much.
Which made me also realize that this kind of stuff:
<cfset fileOutput.append( resultSet[fieldsArray[i]][resultSet.currentRow] )>
might *ALSO* be creating one or more strings.. ie, evaluating
resultSet[fieldsArray[i]][resultSet.currentRow] and then calling
fileOutput.append()
I also discovered that I was running a couple lines of code (that I
didn't share with the list) once per column instead of once per
record...
It also is apparently faster to write out every line to disk rather
than every 500 lines... that surprised me.. cut my processing time
from 90 seconds to 75 seconds.
With other changes, I've gotten my file generation time down to 46 seconds.
however, it's still using 300+ MB of memory!
Here is my current loop code.. approximately 42 seconds of the the 46
seconds total occur during this loop code:
<cfloop query="resultSet">
<cfset row = resultSet.currentRow>
<cfif NOT JOINT_LABELS OR listFind(ArrayToList(spouses),SPOUSE_ID) is 0>
<cfif JOINT_LABELS AND IS_JOINT>
<cfset ArrayAppend(spouses, SPOUSE_ID)>
</cfif>
<cfset fileOutput.append(START_ROW)>
<cfloop from="1" to="#numFields#" index="i" step="1">
<cfset fileOutput.append(
START_FIELDS[fieldsArray[i]] &
resultSet[fieldsArray[i]][row] &
END_FIELD)>
</cfloop>
<cfset fileOutput.append(END_ROW)>
</cfif>
<cfset fileWrite(fh, fileOutput.toString())>
<!--- <cfset fileOutput.delete(0,fileOutput.length())> --->
<cfset fileOutput.setLength(0)>
</cfloop>
Ultimately, I think I might need to write this datadrop gateway in Java....
--
Rick Root
Check out CFMBB, BlogCFM, ImageCFC, ImapCFC, CFFM, and more at
www.opensourcecf.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Enterprise web applications, build robust, secure
scalable apps today - Try it now ColdFusion Today
ColdFusion 8 beta - Build next generation apps
Archive:
http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:290933
Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm
Unsubscribe:
http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
