Chris, I threw together a well-commented example of something that might accomplish what you are talking about. It uses smaller record sets and buffered output via the Java buffered output object. I am no Java coder, so I am not always 100% sure what I am doing, but this might help point you in a particular direction:
http://www.bennadel.com/blog/446-Writing-Enormous-Files-Based-On-Massive -Record-Sets-In-ColdFusion.htm (OR http://bennadel.com/index.cfm?dax=blog:446.view) Let me know if you have any questions. ...................... Ben Nadel Certified Advanced ColdFusion MX7 Developer www.bennadel.com Need ColdFusion Help? www.bennadel.com/ask-ben/ -----Original Message----- From: Chris Ditty [mailto:[EMAIL PROTECTED] Sent: Tuesday, December 26, 2006 2:53 PM To: CF-Talk Subject: Re: Creating text files w/ 1mil + lines WOW. All I can say is duhhhh. This is way over my little CF head. :) On 12/26/06, Jochem van Dieten <[EMAIL PROTECTED]> wrote: > Katz, Dov B \(IT\) wrote: > > You shouldn't need to take memory into consideration if you use a > > java.io.BufferedOutputStream and java.io.PrintWriter etc... > > Yes you should. Let's say we are writing 1.3 million, 20 character records from the database to a .csv. First CF queries the database and buffers the resultset (26 MB). In the proces it needs to convert from the database charset to its internal charset (26 MB, maybe 52 MB). Then you loop over the resultset and generate the .csv which you hold in memory (26 MB). In the process of your 6 million loops you generate 8 bytes of whitespace for each iteration (10.4 MB). Then you write it out. > > Of the total memory consumption of 94.4 MB (120.4 MB if you are unlucky with the charsets), using Java to write it out saves you 26 MB. And that is when we ignore the overhead of the different datatypes, with the overhead the numbers would probably be worse. > > Jochem > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Create robust enterprise, web RIAs. Upgrade & integrate Adobe Coldfusion MX7 with Flex 2 http://ad.doubleclick.net/clk;56760587;14748456;a?http://www.adobe.com/products/coldfusion/flex2/?sdid=LVNU Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:265039 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
