This whole discussion prompted two blog entries...
Regarding the javaCSV library:
http://www.opensourcecf.com/1/2008/06/JavaCSV-for-creating-large-CSV-and-other-delmiited-files-with-Coldfusion.cfm
*or http://tinyurl.com/58o7ox*
**
Regarding my cfsavecontent performance tests:
To: CF-Talk
Subject: RE: CSV Generation MEMORY SUCK
(Sorry, I got a little CTRL-Enter happy and sent before I was ready...)
Building up strings in cfsavecontent also concatenates to the result
variable so the problem is the same.
=
Hmm, I don't think
This whole discussion prompted two blog entries...
Regarding the javaCSV library:
http://www.opensourcecf.
com/1/2008/06/Ja
vaCSV-for-creating-large-CSV-and-other-delmiited-files-with-Coldfusion.
cfm
*or http://tinyurl.com/58o7ox*
**
Regarding my cfsavecontent performance tests:
That's pretty cool, Larry. I was wondering about BD and Smith.
Will J2EE BD let you create the java.lang.runtime object to get memory
usage etc? If so, I would be interested in seeing the results of my
version of the test which reported the memory increase for each test.
(I posted the code
Here are the results of your code (again BD for J2EE running on JBoss AS 4.22):
string string: 33251ms
String Length: 39
cfsavecontent: 62ms
String Length: 570006
I ran the test several times, mainly because the results for cfsavecontent
looked so much like an outlier, but I got similar
Here are the results of your code with java.lang.runtime. Forgot to mention
that the JVM is 1.5.0_15-b04.
Memory Before: 28 Megs
string string: 99642ms
String Length: 65
Memory After: 91 Megs -- Increase of 63 Megs
Memory Before: 29 Megs
cfsavecontent: 63ms
String Length: 850003
Memory
Just ran the same code on Open BlueDragon. NThis test probably is not the
equivalent of the previous tests, at home here I'm running this app on a
MacBook (core duo 2.16 ghz with 2 gb RAM), running OSX 10.4 Tiger. J2SE 5. But
the results are similar:
Memory Before: 26 Megs
string string:
Thanks Larry.
~Brad
-Original Message-
From: Larry Lyons [mailto:[EMAIL PROTECTED]
Sent: Wednesday, June 04, 2008 1:34 PM
To: CF-Talk
Subject: Re: CSV Generation MEMORY SUCK
Here are the results of your code with java.lang.runtime. Forgot to
mention that the JVM is 1.5.0_15-b04
On Monday 02 Jun 2008, Rick Root wrote:
I found a nice little java class library called JavaCSV that handles all
the file writing and dropped my time from 68 seconds to 18 seconds. That
has potential!
Why CF can't translate '' to a StringBuffer append I'll never know...
--
Tom Chiverton
-
From: Brian Kotek [mailto:[EMAIL PROTECTED]
Sent: Tuesday, June 03, 2008 10:12 AM
To: CF-Talk
Subject: Re: CSV Generation MEMORY SUCK
Probably because it can't know if that's what you actually want to do.
We
probably need a new function StringAppend or something that would be
able to
do this. Might
On Tuesday 03 Jun 2008, Brian Kotek wrote:
Probably because it can't know if that's what you actually want to do. We
probably need a new function StringAppend or something that would be able
to do this. Might be time to hit the wish list! ;-)
I'm leaving for Scotch on the Rocks in ~12 hours,
Probably because it can't know if that's what you actually want to do. We
probably need a new function StringAppend or something that would be able to
do this. Might be time to hit the wish list! ;-)
On Tue, Jun 3, 2008 at 4:36 AM, Tom Chiverton [EMAIL PROTECTED]
wrote:
On Monday 02 Jun 2008,
Generation MEMORY SUCK
Probably because it can't know if that's what you actually want to do.
We
probably need a new function StringAppend or something that would be
able to
do this. Might be time to hit the wish list
Good to know.
What is your source of this information?
~Brad
From: Brian Kotek [mailto:[EMAIL PROTECTED]
Sent: Tuesday, June 03, 2008 11:11 AM
Building up strings in cfsavecontent also concatenates to the result
variable so the problem is the same.
On Tue, Jun 3, 2008 at 11:16 AM, Brad
where a fair chunk of the
CF9 team are hosting a BOF session :-)
That was a fun and ruckus BOF session at CF.Objective()!
Wil Genovese
~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to
date
Get
Hmm, I don't think you are correct Brian. I just whipped up a test of
string concatenation.
Please spare the proper load test flames. This is NOT a load test--
it is intended to make a process run long enough to capture a thread
stack. Actually, in the context of large file generations I
-Original Message-
From: Brad Wood
Sent: Tuesday, June 03, 2008 11:45 AM
To: Brad Wood; 'cf-talk@houseoffusion.com'
Subject: RE: CSV Generation MEMORY SUCK
Here are the results:
string string: 17093ms
String Length: 39
cfsavecontent: 125ms
String Length: 39
a slower server too!
string string: 9141ms
String Length: 39
cfsavecontent: 31ms
String Length: 39
-Original Message-
From: Brad Wood
Sent: Tuesday, June 03, 2008 11:45 AM
To: Brad Wood; 'cf-talk@houseoffusion.com'
Subject: RE: CSV Generation MEMORY SUCK
Here are the results
PM
To: CF-Talk
Subject: Re: CSV Generation MEMORY SUCK
Did you compare the memory usage by chance?
G
On Tue, Jun 3, 2008 at 1:07 PM, Brad Wood [EMAIL PROTECTED]
wrote:
~|
Adobe® ColdFusion® 8 software 8 is the most important
Ok, here are my memory usage stats on CF 7. Someone please correct me
if my code is wrong. It's a little messy, and I apologize for that.
Memory Before: 83 Megs
string string: 52795ms
String Length: 65
Memory After: 101 Megs -- Increase of 17 Megs
Memory Before: 85 Megs
cfsavecontent:
I did a million loops - I don't know what possessed me to do that.
Memory was measured using task manager. Totally unscientific.
I did a restart on the service before each trial.
CF 8 developer
2 gig ram
Java v. 1.6.0_01
cfsavecontent
2281 ms
192,356 k start
260,872K after
68.516 k difference
Wow, I just came back to this thread.
REALLY makes me wonder how they're handling cfsavecontent!
~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to
date
Get the Free Trial
+10=55
For the life of me I can't figure out what the Big-O notation would be
for that though...
~Brad
-Original Message-
From: Gerald Guido [mailto:[EMAIL PROTECTED]
Sent: Tuesday, June 03, 2008 1:44 PM
To: CF-Talk
Subject: Re: CSV Generation MEMORY SUCK
I did a million loops - I don't
Well, since we're all conducting our own little tests, here's MY test code:
the cfset method took 64 seconds. The cfsavecontent method only takes
203ms.
It has GOT to be using a stringbuffer then converting the result to a string
at the end.
cfsetting enablecfoutputonly=yes
cfsetting
Dang closed source apps-- if only we could just go look at the code! :)
~Brad
-Original Message-
From: Rick Root [mailto:[EMAIL PROTECTED]
Sent: Tuesday, June 03, 2008 2:26 PM
To: CF-Talk
Subject: Re: CSV Generation MEMORY SUCK
Well, since we're all conducting our own little tests
Just experience, since I've tried all three options (concatenation,
cfsavecontent, and StringBuffer) and have had the first two generate out of
memory errors while the StringBuffer worked correctly. So while
cfsavecontent may indeed be faster and use less memory, I'm still pretty
sure that the
Just a 411
I found a nice little tute on generating csv's using the StringBuffer class
in ColdFusion
http://www.stillnetstudios.com/2007/03/07/java-strings-in-coldfusion/
--
The important thing in science is not so much to obtain new facts as to
discover new ways of thinking about them.
- Sir
(Sorry, I got a little CTRL-Enter happy and sent before I was ready...)
Building up strings in cfsavecontent also concatenates to the result
variable so the problem is the same.
=
Hmm, I don't think you are correct Brian. I just whipped up a test of
string
Rick,
What's your DB platform? Are you sure there is not a better non-cf way to
do it?
Mark A. Kruger, CFG, MCSE
(402) 408-3733 ext 105
www.cfwebtools.com
www.coldfusionmuse.com
www.necfug.com
-Original Message-
From: Rick Root [mailto:[EMAIL PROTECTED]
Sent: Monday, June 02, 2008
SQL Server 2005.
I'm open to suggestion. This is part of an application that allows users to
generate CSV files of their own based on their own criteria, so though I'm
open to non-CF solutions, I'm not sure there really would be anyway except
maybe a homegrown java class to handle the work and
and
you'll have to look at running the query in java too, but I but you'll
get something by handling the file better.
Thanks
Mark
-Original Message-
From: Rick Root [mailto:[EMAIL PROTECTED]
Sent: Monday, June 02, 2008 10:33 AM
To: CF-Talk
Subject: Re: CSV Generation MEMORY SUCK
SQL
I know when I had to do this at a previous job I used ArrayAppend to build
each line in the CSV, but I see you are using the string buffer. I had no
performance diffs at the time, so I just stayed with the CF solution. The
one thing I would look at is not using list functions, but instead using
On Mon, Jun 2, 2008 at 10:50 AM, Gaulin, Mark [EMAIL PROTECTED]
wrote:
Also, test you page with the user's query but with the output part
(actually writing the file) commented out... If the page is still slow
and a huge memory hog then the file stuff above won't help much and
you'll have to
#Chr(34)##replace(arguments
.str,chr(34),#chr(34)##chr(34)#,ALL)##Chr(34)#
There is your bottle neck. CF does not like string manipulation on a large
scale. I have tried to parsed large text files before only to watched my dev
box just keel over.
I see two options off the top of my head, let
Use a Java StringBuffer or StringBuilder. Concatenating large strings in CF
is always a memory hog because every single concatenation creates a new
String instance. Check RIAForge, there are CFC libraries that wrap using
these Java classes for exactly this purpose. You'll find memory usage drops
On Monday 02 Jun 2008, Rick Root wrote:
generating the csv (around line 330-340 of the sample code I posted
earlier) took 62 of the 68 seconds.
Why not output the file all at once, rather than a line at a time (scrap lines
~336 - just keeping .append()'ing to a StringBuffer till your done) ?
On Mon, Jun 2, 2008 at 11:40 AM, Gerald Guido [EMAIL PROTECTED]
wrote:
#Chr(34)##replace(arguments
.str,chr(34),#chr(34)##chr(34)#,ALL)##Chr(34)#
There is your bottle neck. CF does not like string manipulation on a large
scale. I have tried to parsed large text files before only to
On Mon, Jun 2, 2008 at 11:42 AM, Tom Chiverton [EMAIL PROTECTED]
wrote:
On Monday 02 Jun 2008, Rick Root wrote:
generating the csv (around line 330-340 of the sample code I posted
earlier) took 62 of the 68 seconds.
Why not output the file all at once, rather than a line at a time (scrap
Didn't look at the code, eh?
On Mon, Jun 2, 2008 at 12:07 PM, Brian Kotek [EMAIL PROTECTED] wrote:
Use a Java StringBuffer or StringBuilder. Concatenating large strings in CF
On Mon, Jun 2, 2008 at 10:03 AM, Rick Root [EMAIL PROTECTED]
wrote:
http://cfm.pastebin.org/40043
--
Rick
On Mon, Jun 2, 2008 at 12:34 PM, Brian Kotek [EMAIL PROTECTED] wrote:
The difference is that you have to use the StringBuffer for everything.
Since you aren't passing the StringBuffer into the CSVFormat method and I
don't see the code for that method, I assume it is still suffering from the
No, I opened it and saw that it was 400 lines long and didn't have time to
go through it all. But sweeping through it quickly, the same advice applies.
The difference is that you have to use the StringBuffer for everything.
Since you aren't passing the StringBuffer into the CSVFormat method and I
I found a nice little java class library called JavaCSV that handles all the
file writing and dropped my time from 68 seconds to 18 seconds. That has
potential!
It basically handles the writing of delimiters and the proper csv
formatting.. so here's my code:
cfset var fileOutput =
dropped my time from 68 seconds to 18 seconds
Nice. Is that entirety of the code sans the query?
little java class library called JavaCSV that handles
The the one from SourceForge? I am going to need something like this
shortly.
G
--
The important thing in science is not so much to
On Mon, Jun 2, 2008 at 1:44 PM, Gerald Guido [EMAIL PROTECTED] wrote:
dropped my time from 68 seconds to 18 seconds
Nice. Is that entirety of the code sans the query?
Entirety. The query itself takes about 4 seconds to execute and return all
its data.
little java class library called
: Rick Root [mailto:[EMAIL PROTECTED]
Sent: Monday, June 02, 2008 9:33 AM
To: CF-Talk
Subject: Re: CSV Generation MEMORY SUCK
SQL Server 2005.
I'm open to suggestion. This is part of an application that allows users to
generate CSV files of their own based on their own criteria, so though I'm
open
-Talk
Subject: Re: CSV Generation MEMORY SUCK
On Mon, Jun 2, 2008 at 10:50 AM, Gaulin, Mark [EMAIL PROTECTED]
wrote:
Also, test you page with the user's query but with the output part
(actually writing the file) commented out... If the page is still slow
and a huge memory hog then the file
46 matches
Mail list logo
