That's great Ben, just what I needed. Funnily enough, I'd made a start on
this last night in between baby feeds (I have 3 month old twins) but that
looks great.
Cheers, and have a great weekend!
Will
-----Original Message-----
From: Ben Nadel [mailto:[EMAIL PROTECTED]
Sent: 16 December 2006 00:22
To: CF-Talk
Subject: RE: splitting large csv file into smaller parts
Will,
I had some time to throw something together. This should take care of the
out of memory issues as it doesn't read in the whole file at once:
http://www.bennadel.com/blog/436-Breaking-Enormous-CSV-Files-Into-Smalle
r-CSV-Files.htm
(OR http://bennadel.com/index.cfm?dax=blog:436.view)
This splits the CSV file into many smaller files. Then, you would run your
originally parser on the smaller files, perhaps one per page request. This
should not kill your memory at all. Just remember to delete the smaller CSV
files once you are done with them (or they might corrupt the next night's
CSV data).
Have a great weekend.
.......................
Ben Nadel
Certified Advanced ColdFusion MX7 Developer www.bennadel.com
Need ColdFusion Help?
www.bennadel.com/ask-ben/
-----Original Message-----
From: Will Swain [mailto:[EMAIL PROTECTED]
Sent: Friday, December 15, 2006 4:23 PM
To: CF-Talk
Subject: RE: splitting large csv file into smaller parts
Well, my test has just run into problems and thrown a
java.lang.OutOfMemoryError error.
I'm thinking though maybe I can use the LineNumberReader code to split the
csv file into smaller parts. Maybe take 5,000 rows at a time and write them
out to a new csv file.
Then I can just run the processing code that I already have on each smaller
file. Does that logic seem sound to you?
Cheers
Will
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Create robust enterprise, web RIAs.
Upgrade & integrate Adobe Coldfusion MX7 with Flex 2
http://ad.doubleclick.net/clk;56760587;14748456;a?http://www.adobe.com/products/coldfusion/flex2/?sdid=LVNU
Archive:
http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:264245
Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
