Oh, and if you worried about records changing during the process, you can always dump the original query results into a table, then all you need to do is grab the top 50000 rows, process and then delete the top 50000 rows, rinse and repeat.
Then when your batch process is done, just grab any records that changed since the batch process started and process those records again (which should be a relatively small number of records.) -Dan On Friday, July 22, 2011, Dan G. Switzer, II <dswit...@pengoworks.com> wrote: > Eric, > > I'd return the records in like 50,000 row chunks and process them that > way. If you're using SQL Server 2005+ you can use CTEs to paginate the > results (but most modern database have a way of doing pagination.) > > This will keep the memory footprint lower. > > -Dan > > On Friday, July 22, 2011, cft...@ecartech.com <cft...@ecartech.com> wrote: >> >> I'm trying to create a Solr collection from a query that returns about 3 >> million records. The query itself takes about 3 minutes to run in SQL >> Studio. When trying to run it in CF to create my collection, I get >> java.lang.OutOfMemoryError: Java heap space every time. Here's what I've >> tried so far, with no luck: >> >> - Modified Solr Server Settings in CF Admin: set Solr Buffer Limit = 80 >> - Set Solr.lax file to -Xms1024m -Xmx2048m -XX:MaxPermSize=256m >> -XX:-UseConcMarkSweepGC >> - Set my CF JVM to -Xmx2048m >> - Set Solr lax.nl.current.vm to use the 64 bit javaw.exe >> >> Is there anything else I can try or should be looking for? At this point >> I'm thinking, if CF & Solr can't create my collection using 2 gigs of ram >> apiece and a 64 bit jvm, I'm not so sure I want to use it. >> >> Any suggestions? >> >> Thanks, >> >> Eric Cobb >> http://www.cfgears.com >> >> >> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:346318 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
