Eric, I'd return the records in like 50,000 row chunks and process them that way. If you're using SQL Server 2005+ you can use CTEs to paginate the results (but most modern database have a way of doing pagination.)
This will keep the memory footprint lower. -Dan On Friday, July 22, 2011, cft...@ecartech.com <cft...@ecartech.com> wrote: > > I'm trying to create a Solr collection from a query that returns about 3 > million records. The query itself takes about 3 minutes to run in SQL > Studio. When trying to run it in CF to create my collection, I get > java.lang.OutOfMemoryError: Java heap space every time. Here's what I've > tried so far, with no luck: > > - Modified Solr Server Settings in CF Admin: set Solr Buffer Limit = 80 > - Set Solr.lax file to -Xms1024m -Xmx2048m -XX:MaxPermSize=256m > -XX:-UseConcMarkSweepGC > - Set my CF JVM to -Xmx2048m > - Set Solr lax.nl.current.vm to use the 64 bit javaw.exe > > Is there anything else I can try or should be looking for? At this point > I'm thinking, if CF & Solr can't create my collection using 2 gigs of ram > apiece and a 64 bit jvm, I'm not so sure I want to use it. > > Any suggestions? > > Thanks, > > Eric Cobb > http://www.cfgears.com > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:346317 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
