hi all, I have a data processing app that's on a dedicated server, will run once in a long while, to import data into SQL, read with CF and export as XML. Just 1 page to process, with various includes and CFCs. Has been running fine with previous datasets, except with newest huge datasets have been running out of memory. Total size of data is a few hundred MB
Server has 4 GB RAM. CF 7.0.1, CHF2, and SQL 2005 I increased the max heap size in jvm.config to 1500 MB -Xms512m -Xmx1500m -XX:MaxPermSize=128m -XX:PermSize=64m -XX:NewSize=48m (tried -XX:+UseConcMarkSweepGC , CF wouldn't start) With new dataset, Jrun goes up to the 1500 MB limit, and fails. Can anyone look over this list and see if there's anything else I could do for memory? Data and some processing comes from CFC methods, with CFCs (9) cached in application scope. Only select the data that seems valid, and which I need for each part. All local variables are var'ed, especially for the queries, (except for one method, which I've fixed), and even local var'ed the cfquery <cfset var cfquery = ""> at top of any method that has a cfquery in it (in addition to the query name itself). The only data I cache is a small array of system data in one of them. Parts of data are generated in separate templates, and save XML for that part to disk, and clear out the XML for that part and query variables from memory. Each part is now a custom tag, which should throw away all queries and local vars, and I force a garbage collection (have a module that caches a java.lang.Runtime object in session scope, optionally prints the mem usage, and does a gc()) The only data I'm accumulating are lists of bad IDs and an array of structs containing data issues (with IDs, and error message). Did a simple check on how big those get (only simple strings, arrays and structures, so loop over recursively and add up sizes with an extra fudge factor for each struct key and array element) -- on the order of tens of megabytes (has generally been < 10, based on relative size should be no more than 30 or so), not hundreds. I'll try next to store these list of IDs in the DB, and use it to limit the data queries for the later parts. Q: 1. Can I rely on the garbage collection to reclaim all local variables? 2. Is there a more reliable way of finding the size of an array of structures? 3. Any other ideas? Thanks, - David ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:249745 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
