RE: CFThread Memory Issues sending emails
Thanks very much for this great response. I'm going to look at implementing the idea you had for sending things out in batches. I've got another big send to do this week so we'll see how it goes! -Original Message- From: Bobby Hartsfield [mailto:bo...@acoderslife.com] Sent: 21 May 2011 14:25 To: cf-talk Subject: RE: CFThread Memory Issues sending emails For a quick fix, I'd split those up into smaller batches. For instance, your cfthread could process 1000 emails (or whatever number you decide is manageable by your configuration) id make the count a parameter that is easily passed in via URL so all you'd have to do is change the url var in a scheduled task to increase/decrease the count. Then the thread's process would be something like... 1) Grab top X records that need to be sent. 2) Send the emails for those top X records 3) Update those records to indicate that they have been processed so they arent grabbed again by step 1. 4) Stop. Then just schedule it to run every minute or so. If you use timestamps as your step 3 indicator, you still have the start and stop times once there are no emails left to send. For a more elaborate solution, id queue individual emails up into a DB table and have multiple threads process them one at a time until they were done (makeshift CF message queue). I've found that letting a thread process 100s of thousands of small items individually makes that thread much happier than if it had to process even a fraction of that amount all at once. That said, I would have to agree and think your described setup should have zero problem with sending that many emails in a single loop. Have you monitored CF to see how the thread was doing? Maybe watch jconsole memory stats as the thread is running to see if it is indeed climbing rapidly and causing the GC? Maybe the thread is being spawned more than once? Maybe you could try loading all of the used variables into a single struct within that thread so you can clear that struct at the end of the loop? I'm just throwing out ideas here... .:.:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com http://cf4em.com -Original Message- From: Firewall [mailto:firew...@cc.uk.com] Sent: Thursday, May 19, 2011 7:38 AM To: cf-talk Subject: CFThread Memory Issues sending emails This one has me really stumped. I'm running a pretty simple cfmail script to send out email messages to a user list. Up until now I was simply doing it with a browser reload but the client recently doubled the number of emails they want to send to about 57,000. So I decided to implement this with cfthread wrapped around my loop containing the cfquery and cfmail code. The idea being that the server simply returns the page to the browser and kicks of a massive send job in the background. Once the mail script completes an email is send to the user saying I sent your email to X people, I started at xx:xx and finished at xx:xx My testing worked perfectly and the client used it on smaller email jobs without a problem, I tried it for the first time yesterday on the main massive list and about a third of the messages were sent out before the server seemed to hang. Looking at the logs I saw loads of JVM errors referring to GC aka the garbage collector. The specific log entry was: java.lang.OutOfMemoryError: GC overhead limit exceeded The server seems to have no problems sending about 10-15k emails using this process but when I try it on this one big list it results in all kinds of problems so I'm wondering if this is an issue with cfthread, JVM, CF 8.0.1 or my code. I'm wondering if this anyone else has tried to do this and seen the same problem; and if so how did you go about solving it? My setup is Coldfusion 8.0.1 (64bit) running on Red Hat Enterprise 5, the server itself is a nearly new core i7 Dell PowerEdge with Raid SSD hard drives and 12Gb RAM; in other words it should look at 57k emails and just laugh at the smallness of the list :) I've included the outline of my send code just in case someone spots something really stupid in my code: cfthread name=sendEmail#Form.id# action=run cfset startingTime = '#timeformat(now(), HH:MM:SS)#' cfquery datasource=#Application.ds# name=gn Grab info to send in email /cfquery cfloop list=#Form.sendToList# index=e cfoutput cfquery datasource=#Application.ds# name=gu Grab user details based on ID contained in e /cfquery cftry cfmail to=#emailSendTo# from=#gn.addName# #gn.addEmail# subject=#gn.msgSubj# type=html cfinclude template=email-body-#gn.emailType#.cfm
RE: CFThread Memory Issues sending emails
For a quick fix, I'd split those up into smaller batches. For instance, your cfthread could process 1000 emails (or whatever number you decide is manageable by your configuration) id make the count a parameter that is easily passed in via URL so all you'd have to do is change the url var in a scheduled task to increase/decrease the count. Then the thread's process would be something like... 1) Grab top X records that need to be sent. 2) Send the emails for those top X records 3) Update those records to indicate that they have been processed so they arent grabbed again by step 1. 4) Stop. Then just schedule it to run every minute or so. If you use timestamps as your step 3 indicator, you still have the start and stop times once there are no emails left to send. For a more elaborate solution, id queue individual emails up into a DB table and have multiple threads process them one at a time until they were done (makeshift CF message queue). I've found that letting a thread process 100s of thousands of small items individually makes that thread much happier than if it had to process even a fraction of that amount all at once. That said, I would have to agree and think your described setup should have zero problem with sending that many emails in a single loop. Have you monitored CF to see how the thread was doing? Maybe watch jconsole memory stats as the thread is running to see if it is indeed climbing rapidly and causing the GC? Maybe the thread is being spawned more than once? Maybe you could try loading all of the used variables into a single struct within that thread so you can clear that struct at the end of the loop? I'm just throwing out ideas here... .:.:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com http://cf4em.com -Original Message- From: Firewall [mailto:firew...@cc.uk.com] Sent: Thursday, May 19, 2011 7:38 AM To: cf-talk Subject: CFThread Memory Issues sending emails This one has me really stumped. I'm running a pretty simple cfmail script to send out email messages to a user list. Up until now I was simply doing it with a browser reload but the client recently doubled the number of emails they want to send to about 57,000. So I decided to implement this with cfthread wrapped around my loop containing the cfquery and cfmail code. The idea being that the server simply returns the page to the browser and kicks of a massive send job in the background. Once the mail script completes an email is send to the user saying I sent your email to X people, I started at xx:xx and finished at xx:xx My testing worked perfectly and the client used it on smaller email jobs without a problem, I tried it for the first time yesterday on the main massive list and about a third of the messages were sent out before the server seemed to hang. Looking at the logs I saw loads of JVM errors referring to GC aka the garbage collector. The specific log entry was: java.lang.OutOfMemoryError: GC overhead limit exceeded The server seems to have no problems sending about 10-15k emails using this process but when I try it on this one big list it results in all kinds of problems so I'm wondering if this is an issue with cfthread, JVM, CF 8.0.1 or my code. I'm wondering if this anyone else has tried to do this and seen the same problem; and if so how did you go about solving it? My setup is Coldfusion 8.0.1 (64bit) running on Red Hat Enterprise 5, the server itself is a nearly new core i7 Dell PowerEdge with Raid SSD hard drives and 12Gb RAM; in other words it should look at 57k emails and just laugh at the smallness of the list :) I've included the outline of my send code just in case someone spots something really stupid in my code: cfthread name=sendEmail#Form.id# action=run cfset startingTime = '#timeformat(now(), HH:MM:SS)#' cfquery datasource=#Application.ds# name=gn Grab info to send in email /cfquery cfloop list=#Form.sendToList# index=e cfoutput cfquery datasource=#Application.ds# name=gu Grab user details based on ID contained in e /cfquery cftry cfmail to=#emailSendTo# from=#gn.addName# #gn.addEmail# subject=#gn.msgSubj# type=html cfinclude template=email-body-#gn.emailType#.cfm /cfmail cfcatch type=any div style=color:red; Failed send: #gu.email# - #cfcatch.message#br / /div /cfcatch /cftry /cfoutput /cfloop cfset endingTime = '#timeformat(now(), HH:MM:SS)#' cfmail
RE: CFThread Memory Issues sending emails
Sounds about right, I'm currently working on a way to have things split into chunks of 10k :) Thanks anyway! -Original Message- I've found that all sends of email to large lists has to be done in batches. CFMail doesn't really handle massive lists very well. ~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:344741 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
RE: CFThread Memory Issues sending emails
Here's a question for the gurus; if I split a 50k mass email into chunks of 10k could I thread each chunk and have 5 threads running simultaneously sending 10k emails each? Has anyone tried this? -Original Message- From: Edward Chanter [mailto:firew...@cc.uk.com] Sent: 20 May 2011 09:10 To: cf-talk Subject: RE: CFThread Memory Issues sending emails Sounds about right, I'm currently working on a way to have things split into chunks of 10k :) Thanks anyway! -Original Message- I've found that all sends of email to large lists has to be done in batches. CFMail doesn't really handle massive lists very well. ~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:344742 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
CFThread Memory Issues sending emails
This one has me really stumped. I'm running a pretty simple cfmail script to send out email messages to a user list. Up until now I was simply doing it with a browser reload but the client recently doubled the number of emails they want to send to about 57,000. So I decided to implement this with cfthread wrapped around my loop containing the cfquery and cfmail code. The idea being that the server simply returns the page to the browser and kicks of a massive send job in the background. Once the mail script completes an email is send to the user saying I sent your email to X people, I started at xx:xx and finished at xx:xx My testing worked perfectly and the client used it on smaller email jobs without a problem, I tried it for the first time yesterday on the main massive list and about a third of the messages were sent out before the server seemed to hang. Looking at the logs I saw loads of JVM errors referring to GC aka the garbage collector. The specific log entry was: java.lang.OutOfMemoryError: GC overhead limit exceeded The server seems to have no problems sending about 10-15k emails using this process but when I try it on this one big list it results in all kinds of problems so I'm wondering if this is an issue with cfthread, JVM, CF 8.0.1 or my code. I'm wondering if this anyone else has tried to do this and seen the same problem; and if so how did you go about solving it? My setup is Coldfusion 8.0.1 (64bit) running on Red Hat Enterprise 5, the server itself is a nearly new core i7 Dell PowerEdge with Raid SSD hard drives and 12Gb RAM; in other words it should look at 57k emails and just laugh at the smallness of the list :) I've included the outline of my send code just in case someone spots something really stupid in my code: cfthread name=sendEmail#Form.id# action=run cfset startingTime = '#timeformat(now(), HH:MM:SS)#' cfquery datasource=#Application.ds# name=gn Grab info to send in email /cfquery cfloop list=#Form.sendToList# index=e cfoutput cfquery datasource=#Application.ds# name=gu Grab user details based on ID contained in e /cfquery cftry cfmail to=#emailSendTo# from=#gn.addName# #gn.addEmail# subject=#gn.msgSubj# type=html cfinclude template=email-body-#gn.emailType#.cfm /cfmail cfcatch type=any div style=color:red; Failed send: #gu.email# - #cfcatch.message#br / /div /cfcatch /cftry /cfoutput /cfloop cfset endingTime = '#timeformat(now(), HH:MM:SS)#' cfmail to=#getauthuser()# bcc=e...@cc.uk.com from=Ed e...@cc.uk.com subject=I finished sending your newsletter#gn.msgSubj# sent to #listlen(Form.sendToList)# people Started: #startingTime# Finished: #endingTime# /cfmail /cfthread best wishes, -= Ed Edward Chanter Technical Innovations Manager Collaborative Connections Ltd. http://www.cc.uk.com/ ~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:344644 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
Re: CFThread Memory Issues sending emails
I've found that all sends of email to large lists has to be done in batches. CFMail doesn't really handle massive lists very well. On Thu, May 19, 2011 at 4:38 AM, Firewall firew...@cc.uk.com wrote: The server seems to have no problems sending about 10-15k emails using this process but when I try it on this one big list it results in all kinds of problems so I'm wondering if this is an issue with cfthread, JVM, CF 8.0.1 or my code. I'm wondering if this anyone else has tried to do this and seen the same problem; and if so how did you go about solving it? ~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:344733 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
