Re: Sending mass, yet personalised email newsletters using Django
On Tue, 11 Nov 2008 00:27:06 -0800, [EMAIL PROTECTED] wrote: > Hello Patrick, > > On Nov 11, 1:53 am, "Patrick J. Anderson" <[EMAIL PROTECTED]> > wrote: >> >> Some of the main requirements are: >> >> * Personalised HTML/text messages >> * Manual delivery twice a week >> * Message logging/archiving >> * Amount of newsleter issues: ~50 >> * Amount of subscribers: 500-5000 > > We (as company) has created 6 different versions of Newsletter for our > customers. They are > all commercial, but we plan to release some non-commercial version too, > as part of our e-commerce > solution, djWarehouse. > > What about the Newsletter, I can explain the architecture for you, so > you would not fall into problems which had us to rewrite our Newsletter > 6 times until we get satisfied with results. > > > We have the following models: > Newsletter (name, added, sent, status (NEW, READY, SCHEDULING, > SENDING, SENT) > MessageSet (name, newsletter, category, status (same as in > newsletter)) > Message (messageset foreign key, marker_tag, subject, from name, from > email, format, message) > MessageCopy(user, messageset, message) MessageResponse(message, url, > count) > > This newsletter is used on some of our sites, with thousands of > subscribers. (10-20). The logick is the following: customers > are subscribed to some specific categories on site. We have choice to > send general newsletter to all customers, or customize messages by > categories. MesageSet - a number of messages sent to specific > category[ies]. Every newsletter can contain a few message sets. Why so > complicated? Because we need to know what information is more > interesting to end users. Counting the results in MessageResponse we > find out how many customers clicked the links in one or another > newsletter, and know what customers are more interested in - this data > is used for next monthly newsletter from the site. > > Now, the process of sending newsletter. That is one of the most > difficult parts of the process, since the SMTP protocol and message > daemons are not allowing to do many things, so we have to organize > things ourself. For example, since our sites have such a large users > base, Yahoo, Gmail, and specifically Hotmail were blocking our legal > newsletter every time, because from a few hundreds thousands > subscribers, a large part belongs to those online services (a few > thousands) and those services will automatically put you in "block list" > as soon as you try to send messages too quick. > > That is why, we don't just send mails out via SMTP. The process of > sending is the following: > 1. When admin press "send" newsletter button, this only triggers the > process to save data to MesageCopy. > 2. An external program, run in 'daemon' mode, watches the table, and > maintains the list > of exeternal web services (Yahoo, MSN, etc). This daemon controls > that SMTP send out rate should > not exceed some specific amount for specific service. > > So far, this architecture works out the best. We are using this 6th > version of improved newsletter for last 1.5 years, without any problems. > > I can not post the whole code here because the program is commercial, > but... some smaller version will be released soon. Anyway, most of > customers need to only send mails to a few thousands customers, not a > few hundreds thousands, so some advanced functions are just not needed > for an average customer. > > Hope that information is useful. > > Best Regards. > Alex Thanks for your comments and tips. I was actually thinking already about doing it with javascript by sending AJAX requests. I was considering some sort of cron jobs, but this would not be my preference. With sending emails via AJAX requests one consideration was to time it so sending messages doesn't overwhelm Apache. My solution to this would be to segment the list by region, since I will have that information. This should split the load somewhat. But I'd have to test it first to see what kind of burden it'll put on the server. The other main issue is rendering and creating about 1000 or more messages in the database with a click of a button. This could be a bottleneck, but perhaps I could segment that as well. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Sending mass, yet personalised email newsletters using Django
I would like to share with you some different ideology... I don't like to use any external software to work aside with main application. Very often, it's just imposible to use cron on client's hyper-hostings. The idea is simple, use ajax. I got very simple mechanism that uses some javascript to send asynchronous requests to the sending script. I store newsletters prepared earlier(with tinymce) in one table, subscribers (split in groups) in another and smtp accounts in another. To send emails, user have to prepare sending session. He choose smtp account, subscribers group and newsletter from combobox'es and click prepare. combined id's populate session table in amount of rows equal to emails in chosen group. with little optimalization, that process is very fast even for hundreds of thousands of emails. finally, user clicks to send emails and little js script send asynchronously a sending request. after every email succesfully sent, it's information in session table is updated to check it as send. pros. - you never lost any email because of timeout - you can show progress bar to user - you may start and stop whole process whenever needed cons - user have to prepare and start sending process manualy. sometimes it may be needed to do it absolutely automatic. I hope it would help --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Sending mass, yet personalised email newsletters using Django
Hello Patrick, On Nov 11, 1:53 am, "Patrick J. Anderson" <[EMAIL PROTECTED]> wrote: > > Some of the main requirements are: > > * Personalised HTML/text messages > * Manual delivery twice a week > * Message logging/archiving > * Amount of newsleter issues: ~50 > * Amount of subscribers: 500-5000 We (as company) has created 6 different versions of Newsletter for our customers. They are all commercial, but we plan to release some non-commercial version too, as part of our e-commerce solution, djWarehouse. What about the Newsletter, I can explain the architecture for you, so you would not fall into problems which had us to rewrite our Newsletter 6 times until we get satisfied with results. We have the following models: Newsletter (name, added, sent, status (NEW, READY, SCHEDULING, SENDING, SENT) MessageSet (name, newsletter, category, status (same as in newsletter)) Message (messageset foreign key, marker_tag, subject, from name, from email, format, message) MessageCopy(user, messageset, message) MessageResponse(message, url, count) This newsletter is used on some of our sites, with thousands of subscribers. (10-20). The logick is the following: customers are subscribed to some specific categories on site. We have choice to send general newsletter to all customers, or customize messages by categories. MesageSet - a number of messages sent to specific category[ies]. Every newsletter can contain a few message sets. Why so complicated? Because we need to know what information is more interesting to end users. Counting the results in MessageResponse we find out how many customers clicked the links in one or another newsletter, and know what customers are more interested in - this data is used for next monthly newsletter from the site. Now, the process of sending newsletter. That is one of the most difficult parts of the process, since the SMTP protocol and message daemons are not allowing to do many things, so we have to organize things ourself. For example, since our sites have such a large users base, Yahoo, Gmail, and specifically Hotmail were blocking our legal newsletter every time, because from a few hundreds thousands subscribers, a large part belongs to those online services (a few thousands) and those services will automatically put you in "block list" as soon as you try to send messages too quick. That is why, we don't just send mails out via SMTP. The process of sending is the following: 1. When admin press "send" newsletter button, this only triggers the process to save data to MesageCopy. 2. An external program, run in 'daemon' mode, watches the table, and maintains the list of exeternal web services (Yahoo, MSN, etc). This daemon controls that SMTP send out rate should not exceed some specific amount for specific service. So far, this architecture works out the best. We are using this 6th version of improved newsletter for last 1.5 years, without any problems. I can not post the whole code here because the program is commercial, but... some smaller version will be released soon. Anyway, most of customers need to only send mails to a few thousands customers, not a few hundreds thousands, so some advanced functions are just not needed for an average customer. Hope that information is useful. Best Regards. Alex --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Sending mass, yet personalised email newsletters using Django
I can't claim best way, but the way we do it is to have a few django models for storing the messages, and some help from the email, and smptlib libraries. Models: Mailserver: model for storing connection information for connecting to a standard smtp server, with mixin class using smtplib to provide smtp connection. BulkMail: Unrendered Template, and M2M relationship to users. Fk to Mailserver. This model generates Queued messages, in batches until all recipients have a message queued (we trickle messages out). QueuedMessage: Rendered message (pickled email.MimeMultipart object), Fkeys to user, Bulkmail (if message was generated by the BulkMailer) and Mailserver. Every 15min a Bulk Mail cron to generate messages into the queue. Depending on options one bulk mail object may trickle out the messages until all are sent. Every minute a cron runs to check for pending QueuedMessage objects, and sends them to a real smtp server in batches [something like mailserver_object.process_queue(priority=N,limit=100)] Another cron using poplib to check the sending POP3 account for unsubscribe requests, and for bounces using GNU Mailman (it's python, and pretty simple to throw on your path and use the BouncerAPI for bounce detection). Anything that isn't an unsub or bounce gets cloned/ forwarded to a human read pop account. The crons do some checking using memcache okeys to make sure only one process is doing mail at any one time, and no messages get stuck in a loop. All in all, it's been working great. It takes very little time to generate the messages and lets a real mail server handle the heavy lifting/retries. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Sending mass, yet personalised email newsletters using Django
I'm in the process of writing a simple newsletter managers with a web interface and since I've been a Django user for the past 2 years, I thought of using Django for this task. Some of the main requirements are: * Personalised HTML/text messages * Manual delivery twice a week * Message logging/archiving * Amount of newsleter issues: ~50 * Amount of subscribers: 500-5000 I was wondering about sending personalised mass mail and scaling issues, since I assume that each email message would need to be rendered separately to insert subscribers' first and last name, and other related information and sent using its own SMTPConnection. Can Django handle this task reliably? If so, I'd be interested in best, most efficient ways of doing so from other Django users who have done it before. I'd appreciate your comments, suggestions, tips and any other advice you might offer. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
