Re: [Zope-dev] MailHost Improvements
On Thu, 13 Aug 2009 21:22:07 -0700, Alec Mitchell ap...@columbia.edu wrote: On Wed, Aug 12, 2009 at 9:42 PM, Andreas Jungli...@zopyx.com wrote: On 13.08.09 01:03, Alec Mitchell wrote: Hello, I've been working on making Plone use the standard Zope MailHost in place of the custom Products.SecureMailHost we've been using since Plone 2.1 (See: http://dev.plone.org/plone/ticket/8814). During this process I've run into a couple bugs in the MailHost implementation and I believe it is missing some essential functionality. The most significant issue is that if you call send() with a messageText containing just the message body, and that body has a ':' in it (e.g. a url) the body will be treated as a header and you'll send a nonsense message. The current implementation of send() also puts a fairly large burden on developers who want to generate simple, correctly encoded messages. Finally, send() relies heavily on the long deprecated 'rfc822' and 'mimetools' modules which have been removed from Python 3.0. I've attached a patch that updates MailHost to use the 'email' module for parsing and generating messages. In addition to fixing the issues that I ran across, and maintaining compatibility, it provides a number of new features: * send and sendTemplate accept an optional charset argument. Using this will set the content-type charset, as well as trigger appropriate encoding if needed. * send and sendTemplate accept an optional msg_type argument which will set the content type header for the message. * The messageText, mfrom, mto, and subject arguments may now be unicode or encoded non-ascii strings, provided a charset is given. Any unicode input will be automatically encoded to the provided charset (or the default charset). Headers will be further encoded in compliance with rfc2822. The message body will be further encoded using a transfer encoding determined by the email.Charset registry (e.g. 7bit for us-ascii, quoted-printable for utf-8 and iso8859, base64 for most other encodings). * The messageText argument now accepts email.Message.Message objects directly. I'm attaching a patch that includes these changes as well as tests for all new functionality. I hope to integrate these changes into Zope 2.12 before final release, but would like to hear the opinions of Zope developers before committing. Though these are fairly significant changes, I believe they provide very useful functionality as well as at least one critical fix, while maintaining 100% compatibility. This comes very, very late. We are pretty close to a release. Please put the changes on the trunk only. We will check after my vacation if we can move it into the 2.12 beta. I've put my latest changes on Zope trunk. All the existing tests pass (with a couple essentially cosmetic modifications in the MailHost tests), and there are 14 new tests which verify both existing and new functionality, as well as the fixed bugs. The new behavior should be identical to the existing behavior when the new charset and msg_type parameters aren't used, with the following exceptions: 1) Passing a message body containing a ':' no longer produces a nonsense email. 2) Providing unicode strings for the text or headers no longer results in a garbage message (it may produce a UnicodeEncodeError though). 3) 8bit (encoded) strings provided as headers will be converted to 7bit, using encoding determined in messageText headers or the default encoding. It would be very helpful to have these changes in Zope 2.12; otherwise, Plone 4.0 will be stuck with our unmaintained SecureMailHost product for yet another release in order to provide equivalent functionality. Moving to a standard Zope MailHost would be a big benefit for Plone, and all Zope users will benefit from the ability to easily send properly formed non-ASCII messages. There's one additional significant change to Zope behavior here that I forgot to mention. The current implementation sets the python default email transfer and header encoding for 'utf-8' messages to 'quoted-printable', it normally defaults to 'base64'. This is essentially a cosmetic change and makes reading and debugging email messages much more straightforward. It also makes encoded mail less likely to be caught by SPAM filters (some of which dislike base64 mail on principle). At present this change causes one test in CMFDefault to fail, which I'm happy to fix. But it's also not a problem to just remove the line that sets the new 'utf-8' encoding, though I think it dpes have some important advantages. Alec ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] MailHost Improvements
On Aug 13, 2009, at 5:12, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 +1 for the approach in general (I haven't looked at the patch in detail). Assuming all tests pass, and that you have a test exercising the ':' bug you describe, I would just commit it on the trunk (I think such a change should be in the upcoming 2.12 beta). +1 for the approach and for adding it to 2.12 if it is backwards- compatible. jens ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] MailHost Improvements
On Wed, Aug 12, 2009 at 9:42 PM, Andreas Jungli...@zopyx.com wrote: On 13.08.09 01:03, Alec Mitchell wrote: Hello, I've been working on making Plone use the standard Zope MailHost in place of the custom Products.SecureMailHost we've been using since Plone 2.1 (See: http://dev.plone.org/plone/ticket/8814). During this process I've run into a couple bugs in the MailHost implementation and I believe it is missing some essential functionality. The most significant issue is that if you call send() with a messageText containing just the message body, and that body has a ':' in it (e.g. a url) the body will be treated as a header and you'll send a nonsense message. The current implementation of send() also puts a fairly large burden on developers who want to generate simple, correctly encoded messages. Finally, send() relies heavily on the long deprecated 'rfc822' and 'mimetools' modules which have been removed from Python 3.0. I've attached a patch that updates MailHost to use the 'email' module for parsing and generating messages. In addition to fixing the issues that I ran across, and maintaining compatibility, it provides a number of new features: * send and sendTemplate accept an optional charset argument. Using this will set the content-type charset, as well as trigger appropriate encoding if needed. * send and sendTemplate accept an optional msg_type argument which will set the content type header for the message. * The messageText, mfrom, mto, and subject arguments may now be unicode or encoded non-ascii strings, provided a charset is given. Any unicode input will be automatically encoded to the provided charset (or the default charset). Headers will be further encoded in compliance with rfc2822. The message body will be further encoded using a transfer encoding determined by the email.Charset registry (e.g. 7bit for us-ascii, quoted-printable for utf-8 and iso8859, base64 for most other encodings). * The messageText argument now accepts email.Message.Message objects directly. I'm attaching a patch that includes these changes as well as tests for all new functionality. I hope to integrate these changes into Zope 2.12 before final release, but would like to hear the opinions of Zope developers before committing. Though these are fairly significant changes, I believe they provide very useful functionality as well as at least one critical fix, while maintaining 100% compatibility. This comes very, very late. We are pretty close to a release. Please put the changes on the trunk only. We will check after my vacation if we can move it into the 2.12 beta. I've put my latest changes on Zope trunk. All the existing tests pass (with a couple essentially cosmetic modifications in the MailHost tests), and there are 14 new tests which verify both existing and new functionality, as well as the fixed bugs. The new behavior should be identical to the existing behavior when the new charset and msg_type parameters aren't used, with the following exceptions: 1) Passing a message body containing a ':' no longer produces a nonsense email. 2) Providing unicode strings for the text or headers no longer results in a garbage message (it may produce a UnicodeEncodeError though). 3) 8bit (encoded) strings provided as headers will be converted to 7bit, using encoding determined in messageText headers or the default encoding. It would be very helpful to have these changes in Zope 2.12; otherwise, Plone 4.0 will be stuck with our unmaintained SecureMailHost product for yet another release in order to provide equivalent functionality. Moving to a standard Zope MailHost would be a big benefit for Plone, and all Zope users will benefit from the ability to easily send properly formed non-ASCII messages. Alec P.S. Andreas, if there's an address where I can send some nice wine to help facilitate the merge into 2.12, let me know ;-) ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] MailHost Improvements
Hello, I've been working on making Plone use the standard Zope MailHost in place of the custom Products.SecureMailHost we've been using since Plone 2.1 (See: http://dev.plone.org/plone/ticket/8814). During this process I've run into a couple bugs in the MailHost implementation and I believe it is missing some essential functionality. The most significant issue is that if you call send() with a messageText containing just the message body, and that body has a ':' in it (e.g. a url) the body will be treated as a header and you'll send a nonsense message. The current implementation of send() also puts a fairly large burden on developers who want to generate simple, correctly encoded messages. Finally, send() relies heavily on the long deprecated 'rfc822' and 'mimetools' modules which have been removed from Python 3.0. I've attached a patch that updates MailHost to use the 'email' module for parsing and generating messages. In addition to fixing the issues that I ran across, and maintaining compatibility, it provides a number of new features: * send and sendTemplate accept an optional charset argument. Using this will set the content-type charset, as well as trigger appropriate encoding if needed. * send and sendTemplate accept an optional msg_type argument which will set the content type header for the message. * The messageText, mfrom, mto, and subject arguments may now be unicode or encoded non-ascii strings, provided a charset is given. Any unicode input will be automatically encoded to the provided charset (or the default charset). Headers will be further encoded in compliance with rfc2822. The message body will be further encoded using a transfer encoding determined by the email.Charset registry (e.g. 7bit for us-ascii, quoted-printable for utf-8 and iso8859, base64 for most other encodings). * The messageText argument now accepts email.Message.Message objects directly. I'm attaching a patch that includes these changes as well as tests for all new functionality. I hope to integrate these changes into Zope 2.12 before final release, but would like to hear the opinions of Zope developers before committing. Though these are fairly significant changes, I believe they provide very useful functionality as well as at least one critical fix, while maintaining 100% compatibility. Thanks, Alec mailhost.patch Description: Binary data ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] MailHost Improvements
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Alec Mitchell wrote: Hello, I've been working on making Plone use the standard Zope MailHost in place of the custom Products.SecureMailHost we've been using since Plone 2.1 (See: http://dev.plone.org/plone/ticket/8814). During this process I've run into a couple bugs in the MailHost implementation and I believe it is missing some essential functionality. The most significant issue is that if you call send() with a messageText containing just the message body, and that body has a ':' in it (e.g. a url) the body will be treated as a header and you'll send a nonsense message. The current implementation of send() also puts a fairly large burden on developers who want to generate simple, correctly encoded messages. Finally, send() relies heavily on the long deprecated 'rfc822' and 'mimetools' modules which have been removed from Python 3.0. I've attached a patch that updates MailHost to use the 'email' module for parsing and generating messages. In addition to fixing the issues that I ran across, and maintaining compatibility, it provides a number of new features: * send and sendTemplate accept an optional charset argument. Using this will set the content-type charset, as well as trigger appropriate encoding if needed. * send and sendTemplate accept an optional msg_type argument which will set the content type header for the message. * The messageText, mfrom, mto, and subject arguments may now be unicode or encoded non-ascii strings, provided a charset is given. Any unicode input will be automatically encoded to the provided charset (or the default charset). Headers will be further encoded in compliance with rfc2822. The message body will be further encoded using a transfer encoding determined by the email.Charset registry (e.g. 7bit for us-ascii, quoted-printable for utf-8 and iso8859, base64 for most other encodings). * The messageText argument now accepts email.Message.Message objects directly. I'm attaching a patch that includes these changes as well as tests for all new functionality. I hope to integrate these changes into Zope 2.12 before final release, but would like to hear the opinions of Zope developers before committing. Though these are fairly significant changes, I believe they provide very useful functionality as well as at least one critical fix, while maintaining 100% compatibility. +1 for the approach in general (I haven't looked at the patch in detail). Assuming all tests pass, and that you have a test exercising the ':' bug you describe, I would just commit it on the trunk (I think such a change should be in the upcoming 2.12 beta). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKg4SS+gerLs4ltQ4RAgOuAJ0W2CNRCDQglY75s0HVKFnChOwbnwCfSEmu ph+wxAWDwjG3J8xZV1Cr6+U= =tZEF -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] MailHost Improvements
On 13.08.09 01:03, Alec Mitchell wrote: Hello, I've been working on making Plone use the standard Zope MailHost in place of the custom Products.SecureMailHost we've been using since Plone 2.1 (See: http://dev.plone.org/plone/ticket/8814). During this process I've run into a couple bugs in the MailHost implementation and I believe it is missing some essential functionality. The most significant issue is that if you call send() with a messageText containing just the message body, and that body has a ':' in it (e.g. a url) the body will be treated as a header and you'll send a nonsense message. The current implementation of send() also puts a fairly large burden on developers who want to generate simple, correctly encoded messages. Finally, send() relies heavily on the long deprecated 'rfc822' and 'mimetools' modules which have been removed from Python 3.0. I've attached a patch that updates MailHost to use the 'email' module for parsing and generating messages. In addition to fixing the issues that I ran across, and maintaining compatibility, it provides a number of new features: * send and sendTemplate accept an optional charset argument. Using this will set the content-type charset, as well as trigger appropriate encoding if needed. * send and sendTemplate accept an optional msg_type argument which will set the content type header for the message. * The messageText, mfrom, mto, and subject arguments may now be unicode or encoded non-ascii strings, provided a charset is given. Any unicode input will be automatically encoded to the provided charset (or the default charset). Headers will be further encoded in compliance with rfc2822. The message body will be further encoded using a transfer encoding determined by the email.Charset registry (e.g. 7bit for us-ascii, quoted-printable for utf-8 and iso8859, base64 for most other encodings). * The messageText argument now accepts email.Message.Message objects directly. I'm attaching a patch that includes these changes as well as tests for all new functionality. I hope to integrate these changes into Zope 2.12 before final release, but would like to hear the opinions of Zope developers before committing. Though these are fairly significant changes, I believe they provide very useful functionality as well as at least one critical fix, while maintaining 100% compatibility. This comes very, very late. We are pretty close to a release. Please put the changes on the trunk only. We will check after my vacation if we can move it into the 2.12 beta. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] MailHost Improvements
There has been intermittent talk of improving mailhost's ability to scale. i just wanted to solicit some comments to some ideas for improving mail host performance. Stage I. Persistent Socket 1. make the smtp connection persistent 2. preface sending mail with a 'noop' operation to verify connection status 3. overide __setstate__ to open connection i'm not sure if this will yield any real world benefit but i like the style better than opening a new connection for every new message. Stage II. Transaction Aware 1. Store sent mail in a volatile 2. on transaction commit begin (tpc_begin), send all mail, if error abort transaction 3. reset volatile again i'm not sure of real world performance, but the design seems better and the added benefit that it will play nicely with the transaction machinery and not send unesc. emails. Stage III. Intelligent Q. aka the BulkMailHost 1. implement mailer thread / sole function to send mails from queue and intelligently manage a outgoing queue and deferred queue. 2. implement extensions to sendmail tag for immediate sending of mail which will wake/signal mailer thread after depositing into outgoing q. extension options are 3. mailer thread on a timer, so it will wake on either a sendmail tag with immeadiate extension or on timer expiration and send mail from queue, messages with error due to connection or placed in deferred queue. messages with undelivarable errors are either deleted or placed in a deleted queue. 4. all queues are viewable and manageable from the management interface. obviously this makes the sendmail tag async to the message being sent. i'm not sure if this something that really should be implemented in a ZODB default file storage due to the high write nature of the system or just dumped to the fs. again this seems questionable also because a proper mta should be operating on a store and send principle (qmail/postfix). comments? hmmm... looking through the source of the 2.2.2 mailhost i see a scheduledSend method, anyone know what this is from/for? Cheers Kapil ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
