Re: Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

PRs are now open:

   1. Updated test cases (on existing legacy implementation): 
   https://github.com/django/django/pull/18502
   2. Change to modern email API: https://github.com/medmunds/django/pull/2

A few changes of note from the original plan, based on things discovered 
during implementation:

   - I *did* end up deprecating MIMEBase attachments. Modern email has a 
   MIMEPart class that can be used in a similar fashion.
   - Modern email insists that text/* bodies and attachments should end 
   with a newline , and 
   will add one for you if missing. I think that's probably OK, but it affects* 
   a lot of our tests* (which use strings like "Content" that don't include 
   trailing newlines). More details in PR#2 comments, and suggestions are 
   welcome if this seems like a problem.
   - These additional, *undocumented* django.core.mail features will be 
   retired without a deprecation period (but will raise errors on attempted 
   use, and have been identified in the release notes):
  - EmailMessage.mixed_subtype overrides: modern email uses 
  multipart/mixed
  - EmailMultiAlternatives.alternative_subtype overrides: modern email 
  uses multipart/alternative
  - Using a legacy email.charset.Charset object for the undocumented 
  EmailMessage.encoding: modern email doesn't support that. (But 
  EmailMessage.encoding will continue to support string charset names as it 
  has in the past, also still undocumented.)
   
Finally, although Python's modern email API is generally less buggy than 
the legacy API, it does have some issues of its own. (Including a security 
issue in address header generation that was publicly reported 5+ years ago, 
and that probably needs to be resolved before we'd want to merge the second 
PR.)

- Mike


On Saturday, July 6, 2024 at 12:18:22 PM UTC-7 Mike Edmunds wrote:

> Ticket created: https://code.djangoproject.com/ticket/35581
>
> On Friday, July 5, 2024 at 5:27:28 PM UTC-7 Mike Edmunds wrote:
>
>> Thanks to everyone for the feedback so far.
>>
>> It looks like all the responses to date are generally positive, and I 
>> haven't seen any objections, so I'm going to open a ticket.
>>
>> (Of course, additional feedback—positive or negative—and advice is very 
>> much appreciated.)
>>
>> Cheers,
>> Mike
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9ecb3f48-ed85-44a8-863f-d165172cd81cn%40googlegroups.com.

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

> I wish this discussion was on the forum so I could mark Pankaj’s response 
as spam 😅

And here I was thinking it was another +1 (that also gently poked fun at my 
own overly-verbose tendencies). 

Speaking of the forum, I've taken the "combine EmailMessage and 
EmailMultiAlternatives" idea that was dropped from this proposal, and 
incorporated it into a separate proposal that's looking for feedback over 
there: 
https://forum.djangoproject.com/t/proposal-simplify-sending-testing-html-in-emailmessage/32844.
 
(Apologies for the cross-post; I get the impression some of the people with 
opinions on django.core.mail are maybe here on django-developers but not 
the forum. There are also a couple of other interesting email-related 
threads active over there.)

- Mike

On Sunday, July 21, 2024 at 3:59:22 AM UTC-7 Adam Johnson wrote:

> I wish this discussion was on the forum so I could mark Pankaj’s response 
> as spam 😅
>
> On Sun, 21 Jul 2024, at 08:26, Othniel Davidson wrote:
>
> Thank you for the update 
>
> On Sat, Jul 20, 2024 at 15:26 Pankaj Kumar  wrote:
>
> Hi Sir,
>I hope this message finds you well.
>
> I am writing to inform you that we have reviewed and decided to accept the 
> proposal to upgrade django.core.mail to Python's modern email API. We 
> believe that this upgrade will bring significant improvements to our 
> system's performance and maintainability.
>
> I would also like to extend my sincere apologies for the delay in our 
> response. I received the proposal seven days ago, and I regret not being 
> able to get back to you sooner. Thank you for your patience and 
> understanding in this matter.
>
> We are looking forward to the successful implementation of this upgrade 
> and are excited about the enhancements it will bring to our project. Please 
> let us know the next steps and how we can assist in the process.
>
> Thank you once again for your proposal and for your continued support.
>
> Best regards,
>
> - PANKAJ NEEMA
>
>
>
> On Sat, 6 Jul 2024 at 04:00, Mike Edmunds  wrote:
>
> On Friday, June 28, 2024 at 1:07:35 PM UTC-7 Florian Apolloner wrote:
> > Are all of those documented? If not we can simply remove them 
> (especially if the deprecation implementation turns out to be a PITA).
>
> It sort of depends on the definition of "documented." I've dug into 
> real-world usage; results and proposals below.
>
> Incidentally, I thought there was (used to be?) a policy that internal 
> undocumented APIs were fair game for use by third-party libraries, 
> subclassing, etc., so long as they didn't start with an underscore. (But 
> "private" underscore APIs could have breaking changes at any time.) Am I 
> remembering that wrong? Or was internal API stability only guaranteed for 
> patch-level releases?
>
> *Should go through deprecation process* (in my opinion, sorted roughly by 
> real-world usage)
>
>- *BadHeaderError* is mentioned in the docs 
>
> 
>  as 
>potentially raised during sending
>- Exposed in django.core.mail.__all__
>   - Real-world use: ~2300+ cases in GitHub code search 
>   
> ,
>  
>   seems to be primarily error handling 
>   
> 
>   - Proposal: deprecate on import. (Modern Python email API raises 
>   ValueError for newlines in headers. In our deprecation, we'll need 
>   `BadHeaderError = ValueError`—rather than subclassing ValueError—to 
>   preserve behavior of existing error handling code.)
>   - Suggested replacement for docs: change `except BadHeaderError` to 
>   `except ValueError`
>   - *sanitize_address()* is not documented, but is widely used
>- Not exposed in django.core.mail (have to import from 
>   django.core.mail.message)
>   - Real-world use: ~430 cases in GitHub code search 
>   
> ,
>  
>   often in custom EmailBackend implementations 
>   
> 
>  or 
>   for special case handling 
>   
> 
>  that 
>   needs knowledge of how django.core.mail handles addresses.
>   - Proposal: deprecate. (Have to keep implementation around anyway 
>   as part of other deprecated code.)
>   - Suggested replacement for docs: none (it's an internal, 
>   undocumented AP

Re: Proposal to upgrade django.core.mail to Python's modern email API

['Adam Johnson' via Django developers (Contributions to Django itself)]

I wish this discussion was on the forum so I could mark Pankaj’s response as 
spam 😅

On Sun, 21 Jul 2024, at 08:26, Othniel Davidson wrote:
> Thank you for the update 
> 
> On Sat, Jul 20, 2024 at 15:26 Pankaj Kumar  wrote:
>> Hi Sir,
>>I hope this message finds you well.
>> I am writing to inform you that we have reviewed and decided to accept the 
>> proposal to upgrade `django.core.mail` to Python's modern email API. We 
>> believe that this upgrade will bring significant improvements to our 
>> system's performance and maintainability.
>> 
>> I would also like to extend my sincere apologies for the delay in our 
>> response. I received the proposal seven days ago, and I regret not being 
>> able to get back to you sooner. Thank you for your patience and 
>> understanding in this matter.
>> 
>> We are looking forward to the successful implementation of this upgrade and 
>> are excited about the enhancements it will bring to our project. Please let 
>> us know the next steps and how we can assist in the process.
>> 
>> Thank you once again for your proposal and for your continued support.
>> 
>> Best regards,
>> 
>> - PANKAJ NEEMA
>> 
>> 
>> 
>> 
>> On Sat, 6 Jul 2024 at 04:00, Mike Edmunds  wrote:
>>> On Friday, June 28, 2024 at 1:07:35 PM UTC-7 Florian Apolloner wrote:
>>> > Are all of those documented? If not we can simply remove them (especially 
>>> > if the deprecation implementation turns out to be a PITA).
>>> 
>>> It sort of depends on the definition of "documented." I've dug into 
>>> real-world usage; results and proposals below.
>>> 
>>> Incidentally, I thought there was (used to be?) a policy that internal 
>>> undocumented APIs were fair game for use by third-party libraries, 
>>> subclassing, etc., so long as they didn't start with an underscore. (But 
>>> "private" underscore APIs could have breaking changes at any time.) Am I 
>>> remembering that wrong? Or was internal API stability only guaranteed for 
>>> patch-level releases?
>>> 
>>> *Should go through deprecation process* (in my opinion, sorted roughly by 
>>> real-world usage)
>>>  • *BadHeaderError* is mentioned in the docs 
>>> 
>>>  as potentially raised during sending
>>>• Exposed in django.core.mail.__all__
>>>• Real-world use: ~2300+ cases in GitHub code search 
>>> ,
>>>  seems to be primarily error handling 
>>> 
>>>• Proposal: deprecate on import. (Modern Python email API raises 
>>> ValueError for newlines in headers. In our deprecation, we'll need 
>>> `BadHeaderError = ValueError`—rather than subclassing ValueError—to 
>>> preserve behavior of existing error handling code.)
>>>• Suggested replacement for docs: change `except BadHeaderError` to 
>>> `except ValueError`
>>>  • *sanitize_address()* is not documented, but is widely used
>>>• Not exposed in django.core.mail (have to import from 
>>> django.core.mail.message)
>>>• Real-world use: ~430 cases in GitHub code search 
>>> ,
>>>  often in custom EmailBackend implementations 
>>> 
>>>  or for special case handling 
>>> 
>>>  that needs knowledge of how django.core.mail handles addresses.
>>>• Proposal: deprecate. (Have to keep implementation around anyway as 
>>> part of other deprecated code.)
>>>• Suggested replacement for docs: none (it's an internal, undocumented 
>>> API). Or we could say it depends on the use case: just skip it if no longer 
>>> relevant, use something like Python's modern email Address object if you 
>>> need formatting cleanup, or copy sanitize_address() into your code and 
>>> adapt for your needs. (Incidentally, it's already pretty common to copy 
>>> 
>>>  and adapt sanitize_address().)
>>>  • *SafeMIMEText* and *SafeMIMEMultipart* are mentioned in the docs 
>>> 
>>>  as the return type of EmailMessage.message(), but not further documented
>>>• Expo

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Othniel Davidson]

Thank you for the update

On Sat, Jul 20, 2024 at 15:26 Pankaj Kumar  wrote:

> Hi Sir,
>I hope this message finds you well.
>
> I am writing to inform you that we have reviewed and decided to accept the
> proposal to upgrade django.core.mail to Python's modern email API. We
> believe that this upgrade will bring significant improvements to our
> system's performance and maintainability.
>
> I would also like to extend my sincere apologies for the delay in our
> response. I received the proposal seven days ago, and I regret not being
> able to get back to you sooner. Thank you for your patience and
> understanding in this matter.
>
> We are looking forward to the successful implementation of this upgrade
> and are excited about the enhancements it will bring to our project. Please
> let us know the next steps and how we can assist in the process.
>
> Thank you once again for your proposal and for your continued support.
>
> Best regards,
>
> - PANKAJ NEEMA
>
>
>
> On Sat, 6 Jul 2024 at 04:00, Mike Edmunds  wrote:
>
>> On Friday, June 28, 2024 at 1:07:35 PM UTC-7 Florian Apolloner wrote:
>> > Are all of those documented? If not we can simply remove them
>> (especially if the deprecation implementation turns out to be a PITA).
>>
>> It sort of depends on the definition of "documented." I've dug into
>> real-world usage; results and proposals below.
>>
>> Incidentally, I thought there was (used to be?) a policy that internal
>> undocumented APIs were fair game for use by third-party libraries,
>> subclassing, etc., so long as they didn't start with an underscore. (But
>> "private" underscore APIs could have breaking changes at any time.) Am I
>> remembering that wrong? Or was internal API stability only guaranteed for
>> patch-level releases?
>>
>> *Should go through deprecation process* (in my opinion, sorted roughly
>> by real-world usage)
>>
>>- *BadHeaderError* is mentioned in the docs
>>
>> 
>>  as
>>potentially raised during sending
>>   - Exposed in django.core.mail.__all__
>>   - Real-world use: ~2300+ cases in GitHub code search
>>   
>> ,
>>   seems to be primarily error handling
>>   
>> 
>>   - Proposal: deprecate on import. (Modern Python email API raises
>>   ValueError for newlines in headers. In our deprecation, we'll need
>>   `BadHeaderError = ValueError`—rather than subclassing ValueError—to
>>   preserve behavior of existing error handling code.)
>>   - Suggested replacement for docs: change `except BadHeaderError`
>>   to `except ValueError`
>>
>>   - *sanitize_address()* is not documented, but is widely used
>>   - Not exposed in django.core.mail (have to import from
>>   django.core.mail.message)
>>   - Real-world use: ~430 cases in GitHub code search
>>   
>> ,
>>   often in custom EmailBackend implementations
>>   
>> 
>>  or
>>   for special case handling
>>   
>> 
>>  that
>>   needs knowledge of how django.core.mail handles addresses.
>>   - Proposal: deprecate. (Have to keep implementation around anyway
>>   as part of other deprecated code.)
>>   - Suggested replacement for docs: none (it's an internal,
>>   undocumented API). Or we could say it depends on the use case: just 
>> skip it
>>   if no longer relevant, use something like Python's modern email Address
>>   object if you need formatting cleanup, or copy sanitize_address() into 
>> your
>>   code and adapt for your needs. (Incidentally, it's already pretty
>>   common to copy
>>   
>> 
>>   and adapt sanitize_address().)
>>
>>   - *SafeMIMEText* and *SafeMIMEMultipart* are mentioned in the docs
>>
>> 
>>as the return type of EmailMessage.message(), but not further documented
>>   - Exposed in django.core.mail.__all__
>>

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Pankaj Kumar]

Hi Sir,
   I hope this message finds you well.

I am writing to inform you that we have reviewed and decided to accept the
proposal to upgrade django.core.mail to Python's modern email API. We
believe that this upgrade will bring significant improvements to our
system's performance and maintainability.

I would also like to extend my sincere apologies for the delay in our
response. I received the proposal seven days ago, and I regret not being
able to get back to you sooner. Thank you for your patience and
understanding in this matter.

We are looking forward to the successful implementation of this upgrade and
are excited about the enhancements it will bring to our project. Please let
us know the next steps and how we can assist in the process.

Thank you once again for your proposal and for your continued support.

Best regards,

- PANKAJ NEEMA



On Sat, 6 Jul 2024 at 04:00, Mike Edmunds  wrote:

> On Friday, June 28, 2024 at 1:07:35 PM UTC-7 Florian Apolloner wrote:
> > Are all of those documented? If not we can simply remove them
> (especially if the deprecation implementation turns out to be a PITA).
>
> It sort of depends on the definition of "documented." I've dug into
> real-world usage; results and proposals below.
>
> Incidentally, I thought there was (used to be?) a policy that internal
> undocumented APIs were fair game for use by third-party libraries,
> subclassing, etc., so long as they didn't start with an underscore. (But
> "private" underscore APIs could have breaking changes at any time.) Am I
> remembering that wrong? Or was internal API stability only guaranteed for
> patch-level releases?
>
> *Should go through deprecation process* (in my opinion, sorted roughly by
> real-world usage)
>
>- *BadHeaderError* is mentioned in the docs
>
> 
>  as
>potentially raised during sending
>   - Exposed in django.core.mail.__all__
>   - Real-world use: ~2300+ cases in GitHub code search
>   
> ,
>   seems to be primarily error handling
>   
> 
>   - Proposal: deprecate on import. (Modern Python email API raises
>   ValueError for newlines in headers. In our deprecation, we'll need
>   `BadHeaderError = ValueError`—rather than subclassing ValueError—to
>   preserve behavior of existing error handling code.)
>   - Suggested replacement for docs: change `except BadHeaderError` to
>   `except ValueError`
>
>   - *sanitize_address()* is not documented, but is widely used
>   - Not exposed in django.core.mail (have to import from
>   django.core.mail.message)
>   - Real-world use: ~430 cases in GitHub code search
>   
> ,
>   often in custom EmailBackend implementations
>   
> 
>  or
>   for special case handling
>   
> 
>  that
>   needs knowledge of how django.core.mail handles addresses.
>   - Proposal: deprecate. (Have to keep implementation around anyway
>   as part of other deprecated code.)
>   - Suggested replacement for docs: none (it's an internal,
>   undocumented API). Or we could say it depends on the use case: just 
> skip it
>   if no longer relevant, use something like Python's modern email Address
>   object if you need formatting cleanup, or copy sanitize_address() into 
> your
>   code and adapt for your needs. (Incidentally, it's already pretty
>   common to copy
>   
> 
>   and adapt sanitize_address().)
>
>   - *SafeMIMEText* and *SafeMIMEMultipart* are mentioned in the docs
>
> 
>as the return type of EmailMessage.message(), but not further documented
>   - Exposed in django.core.mail.__all__
>   - Real-world use: ~240 cases in GitHub code search
>   
> 

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

On Saturday, July 6, 2024 at 11:17:35 AM UTC-7 Florian Apolloner wrote:

On Saturday, July 6, 2024 at 12:30:32 AM UTC+2 Mike Edmunds wrote:

Incidentally, I thought there was (used to be?) a policy that internal 
undocumented APIs were fair game for use by third-party libraries, 
subclassing, etc., so long as they didn't start with an underscore. (But 
"private" underscore APIs could have breaking changes at any time.) Am I 
remembering that wrong? Or was internal API stability only guaranteed for 
patch-level releases?


The API stability contract is documented here: 
https://docs.djangoproject.com/en/5.0/misc/api-stability/ -- Everything 
documented is considered part of the API, the rest is private.So yeah, it 
seems like you are remembering that wrong. 


Heh. The policy was changed in ticket 19728 
—prior to that, stable "public 
APIs" included 

 "everything 
documented …, *and *all methods that don't begin with an underscore" 
[emphasis added]. So apparently I haven't looked at Django's api-stability 
docs in over a decade. (And had a really obscure detail get stuck in my 
memory. :-) )

- Mike

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f393cbfe-4e67-488c-9b2d-c31580e2b7a9n%40googlegroups.com.

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

Ticket created: https://code.djangoproject.com/ticket/35581

On Friday, July 5, 2024 at 5:27:28 PM UTC-7 Mike Edmunds wrote:

> Thanks to everyone for the feedback so far.
>
> It looks like all the responses to date are generally positive, and I 
> haven't seen any objections, so I'm going to open a ticket.
>
> (Of course, additional feedback—positive or negative—and advice is very 
> much appreciated.)
>
> Cheers,
> Mike
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a0c2da3e-447e-4459-973e-89b25dbe0e5en%40googlegroups.com.

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Florian Apolloner]

On Saturday, July 6, 2024 at 12:30:32 AM UTC+2 Mike Edmunds wrote:

Incidentally, I thought there was (used to be?) a policy that internal 
undocumented APIs were fair game for use by third-party libraries, 
subclassing, etc., so long as they didn't start with an underscore. (But 
"private" underscore APIs could have breaking changes at any time.) Am I 
remembering that wrong? Or was internal API stability only guaranteed for 
patch-level releases?


The API stability contract is documented 
here: https://docs.djangoproject.com/en/5.0/misc/api-stability/ -- 
Everything documented is considered part of the API, the rest is private.So 
yeah, it seems like you are remembering that wrong. Your suggested 
deprecations look okay. I have no strong feelings on 
`sanitize_address/forbid_multi_line_headers` -- if you want to deprecate 
instead of simply remove that is okay with me.

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/21395dd0-e1f0-4e07-836b-d57c0516365cn%40googlegroups.com.

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Jörg Breitbart]

We recently had quite a struggle with programming a newsletter extension 
for django-cms, mainly around the html-mails with CID embedded images.


I am not deep into the details, still want to express my support for the 
idea to reshape the email abstraction in django, thus +1.


Cheers,
Jörg

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/19bb2be0-b222-44af-8754-2a63d969742c%40netzkolchose.de.


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

Thanks to everyone for the feedback so far.

It looks like all the responses to date are generally positive, and I 
haven't seen any objections, so I'm going to open a ticket.

(Of course, additional feedback—positive or negative—and advice is very 
much appreciated.)

Cheers,
Mike

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/092bfdd6-4161-4d5c-a21b-82e778f98ec7n%40googlegroups.com.

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

About MIMEBase attachments: I propose we continue supporting them in 
Django, without deprecation, for now. We can investigate adding support for 
the equivalent in Python's modern email API later, as a separate proposal.

More details…

On Wednesday, June 26, 2024 at 6:28:06 PM UTC-7 I wrote:
> Tricky bits and things requiring longer discussion…
> *Legacy MIMEBase attachments:* I'll post a separate message about this 
sometime later.

Since 2007 
,
 
Django has supported and documented 

 
using MIMEBase objects in Django's EmailMessage.attachments list. 

Using MIMEBase is necessary for any "complex attachment" that can't be 
simply expressed as filename + content data + mimetype. This includes 
things like text attachments with a different charset than the main 
message, inline images (which require Content-Disposition: inline and 
Content-ID MIME headers), adding params to the Content-Type header, or in 
general anything where you need additional control over the attachment's 
MIME headers and content encoding.

MIMEBase is part of Python's legacy 
 email 
API. The modern email replacement is add_attachment() 
,
 which 
offers everything we need for simple attachments. It also supports complex 
attachments via "contentmanager" kwargs 

 
like `disposition` for the Content-Disposition header, `cid` for 
Content-ID, and `params` for Content-Type extensions. But add_attachment() 
doesn't support MIMEBase.

So as part of this proposal, we'll handle simple filename+content+mimetype 
attachments through modern add_attachment(). I had originally planned to 
treat MIMEBase attachments as deprecated, and find a way to convert them to 
modern add_attachment() kwargs. And I was going to propose a way to somehow 
include add_attachment() kwargs directly in Django's 
EmailMessage.attachments, as the non-deprecated way to specify complex 
attachments. I still think this is the right long-term direction.

But in the interests of limiting scope for the current proposal, I think we 
can treat that as a separate project, and just continue to support MIMEBase 
attachments for now:

   - From what I can tell, it's still possible to attach legacy MIMEBase 
   objects to a modern Python EmailMessage, and get results that work at least 
   as well as they did with entirely legacy APIs.
   - You can also opt into the modern API when creating a MIMEBase object, 
   by passing policy=email.policy.default to its constructor. This might avoid 
   some obscure bugs that occur with the legacy APIs. I think Django's docs 
   should suggest this approach "for improved compatibility" or something like 
   that.
   - When the time comes to support complex Django EmailMessage.attachments 
   using Python's modern add_attachment() kwargs, I think we should also look 
   into supporting Python EmailMessage's add_related() 
   
.
 
   Right now, there's no way to get Django to send a properly constructed 
   

 
   multipart message with html, inline images, and attachments: the inline 
   images end up in the wrong part. Most email clients aren't that picky, and 
   display the inlines anyway (if they handle inline images at all), but it 
   would be helpful to have a way to generate the correct message structure. 
   Just… later.


- Mike

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9aa610dd-01de-4f98-bcfa-f982872ebdban%40googlegroups.com.

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

On Friday, June 28, 2024 at 1:07:35 PM UTC-7 Florian Apolloner wrote:
> Are all of those documented? If not we can simply remove them (especially 
if the deprecation implementation turns out to be a PITA).

It sort of depends on the definition of "documented." I've dug into 
real-world usage; results and proposals below.

Incidentally, I thought there was (used to be?) a policy that internal 
undocumented APIs were fair game for use by third-party libraries, 
subclassing, etc., so long as they didn't start with an underscore. (But 
"private" underscore APIs could have breaking changes at any time.) Am I 
remembering that wrong? Or was internal API stability only guaranteed for 
patch-level releases?

*Should go through deprecation process* (in my opinion, sorted roughly by 
real-world usage)

   - *BadHeaderError* is mentioned in the docs 
   

 as 
   potentially raised during sending
  - Exposed in django.core.mail.__all__
  - Real-world use: ~2300+ cases in GitHub code search 
  
,
 
  seems to be primarily error handling 
  

  - Proposal: deprecate on import. (Modern Python email API raises 
  ValueError for newlines in headers. In our deprecation, we'll need 
  `BadHeaderError = ValueError`—rather than subclassing ValueError—to 
  preserve behavior of existing error handling code.)
  - Suggested replacement for docs: change `except BadHeaderError` to 
  `except ValueError`
  
  - *sanitize_address()* is not documented, but is widely used
  - Not exposed in django.core.mail (have to import from 
  django.core.mail.message)
  - Real-world use: ~430 cases in GitHub code search 
  
,
 
  often in custom EmailBackend implementations 
  

 or 
  for special case handling 
  

 that 
  needs knowledge of how django.core.mail handles addresses.
  - Proposal: deprecate. (Have to keep implementation around anyway as 
  part of other deprecated code.)
  - Suggested replacement for docs: none (it's an internal, 
  undocumented API). Or we could say it depends on the use case: just skip 
it 
  if no longer relevant, use something like Python's modern email Address 
  object if you need formatting cleanup, or copy sanitize_address() into 
your 
  code and adapt for your needs. (Incidentally, it's already pretty 
  common to copy 
  

 
  and adapt sanitize_address().)
  
  - *SafeMIMEText* and *SafeMIMEMultipart* are mentioned in the docs 
   

 
   as the return type of EmailMessage.message(), but not further documented
  - Exposed in django.core.mail.__all__
  - Real-world use: ~240 cases in GitHub code search 
  
,
 
  both for type checking 
  

 
  and for constructing text or message attachments 
  

  - Proposal: deprecate (on import, to catch type checking uses)
  - Suggested replacement for docs: django.mail.EmailMessage.message() 
  will now return an email.message.EmailMessage; to construct text or 
message 
  attachments, switch to modern email.message.EmailMessage
  - [Tip for django-stubs: SafeMIMEText, SafeMIMEMultipart, and 
  email.message.EmailMessage are all subclasses of email.message.Message]
  
  - *forbid_multi_line_headers()* is not documented, but seems to be an 
   intentional part of the public API for implementin

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Ronny V.]

Adding my support for the suggested approach. Start small and central and 
step-by-step. 👍

> (I'm going to add a link in django-anymail's "you probably don't need 
proprietary ESP templates 
" docs.) 

Thanks! Much appreciated!

And thx on your thoughts about the class-based emails. I do agree that we 
should start with the proposed changes. Then we'll worry about the 
high-level stuff.

Best regards from Cologne
Ronny

PS: Sorry for misspelling Jacob Rief in my earlier posting!


Florian Apolloner schrieb am Freitag, 28. Juni 2024 um 22:07:35 UTC+2:

> Hi Mike,
>
> overall the plan sounds good. I especially like the approach to "fix" the 
> tests first. This can happen in an extra merge request and reviewed 
> independently so we can be sure that we are still testing what we want to 
> test before moving to the tricky parts.
>
> On Thursday, June 27, 2024 at 3:28:06 AM UTC+2 Mike Edmunds wrote:
>
>
>1. 
>
>Deprecate unused internal APIs from django.core.mail.message
>
>Django will no longer need these (Python's modern email API covers 
>their functionality), but they may be in use by third-party libraries:
>- utf8_charset
>   - utf8_charset_qp
>   - RFC5322_EMAIL_LINE_LENGTH_LIMIT
>   - BadHeaderError^ (more details below)
>   - ADDRESS_HEADERS
>   - forbid_multi_line_headers()^
>   - sanitize_address() (more details below)
>   - MIMEMixin
>   - SafeMIMEMessage
>   - SafeMIMEText^
>   - SafeMIMEMultipart^
>
>(Items marked ^ are exposed in django.core.mail via __all__. I haven't 
>looked into the reason for that.)
>
>
> Are all of those documented? If not we can simply remove them (especially 
> if the deprecation implementation turns out to be a PITA).  
>
> Cheers,
> Florian
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b8d51b07-321b-4540-8188-46cd0b9a711fn%40googlegroups.com.

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Florian Apolloner]

Hi Mike,

overall the plan sounds good. I especially like the approach to "fix" the 
tests first. This can happen in an extra merge request and reviewed 
independently so we can be sure that we are still testing what we want to 
test before moving to the tricky parts.

On Thursday, June 27, 2024 at 3:28:06 AM UTC+2 Mike Edmunds wrote:


   1. 
   
   Deprecate unused internal APIs from django.core.mail.message
   
   Django will no longer need these (Python's modern email API covers their 
   functionality), but they may be in use by third-party libraries:
   - utf8_charset
  - utf8_charset_qp
  - RFC5322_EMAIL_LINE_LENGTH_LIMIT
  - BadHeaderError^ (more details below)
  - ADDRESS_HEADERS
  - forbid_multi_line_headers()^
  - sanitize_address() (more details below)
  - MIMEMixin
  - SafeMIMEMessage
  - SafeMIMEText^
  - SafeMIMEMultipart^
   
   (Items marked ^ are exposed in django.core.mail via __all__. I haven't 
   looked into the reason for that.)
   

Are all of those documented? If not we can simply remove them (especially 
if the deprecation implementation turns out to be a PITA).  

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/df3db456-b847-416b-b496-b6e70e824fc7n%40googlegroups.com.

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

> focus it on purely the strictly necessary parts to update to the new API 
… before dealing with potentially contentious things 
like EmailMultiAlternatives, MIMEBase, etc.

Appreciate the advice. I'll try to keep EmailMultiAlternatives and see 
whether that makes the new code hard to follow.

Sadly, MIMEBase attachments are a documented part of Django's EmailMessage 
API (and have been necessary for certain kinds of attachments), so they'll 
have to be dealt with one way or another.

- Mike

On Thursday, June 27, 2024 at 2:14:12 AM UTC-7 Tom Carrick wrote:

> I'm in favour of this change, and nice that you're thinking about the 
> future, but if you're going to write a ticket for this I would focus it on 
> purely the strictly necessary parts to update to the new API, and when that 
> is done, make more proposals to simplify the API as you suggest.
>
> I say this just with the goal of getting something that is easy to agree 
> on merged, before dealing with potentially contentious things 
> like EmailMultiAlternatives, MIMEBase, etc.
>
> Ignore me if this is irrelevant and those things are indeed necessary, my 
> knowledge on the topic is a bit lacking.
>
> Tom
>
> On Thu, 27 Jun 2024 at 03:28, Mike Edmunds  wrote:
>
>> Since the early feedback seems positive (though we're still waiting for 
>> more votes), here's some additional detail on the changes I think would be 
>> involved to update django.core.mail to use Python's modern email API.
>>
>> (See my earlier message 
>>  
>> for 
>> background on Python's legacy vs. modern email APIs and why updating is 
>> useful.)
>>
>> Note: Django and Python both have classes named EmailMessage. I'm using 
>> "Django EmailMessage" to refer to django.core.mail.message.EmailMessage, 
>> and "Python EmailMessage" (or sometimes just "modern API") to refer to 
>> Python's modern email.message.EmailMessage.
>> Necessary work 
>> 
>>
>>1. 
>>
>>Update tests.mail.tests to use modern email APIs
>>
>>Where a test relies on legacy implementation details, try to rewrite 
>>it to be implementation agnostic if possible; otherwise try to retain the 
>>spirit of the test using modern APIs.
>>
>>Retain all security related tests, with updates as appropriate. (Even 
>>where we know Python's modern API handles the security for us, it doesn't 
>>hurt to double check.)
>>
>>Probably need to add some cases for existing behavior not currently 
>>covered by tests, particularly around attachments and alternatives.
>>
>>Remove a test case *only* if it's truly no longer relevant and can't 
>>be usefully updated. (And perhaps leave behind a brief comment explaining 
>>why.)
>>
>>(Legacy email APIs used in tests.mail.tests: email.charset, 
>>email.header.Header, MIMEText, parseaddr. Also message_from_… currently 
>>defaults to legacy policy.compat32.)
>>2. 
>>
>>Update django.core.mail.message.EmailMessage to use modern email APIs
>>
>>Change Django's EmailMessage.message() to construct a modern 
>>email.message.EmailMessage rather than a SafeMIME object (which is based 
>> on 
>>legacy email.message.Message with policy=compat32). Add a 
>>message(policy=default) param forwarded to Python's EmailMessage 
>>constructor.
>>
>>Hoist alternative part handling from Django's EmailMultiAlternatives 
>>into Django's base EmailMessage to simplify the code. (More on this 
>> below.)
>>
>>In _create_alternatives(), use modern add_alternative() 
>>
>> 
>>  and 
>>friends to replace legacy SafeMIME objects.
>>
>>In _create_attachments(), use modern add_attachment() 
>>
>> .
>>  
>>Handle (legacy Python) MIMEBase objects as deprecated, and convert to 
>>modern equivalent. This is a relatively complex topic; I'll post a 
>> separate 
>>message about it later. (I also have some questions about how best to 
>>handle content-disposition "inline" and whether we want to somehow 
>> support 
>>multipart/related.)
>>
>>Remove the private Django EmailMessage methods 
>>_create_mime_attachment() and _create_attachment() (without deprecation).
>>
>>(Legacy APIs used in django.core.mail.message: email.message.Message, 
>>email.charset, email.encoders, email.header, email.mime, email.utils)
>>3. 
>>
>>Deprecate unused internal APIs from django.core.mail.message
>>
>>Django will no longer need these (Python's modern email API covers 
>>their functionality), but they may be in use by third-party libraries:
>>- utf8_charset
>>   - utf8_chars

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Paolo Melchiorre]

I agree with the approach suggested by Tom.

And thanks for proposing this enhancement.

Ciao,
Paolo

On Thu, Jun 27, 2024, 11:14 Tom Carrick  wrote:

> I'm in favour of this change, and nice that you're thinking about the
> future, but if you're going to write a ticket for this I would focus it on
> purely the strictly necessary parts to update to the new API, and when that
> is done, make more proposals to simplify the API as you suggest.
>
> I say this just with the goal of getting something that is easy to agree
> on merged, before dealing with potentially contentious things
> like EmailMultiAlternatives, MIMEBase, etc.
>
> Ignore me if this is irrelevant and those things are indeed necessary, my
> knowledge on the topic is a bit lacking.
>
> Tom
>
> On Thu, 27 Jun 2024 at 03:28, Mike Edmunds  wrote:
>
>> Since the early feedback seems positive (though we're still waiting for
>> more votes), here's some additional detail on the changes I think would be
>> involved to update django.core.mail to use Python's modern email API.
>>
>> (See my earlier message
>>  
>> for
>> background on Python's legacy vs. modern email APIs and why updating is
>> useful.)
>>
>> Note: Django and Python both have classes named EmailMessage. I'm using
>> "Django EmailMessage" to refer to django.core.mail.message.EmailMessage,
>> and "Python EmailMessage" (or sometimes just "modern API") to refer to
>> Python's modern email.message.EmailMessage.
>> Necessary work
>> 
>>
>>1.
>>
>>Update tests.mail.tests to use modern email APIs
>>
>>Where a test relies on legacy implementation details, try to rewrite
>>it to be implementation agnostic if possible; otherwise try to retain the
>>spirit of the test using modern APIs.
>>
>>Retain all security related tests, with updates as appropriate. (Even
>>where we know Python's modern API handles the security for us, it doesn't
>>hurt to double check.)
>>
>>Probably need to add some cases for existing behavior not currently
>>covered by tests, particularly around attachments and alternatives.
>>
>>Remove a test case *only* if it's truly no longer relevant and can't
>>be usefully updated. (And perhaps leave behind a brief comment explaining
>>why.)
>>
>>(Legacy email APIs used in tests.mail.tests: email.charset,
>>email.header.Header, MIMEText, parseaddr. Also message_from_… currently
>>defaults to legacy policy.compat32.)
>>2.
>>
>>Update django.core.mail.message.EmailMessage to use modern email APIs
>>
>>Change Django's EmailMessage.message() to construct a modern
>>email.message.EmailMessage rather than a SafeMIME object (which is based 
>> on
>>legacy email.message.Message with policy=compat32). Add a
>>message(policy=default) param forwarded to Python's EmailMessage
>>constructor.
>>
>>Hoist alternative part handling from Django's EmailMultiAlternatives
>>into Django's base EmailMessage to simplify the code. (More on this 
>> below.)
>>
>>In _create_alternatives(), use modern add_alternative()
>>
>> 
>>  and
>>friends to replace legacy SafeMIME objects.
>>
>>In _create_attachments(), use modern add_attachment()
>>
>> .
>>Handle (legacy Python) MIMEBase objects as deprecated, and convert to
>>modern equivalent. This is a relatively complex topic; I'll post a 
>> separate
>>message about it later. (I also have some questions about how best to
>>handle content-disposition "inline" and whether we want to somehow support
>>multipart/related.)
>>
>>Remove the private Django EmailMessage methods
>>_create_mime_attachment() and _create_attachment() (without deprecation).
>>
>>(Legacy APIs used in django.core.mail.message: email.message.Message,
>>email.charset, email.encoders, email.header, email.mime, email.utils)
>>3.
>>
>>Deprecate unused internal APIs from django.core.mail.message
>>
>>Django will no longer need these (Python's modern email API covers
>>their functionality), but they may be in use by third-party libraries:
>>- utf8_charset
>>   - utf8_charset_qp
>>   - RFC5322_EMAIL_LINE_LENGTH_LIMIT
>>   - BadHeaderError^ (more details below)
>>   - ADDRESS_HEADERS
>>   - forbid_multi_line_headers()^
>>   - sanitize_address() (more details below)
>>   - MIMEMixin
>>   - SafeMIMEMessage
>>   - SafeMIMEText^
>>   - SafeMIMEMultipart^
>>
>>(Items marked ^ are exposed in django.core.mail via __all__. I
>>haven't looked into the reason for that.)
>>4.
>>
>>Update django.core.mail.__init__ to avoid EmailMultiAlternatives, and

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Tom Carrick]

I'm in favour of this change, and nice that you're thinking about the
future, but if you're going to write a ticket for this I would focus it on
purely the strictly necessary parts to update to the new API, and when that
is done, make more proposals to simplify the API as you suggest.

I say this just with the goal of getting something that is easy to agree on
merged, before dealing with potentially contentious things
like EmailMultiAlternatives, MIMEBase, etc.

Ignore me if this is irrelevant and those things are indeed necessary, my
knowledge on the topic is a bit lacking.

Tom

On Thu, 27 Jun 2024 at 03:28, Mike Edmunds  wrote:

> Since the early feedback seems positive (though we're still waiting for
> more votes), here's some additional detail on the changes I think would be
> involved to update django.core.mail to use Python's modern email API.
>
> (See my earlier message
>  
> for
> background on Python's legacy vs. modern email APIs and why updating is
> useful.)
>
> Note: Django and Python both have classes named EmailMessage. I'm using
> "Django EmailMessage" to refer to django.core.mail.message.EmailMessage,
> and "Python EmailMessage" (or sometimes just "modern API") to refer to
> Python's modern email.message.EmailMessage.
> Necessary work
> 
>
>1.
>
>Update tests.mail.tests to use modern email APIs
>
>Where a test relies on legacy implementation details, try to rewrite
>it to be implementation agnostic if possible; otherwise try to retain the
>spirit of the test using modern APIs.
>
>Retain all security related tests, with updates as appropriate. (Even
>where we know Python's modern API handles the security for us, it doesn't
>hurt to double check.)
>
>Probably need to add some cases for existing behavior not currently
>covered by tests, particularly around attachments and alternatives.
>
>Remove a test case *only* if it's truly no longer relevant and can't
>be usefully updated. (And perhaps leave behind a brief comment explaining
>why.)
>
>(Legacy email APIs used in tests.mail.tests: email.charset,
>email.header.Header, MIMEText, parseaddr. Also message_from_… currently
>defaults to legacy policy.compat32.)
>2.
>
>Update django.core.mail.message.EmailMessage to use modern email APIs
>
>Change Django's EmailMessage.message() to construct a modern
>email.message.EmailMessage rather than a SafeMIME object (which is based on
>legacy email.message.Message with policy=compat32). Add a
>message(policy=default) param forwarded to Python's EmailMessage
>constructor.
>
>Hoist alternative part handling from Django's EmailMultiAlternatives
>into Django's base EmailMessage to simplify the code. (More on this below.)
>
>In _create_alternatives(), use modern add_alternative()
>
> 
>  and
>friends to replace legacy SafeMIME objects.
>
>In _create_attachments(), use modern add_attachment()
>
> .
>Handle (legacy Python) MIMEBase objects as deprecated, and convert to
>modern equivalent. This is a relatively complex topic; I'll post a separate
>message about it later. (I also have some questions about how best to
>handle content-disposition "inline" and whether we want to somehow support
>multipart/related.)
>
>Remove the private Django EmailMessage methods
>_create_mime_attachment() and _create_attachment() (without deprecation).
>
>(Legacy APIs used in django.core.mail.message: email.message.Message,
>email.charset, email.encoders, email.header, email.mime, email.utils)
>3.
>
>Deprecate unused internal APIs from django.core.mail.message
>
>Django will no longer need these (Python's modern email API covers
>their functionality), but they may be in use by third-party libraries:
>- utf8_charset
>   - utf8_charset_qp
>   - RFC5322_EMAIL_LINE_LENGTH_LIMIT
>   - BadHeaderError^ (more details below)
>   - ADDRESS_HEADERS
>   - forbid_multi_line_headers()^
>   - sanitize_address() (more details below)
>   - MIMEMixin
>   - SafeMIMEMessage
>   - SafeMIMEText^
>   - SafeMIMEMultipart^
>
>(Items marked ^ are exposed in django.core.mail via __all__. I haven't
>looked into the reason for that.)
>4.
>
>Update django.core.mail.__init__ to avoid EmailMultiAlternatives, and
>django.core.mail.backend.smtp to replace sanitize_address. (More details
>below.)
>5.
>
>Update docs
>- deprecation of legacy MIMEBase in attachments list, what to do
>   instead
>   - eliminate EmailMessage/EmailMultiAlternatives distinction
>   - dep

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

Since the early feedback seems positive (though we're still waiting for 
more votes), here's some additional detail on the changes I think would be 
involved to update django.core.mail to use Python's modern email API.

(See my earlier message 
 
for 
background on Python's legacy vs. modern email APIs and why updating is 
useful.)

Note: Django and Python both have classes named EmailMessage. I'm using 
"Django EmailMessage" to refer to django.core.mail.message.EmailMessage, 
and "Python EmailMessage" (or sometimes just "modern API") to refer to 
Python's modern email.message.EmailMessage.
Necessary work 


   1. 
   
   Update tests.mail.tests to use modern email APIs
   
   Where a test relies on legacy implementation details, try to rewrite it 
   to be implementation agnostic if possible; otherwise try to retain the 
   spirit of the test using modern APIs.
   
   Retain all security related tests, with updates as appropriate. (Even 
   where we know Python's modern API handles the security for us, it doesn't 
   hurt to double check.)
   
   Probably need to add some cases for existing behavior not currently 
   covered by tests, particularly around attachments and alternatives.
   
   Remove a test case *only* if it's truly no longer relevant and can't be 
   usefully updated. (And perhaps leave behind a brief comment explaining why.)
   
   (Legacy email APIs used in tests.mail.tests: email.charset, 
   email.header.Header, MIMEText, parseaddr. Also message_from_… currently 
   defaults to legacy policy.compat32.)
   2. 
   
   Update django.core.mail.message.EmailMessage to use modern email APIs
   
   Change Django's EmailMessage.message() to construct a modern 
   email.message.EmailMessage rather than a SafeMIME object (which is based on 
   legacy email.message.Message with policy=compat32). Add a 
   message(policy=default) param forwarded to Python's EmailMessage 
   constructor.
   
   Hoist alternative part handling from Django's EmailMultiAlternatives 
   into Django's base EmailMessage to simplify the code. (More on this below.)
   
   In _create_alternatives(), use modern add_alternative() 
   

 and 
   friends to replace legacy SafeMIME objects.
   
   In _create_attachments(), use modern add_attachment() 
   
.
 
   Handle (legacy Python) MIMEBase objects as deprecated, and convert to 
   modern equivalent. This is a relatively complex topic; I'll post a separate 
   message about it later. (I also have some questions about how best to 
   handle content-disposition "inline" and whether we want to somehow support 
   multipart/related.)
   
   Remove the private Django EmailMessage methods _create_mime_attachment() 
   and _create_attachment() (without deprecation).
   
   (Legacy APIs used in django.core.mail.message: email.message.Message, 
   email.charset, email.encoders, email.header, email.mime, email.utils)
   3. 
   
   Deprecate unused internal APIs from django.core.mail.message
   
   Django will no longer need these (Python's modern email API covers their 
   functionality), but they may be in use by third-party libraries:
   - utf8_charset
  - utf8_charset_qp
  - RFC5322_EMAIL_LINE_LENGTH_LIMIT
  - BadHeaderError^ (more details below)
  - ADDRESS_HEADERS
  - forbid_multi_line_headers()^
  - sanitize_address() (more details below)
  - MIMEMixin
  - SafeMIMEMessage
  - SafeMIMEText^
  - SafeMIMEMultipart^
   
   (Items marked ^ are exposed in django.core.mail via __all__. I haven't 
   looked into the reason for that.)
   4. 
   
   Update django.core.mail.__init__ to avoid EmailMultiAlternatives, and 
   django.core.mail.backend.smtp to replace sanitize_address. (More details 
   below.)
   5. 
   
   Update docs
   - deprecation of legacy MIMEBase in attachments list, what to do instead
  - eliminate EmailMessage/EmailMultiAlternatives distinction
  - deprecation of internal legacy items from #3
   
Tricky bits and things requiring longer discussion 


   - 
   
   *Legacy MIMEBase attachments:* I'll post a separate message about this 
   sometime later. (Maybe next week. It's a long topic, plus I'm still 
   experimenting.)
   - 
   
   *EmailMultiAlternatives:* I'd like to get rid of the distinction between 
   Django's EmailMessage and EmailMultiAlternatives, and just move the logic 
   for alternative parts into base EmailMessage. (And then simplify the docs.)
   
   EmailMultiAlternatives is additional complexity for users, and it isn't 
   really helpful wi

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

> After my comment in the steering council vote and in-person conversation 
with Jake, I believe the SMTP backend will not be implemented for DEP 14

Then I can improve my earlier response: I *am confident* this proposal will 
have *no impact* on background workers. :-)

- Mike

On Tuesday, June 25, 2024 at 4:38:11 AM UTC-7 Adam Johnson wrote:

> After my comment in the steering council vote and in-person conversation 
> with Jake, I believe the SMTP backend will not be implemented for DEP 14 : 
> https://forum.djangoproject.com/t/steering-council-vote-on-background-tasks-dep-14/31131/20
>
> On Tue, 25 Jun 2024, at 10:27, Arthur Pemberton wrote:
>
> >  The background workers proposal will implement a new background SMTP 
> EmailBackend (in django.core.mail.backends).
>
> I had missed that fact. Thanks for the explanation.
>
> I for one think that this is a good proposal -- this would modern 
> transactional email sending.
>
> - Arthur
>
>
> On Mon, Jun 24, 2024 at 3:14 PM Mike Edmunds  wrote:
>
> > Would this be designed to be compatible with "Proposal 14: Background 
> Workers"?
>
> I wouldn't expect this to impact background workers one way or the other. 
> The same goes for the async email proposal(s) floating around.
>
> Virtually all of this work would occur in django.core.mail.message, where 
> Python's `email` APIs are used. A goal is to avoid changes that would break 
> existing email backends (Django or third-party).
>
> The background workers proposal will implement a new background SMTP 
> EmailBackend (in django.core.mail.backends). The existing SMTP EmailBackend 
> doesn't directly use Python's `email` APIs, and there should be no reason 
> for background SMTP to be any different. (It might be helpful to know that 
> Python's `email` library isn't involved in *sending* email; that's 
> handled by Python's `smtplib`, which Django uses only in the SMTP 
> EmailBackend.)
>
> The existing SMTP EmailBackend *does* use one function I expect will 
> become deprecated: django.core.mail.message.sanitize_address(). I haven't 
> yet investigated whether that use is still necessary, or whether it's there 
> to get around past bugs/limitations in Python's smtplib. If any of it is 
> still needed for SMTP, I'd probably want to move that code into the SMTP 
> EmailBackend(s).
>
> - Mike
> On Sunday, June 23, 2024 at 10:22:03 PM UTC-7 Arthur Pemberton wrote:
>
> Would this be designed to be compatible with "Proposal 14: Background 
> Workers"?
>
>
>
>
> On Sun, Jun 23, 2024 at 10:46 PM Mike Edmunds  wrote:
>
> I want to propose updating django.core.mail to replace use of Python's 
> legacy email.message.Message (and other legacy email APIs) with 
> email.message.EmailMessage (and other modern APIs).
>
> If there's interest, I can put together a more detailed proposal and/or 
> ticket, but was hoping to get some initial feedback first. (I searched for 
> relevant discussions in django-developers and the issue tracker, and didn't 
> find any. Apologies if this has come up before.)
>
> [Note: I maintain django-anymail 
> , which implements Django 
> integration with several transactional email service providers.]
>
>
> *Background*
>
> Since Python ~3.6, Python's email package has included two largely 
> separate implementations:
>
>- a "modern (unicode friendly) API" based on 
>email.message.EmailMessage and email.policy.default
>- a "legacy API" providing compatibility with older Python versions, 
>based on email.message.Message and email.policy.compat32
>
> (See https://docs.python.org/3/library/email.html, especially toward the 
> end.)
>
> django.core.mail currently uses the legacy API.
>
>
> *Why switch?*
>
> There are no plans to deprecate Python's legacy email API, and it's 
> working, so why change Django?
>
>- 
>
>*Fewer bugs:* The modern API fixes a *lot* of bugs in the legacy API. 
>My understanding is legacy bugs will generally not be fixed. (And in fact, 
>there are some cases where the legacy API deliberately replicates earlier 
>buggy behavior.)
>
>Django #35497  is an 
>example of a legacy bug (with a problematic proposed workaround) which is 
>just fixed in the modern API.
>- 
>
>*Simpler, safer code:* A substantial portion 
>
> 
>  of 
>django.core.mail's internals implements workarounds and security fixes for 
>problems in the legacy API. This would be greatly simplified—maybe 
>eliminated completely—using the modern API.
>
>Examples: the modern API prevents CR/NL injections in message headers. 
>It serializes and folds address headers properly—even ones with unicode 
>names. It enforces single-instance headers where appropriate.
>- 
>
>*Easier to work with:* The modern API adds some nice conveniences for

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

Hi Ronny,

django-pony-express looks really interesting, thanks. (I'm going to add a 
link in django-anymail's "you probably don't need proprietary ESP templates 
" docs.)

I think django-pony-express is very complementary to this proposal, both as 
a third-party library and if some version of class-based emails finds its 
way into Django core. 

Here's my mental model of Django's email sending functionality:

   - A django.core.mail.message.EmailMessage (or EmailMultiAlternatives*) 
   is a list of ingredients for building an email message—it's mostly just a 
   "bucket of properties" that specifies what should end up in the message.**
   - An EmailBackend is responsible for transmitting that EmailMessage to a 
   server that will actually send an email—an SMTP server, an email service 
   provider's HTTP API, etc. Each backend implements its own recipe to bake 
   the Django EmailMessage ingredients into the form expected by its server—an 
   RFC 2822+ message for SMTP, a JSON API payload for an ESP, etc.
   - Because a Django EmailMessage can be "a pain" (you're not wrong!), 
   there are convenience APIs that simplify constructing and sending it. Some 
   come with Django: send_mail(), send_mass_mail(), mail_admins(). Some come 
   from third party libraries: django-pony-express, django-templated-mail, 
   etc. *Many* get built (and repeatedly re-built) by developers in 
   individual Django projects.

To stretch your class-based view analogy, Django's EmailMessage is roughly 
akin to Django's HttpResponse. You can write a view function that builds up 
an HttpResponse from scratch. For a lot of common cases, though, it's much 
easier to work one of Django's helper functions or class-based View 
subclasses. But however you go about it, every view eventually has to 
return an HttpResponse that Django can hand off to its "http backend" 
(wsgi/asgi).

Similarly, there are several ways to build a Django EmailMessage. At the 
end of the day, though, all of these need to return a 
django.core.mail.EmailMessage that Django can hand off to an email backend 
for sending.

- Mike

_

* I suspect the EmailMultiAlternatives/EmailMessage distinction is no 
longer helpful. I'm fairly certain it's confusing to users, and I *know* it 
adds complexity for third-party email backends. Given that modern Python 
email convenience 

 
methods 

 handle 
all the multipart restructuring, I'll propose collapsing 
EmailMultiAlternatives/EmailMessage into a single class as part of this 
work.

** Along with the "bucket of properties," Django's EmailMessage also 
implements serialization to RFC 2822+ format in as_message(). This is used 
by many—but certainly not all—email backends. And (just to come full circle 
here in the footnotes), the bulk of work in this proposal is actually about 
updating as_message() and the private helpers it calls.

On Tuesday, June 25, 2024 at 7:23:42 AM UTC-7 Ronny V. wrote:

> Hi all!
>
> Jakob Rief pointed this discussion out to me. I've been going around 
> lately to make some advertisement for my idea of class-based emails.
>
> I've implemented a package called "django-pony-express" which in a 
> nutshell provides to things:
>
> * A class-based way of creating new emails (very similar to class-based 
> views)
> * A test suite for easy unit-testing of emails (we are currently working 
> on bringing this to Django core)
>
> Here's a documentation link: 
> https://django-pony-express.readthedocs.io/en/latest/features/introduction.html#create-a-single-email
>  
>
> The idea is to create emails like you create views. All important stuff is 
> encapsulated but you can overwrite everything you need. There are a bunch 
> of examples in the docs.
>
> Apart from not having to deal with low-level email API stuff (which is a 
> pain IMHO), it provides lots of neat improvements I had to (re-)invent in 
> every project I was working on.
>
> What do you people think about this? I got quite good feedback in Vigo and 
> in my opinion, the solution is very Django-esque.
>
> I'd be happy to go into more detail if required but wanted to keep this 
> first commend brief.
>
> Best from Cologne
> Ronny
> Mike Edmunds schrieb am Montag, 24. Juni 2024 um 21:14:47 UTC+2:
>
>> > Would this be designed to be compatible with "Proposal 14: Background 
>> Workers"?
>>
>> I wouldn't expect this to impact background workers one way or the other. 
>> The same goes for the async email proposal(s) floating around.
>>
>> Virtually all of this work would occur in django.core.mail.message, where 
>> Python's `email` APIs are used. A goal is to avoid changes that would break 
>> existing email backends (Django or third-party).
>>
>> The background workers proposal will implement a new back

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Ronny V.]

Hi all!

Jakob Rief pointed this discussion out to me. I've been going around lately 
to make some advertisement for my idea of class-based emails.

I've implemented a package called "django-pony-express" which in a nutshell 
provides to things:

* A class-based way of creating new emails (very similar to class-based 
views)
* A test suite for easy unit-testing of emails (we are currently working on 
bringing this to Django core)

Here's a documentation link: 
https://django-pony-express.readthedocs.io/en/latest/features/introduction.html#create-a-single-email
 


The idea is to create emails like you create views. All important stuff is 
encapsulated but you can overwrite everything you need. There are a bunch 
of examples in the docs.

Apart from not having to deal with low-level email API stuff (which is a 
pain IMHO), it provides lots of neat improvements I had to (re-)invent in 
every project I was working on.

What do you people think about this? I got quite good feedback in Vigo and 
in my opinion, the solution is very Django-esque.

I'd be happy to go into more detail if required but wanted to keep this 
first commend brief.

Best from Cologne
Ronny
Mike Edmunds schrieb am Montag, 24. Juni 2024 um 21:14:47 UTC+2:

> > Would this be designed to be compatible with "Proposal 14: Background 
> Workers"?
>
> I wouldn't expect this to impact background workers one way or the other. 
> The same goes for the async email proposal(s) floating around.
>
> Virtually all of this work would occur in django.core.mail.message, where 
> Python's `email` APIs are used. A goal is to avoid changes that would break 
> existing email backends (Django or third-party).
>
> The background workers proposal will implement a new background SMTP 
> EmailBackend (in django.core.mail.backends). The existing SMTP EmailBackend 
> doesn't directly use Python's `email` APIs, and there should be no reason 
> for background SMTP to be any different. (It might be helpful to know that 
> Python's `email` library isn't involved in *sending* email; that's 
> handled by Python's `smtplib`, which Django uses only in the SMTP 
> EmailBackend.)
>
> The existing SMTP EmailBackend *does* use one function I expect will 
> become deprecated: django.core.mail.message.sanitize_address(). I haven't 
> yet investigated whether that use is still necessary, or whether it's there 
> to get around past bugs/limitations in Python's smtplib. If any of it is 
> still needed for SMTP, I'd probably want to move that code into the SMTP 
> EmailBackend(s).
>
> - Mike
> On Sunday, June 23, 2024 at 10:22:03 PM UTC-7 Arthur Pemberton wrote:
>
>> Would this be designed to be compatible with "Proposal 14: Background 
>> Workers"?
>>
>>
>>
>> On Sun, Jun 23, 2024 at 10:46 PM Mike Edmunds  wrote:
>>
>>> I want to propose updating django.core.mail to replace use of Python's 
>>> legacy email.message.Message (and other legacy email APIs) with 
>>> email.message.EmailMessage (and other modern APIs).
>>>
>>> If there's interest, I can put together a more detailed proposal and/or 
>>> ticket, but was hoping to get some initial feedback first. (I searched for 
>>> relevant discussions in django-developers and the issue tracker, and didn't 
>>> find any. Apologies if this has come up before.)
>>>
>>> [Note: I maintain django-anymail 
>>> , which implements Django 
>>> integration with several transactional email service providers.]
>>>
>>>
>>> *Background*
>>>
>>> Since Python ~3.6, Python's email package has included two largely 
>>> separate implementations:
>>>
>>>- a "modern (unicode friendly) API" based on 
>>>email.message.EmailMessage and email.policy.default
>>>- a "legacy API" providing compatibility with older Python versions, 
>>>based on email.message.Message and email.policy.compat32
>>>
>>> (See https://docs.python.org/3/library/email.html, especially toward 
>>> the end.)
>>>
>>> django.core.mail currently uses the legacy API.
>>>
>>>
>>> *Why switch?*
>>>
>>> There are no plans to deprecate Python's legacy email API, and it's 
>>> working, so why change Django?
>>>
>>>- 
>>>
>>>*Fewer bugs:* The modern API fixes a *lot* of bugs in the legacy 
>>>API. My understanding is legacy bugs will generally not be fixed. (And 
>>> in 
>>>fact, there are some cases where the legacy API deliberately replicates 
>>>earlier buggy behavior.)
>>>
>>>Django #35497  is an 
>>>example of a legacy bug (with a problematic proposed workaround) which 
>>> is 
>>>just fixed in the modern API.
>>>- 
>>>
>>>*Simpler, safer code:* A substantial portion 
>>>
>>> 
>>>  of 
>>>django.core.mail's internals implements workarounds and security fixes 
>>> for 
>>>problems in the legacy API. This would be greatly simplified—maybe 
>>>elim

Re: Proposal to upgrade django.core.mail to Python's modern email API

['Adam Johnson' via Django developers (Contributions to Django itself)]

After my comment in the steering council vote and in-person conversation with 
Jake, I believe the SMTP backend will not be implemented for DEP 14 : 
https://forum.djangoproject.com/t/steering-council-vote-on-background-tasks-dep-14/31131/20

On Tue, 25 Jun 2024, at 10:27, Arthur Pemberton wrote:
> >  The background workers proposal will implement a new background SMTP 
> > EmailBackend (in django.core.mail.backends).
> 
> I had missed that fact. Thanks for the explanation.
> 
> I for one think that this is a good proposal -- this would modern 
> transactional email sending.
> 
> - Arthur
> 
> 
> On Mon, Jun 24, 2024 at 3:14 PM Mike Edmunds  wrote:
>> > Would this be designed to be compatible with "Proposal 14: Background 
>> > Workers"?
>> 
>> I wouldn't expect this to impact background workers one way or the other. 
>> The same goes for the async email proposal(s) floating around.
>> 
>> Virtually all of this work would occur in django.core.mail.message, where 
>> Python's `email` APIs are used. A goal is to avoid changes that would break 
>> existing email backends (Django or third-party).
>> 
>> The background workers proposal will implement a new background SMTP 
>> EmailBackend (in django.core.mail.backends). The existing SMTP EmailBackend 
>> doesn't directly use Python's `email` APIs, and there should be no reason 
>> for background SMTP to be any different. (It might be helpful to know that 
>> Python's `email` library isn't involved in *sending* email; that's handled 
>> by Python's `smtplib`, which Django uses only in the SMTP EmailBackend.)
>> 
>> The existing SMTP EmailBackend *does* use one function I expect will become 
>> deprecated: django.core.mail.message.sanitize_address(). I haven't yet 
>> investigated whether that use is still necessary, or whether it's there to 
>> get around past bugs/limitations in Python's smtplib. If any of it is still 
>> needed for SMTP, I'd probably want to move that code into the SMTP 
>> EmailBackend(s).
>> 
>> - Mike
>> On Sunday, June 23, 2024 at 10:22:03 PM UTC-7 Arthur Pemberton wrote:
>>> Would this be designed to be compatible with "Proposal 14: Background 
>>> Workers"?
>>> 
>>> 
>>> 
>>> 
>>> On Sun, Jun 23, 2024 at 10:46 PM Mike Edmunds  wrote:
 I want to propose updating django.core.mail to replace use of Python's 
 legacy email.message.Message (and other legacy email APIs) with 
 email.message.EmailMessage (and other modern APIs).
 
 If there's interest, I can put together a more detailed proposal and/or 
 ticket, but was hoping to get some initial feedback first. (I searched for 
 relevant discussions in django-developers and the issue tracker, and 
 didn't find any. Apologies if this has come up before.)
 
 [Note: I maintain django-anymail 
 , which implements Django 
 integration with several transactional email service providers.]
 
 
 
 *Background*
 
 Since Python ~3.6, Python's email package has included two largely 
 separate implementations:
 
  • a "modern (unicode friendly) API" based on email.message.EmailMessage 
 and email.policy.default
  • a "legacy API" providing compatibility with older Python versions, 
 based on email.message.Message and email.policy.compat32
 (See https://docs.python.org/3/library/email.html, especially toward the 
 end.)
 
 django.core.mail currently uses the legacy API.
 
 
 
 *Why switch?*
 
 There are no plans to deprecate Python's legacy email API, and it's 
 working, so why change Django?
 
  • *Fewer bugs:* The modern API fixes a *lot* of bugs in the legacy API. 
 My understanding is legacy bugs will generally not be fixed. (And in fact, 
 there are some cases where the legacy API deliberately replicates earlier 
 buggy behavior.)
 
 Django #35497  is an example 
 of a legacy bug (with a problematic proposed workaround) which is just 
 fixed in the modern API.
 
  • *Simpler, safer code:* A substantial portion 
 
  of django.core.mail's internals implements workarounds and security fixes 
 for problems in the legacy API. This would be greatly simplified—maybe 
 eliminated completely—using the modern API.
 
 Examples: the modern API prevents CR/NL injections in message headers. It 
 serializes and folds address headers properly—even ones with unicode 
 names. It enforces single-instance headers where appropriate.
 
  • *Easier to work with:* The modern API adds some nice conveniences for 
 developers working in django.core.mail, third-party library developers, 
 and (depending on what we choose to expose) users of Django's mail APIs.
 
 Examples: populating the "Date" header with a datetime or 

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Arthur Pemberton]

>  The background workers proposal will implement a new background SMTP
EmailBackend (in django.core.mail.backends).

I had missed that fact. Thanks for the explanation.

I for one think that this is a good proposal -- this would modern
transactional email sending.

- Arthur


On Mon, Jun 24, 2024 at 3:14 PM Mike Edmunds  wrote:

> > Would this be designed to be compatible with "Proposal 14: Background
> Workers"?
>
> I wouldn't expect this to impact background workers one way or the other.
> The same goes for the async email proposal(s) floating around.
>
> Virtually all of this work would occur in django.core.mail.message, where
> Python's `email` APIs are used. A goal is to avoid changes that would break
> existing email backends (Django or third-party).
>
> The background workers proposal will implement a new background SMTP
> EmailBackend (in django.core.mail.backends). The existing SMTP EmailBackend
> doesn't directly use Python's `email` APIs, and there should be no reason
> for background SMTP to be any different. (It might be helpful to know that
> Python's `email` library isn't involved in *sending* email; that's
> handled by Python's `smtplib`, which Django uses only in the SMTP
> EmailBackend.)
>
> The existing SMTP EmailBackend *does* use one function I expect will
> become deprecated: django.core.mail.message.sanitize_address(). I haven't
> yet investigated whether that use is still necessary, or whether it's there
> to get around past bugs/limitations in Python's smtplib. If any of it is
> still needed for SMTP, I'd probably want to move that code into the SMTP
> EmailBackend(s).
>
> - Mike
> On Sunday, June 23, 2024 at 10:22:03 PM UTC-7 Arthur Pemberton wrote:
>
>> Would this be designed to be compatible with "Proposal 14: Background
>> Workers"?
>>
>>
>>
>> On Sun, Jun 23, 2024 at 10:46 PM Mike Edmunds  wrote:
>>
>>> I want to propose updating django.core.mail to replace use of Python's
>>> legacy email.message.Message (and other legacy email APIs) with
>>> email.message.EmailMessage (and other modern APIs).
>>>
>>> If there's interest, I can put together a more detailed proposal and/or
>>> ticket, but was hoping to get some initial feedback first. (I searched for
>>> relevant discussions in django-developers and the issue tracker, and didn't
>>> find any. Apologies if this has come up before.)
>>>
>>> [Note: I maintain django-anymail
>>> , which implements Django
>>> integration with several transactional email service providers.]
>>>
>>>
>>> *Background*
>>>
>>> Since Python ~3.6, Python's email package has included two largely
>>> separate implementations:
>>>
>>>- a "modern (unicode friendly) API" based on
>>>email.message.EmailMessage and email.policy.default
>>>- a "legacy API" providing compatibility with older Python versions,
>>>based on email.message.Message and email.policy.compat32
>>>
>>> (See https://docs.python.org/3/library/email.html, especially toward
>>> the end.)
>>>
>>> django.core.mail currently uses the legacy API.
>>>
>>>
>>> *Why switch?*
>>>
>>> There are no plans to deprecate Python's legacy email API, and it's
>>> working, so why change Django?
>>>
>>>-
>>>
>>>*Fewer bugs:* The modern API fixes a *lot* of bugs in the legacy
>>>API. My understanding is legacy bugs will generally not be fixed. (And in
>>>fact, there are some cases where the legacy API deliberately replicates
>>>earlier buggy behavior.)
>>>
>>>Django #35497  is an
>>>example of a legacy bug (with a problematic proposed workaround) which is
>>>just fixed in the modern API.
>>>-
>>>
>>>*Simpler, safer code:* A substantial portion
>>>
>>> 
>>>  of
>>>django.core.mail's internals implements workarounds and security fixes 
>>> for
>>>problems in the legacy API. This would be greatly simplified—maybe
>>>eliminated completely—using the modern API.
>>>
>>>Examples: the modern API prevents CR/NL injections in message
>>>headers. It serializes and folds address headers properly—even ones with
>>>unicode names. It enforces single-instance headers where appropriate.
>>>-
>>>
>>>*Easier to work with:* The modern API adds some nice conveniences
>>>for developers working in django.core.mail, third-party library 
>>> developers,
>>>and (depending on what we choose to expose) users of Django's mail APIs.
>>>
>>>Examples: populating the "Date" header with a datetime or an address
>>>header with Address
>>>
>>> 
>>>  objects—without
>>>needing intricate knowledge of email header formats. Using
>>>email.policy to generate a 7-bit clean serialization (without having
>>>to muck about with the MIME parts
>>>
>>> 

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

> Would this be designed to be compatible with "Proposal 14: Background 
Workers"?

I wouldn't expect this to impact background workers one way or the other. 
The same goes for the async email proposal(s) floating around.

Virtually all of this work would occur in django.core.mail.message, where 
Python's `email` APIs are used. A goal is to avoid changes that would break 
existing email backends (Django or third-party).

The background workers proposal will implement a new background SMTP 
EmailBackend (in django.core.mail.backends). The existing SMTP EmailBackend 
doesn't directly use Python's `email` APIs, and there should be no reason 
for background SMTP to be any different. (It might be helpful to know that 
Python's `email` library isn't involved in *sending* email; that's handled 
by Python's `smtplib`, which Django uses only in the SMTP EmailBackend.)

The existing SMTP EmailBackend *does* use one function I expect will become 
deprecated: django.core.mail.message.sanitize_address(). I haven't yet 
investigated whether that use is still necessary, or whether it's there to 
get around past bugs/limitations in Python's smtplib. If any of it is still 
needed for SMTP, I'd probably want to move that code into the SMTP 
EmailBackend(s).

- Mike
On Sunday, June 23, 2024 at 10:22:03 PM UTC-7 Arthur Pemberton wrote:

> Would this be designed to be compatible with "Proposal 14: Background 
> Workers"?
>
>
>
> On Sun, Jun 23, 2024 at 10:46 PM Mike Edmunds  wrote:
>
>> I want to propose updating django.core.mail to replace use of Python's 
>> legacy email.message.Message (and other legacy email APIs) with 
>> email.message.EmailMessage (and other modern APIs).
>>
>> If there's interest, I can put together a more detailed proposal and/or 
>> ticket, but was hoping to get some initial feedback first. (I searched for 
>> relevant discussions in django-developers and the issue tracker, and didn't 
>> find any. Apologies if this has come up before.)
>>
>> [Note: I maintain django-anymail 
>> , which implements Django 
>> integration with several transactional email service providers.]
>>
>>
>> *Background*
>>
>> Since Python ~3.6, Python's email package has included two largely 
>> separate implementations:
>>
>>- a "modern (unicode friendly) API" based on 
>>email.message.EmailMessage and email.policy.default
>>- a "legacy API" providing compatibility with older Python versions, 
>>based on email.message.Message and email.policy.compat32
>>
>> (See https://docs.python.org/3/library/email.html, especially toward the 
>> end.)
>>
>> django.core.mail currently uses the legacy API.
>>
>>
>> *Why switch?*
>>
>> There are no plans to deprecate Python's legacy email API, and it's 
>> working, so why change Django?
>>
>>- 
>>
>>*Fewer bugs:* The modern API fixes a *lot* of bugs in the legacy API. 
>>My understanding is legacy bugs will generally not be fixed. (And in 
>> fact, 
>>there are some cases where the legacy API deliberately replicates earlier 
>>buggy behavior.)
>>
>>Django #35497  is an 
>>example of a legacy bug (with a problematic proposed workaround) which is 
>>just fixed in the modern API.
>>- 
>>
>>*Simpler, safer code:* A substantial portion 
>>
>> 
>>  of 
>>django.core.mail's internals implements workarounds and security fixes 
>> for 
>>problems in the legacy API. This would be greatly simplified—maybe 
>>eliminated completely—using the modern API.
>>
>>Examples: the modern API prevents CR/NL injections in message 
>>headers. It serializes and folds address headers properly—even ones with 
>>unicode names. It enforces single-instance headers where appropriate.
>>- 
>>
>>*Easier to work with:* The modern API adds some nice conveniences for 
>>developers working in django.core.mail, third-party library developers, 
>> and 
>>(depending on what we choose to expose) users of Django's mail APIs.
>>
>>Examples: populating the "Date" header with a datetime or an address 
>>header with Address 
>>
>> 
>>  objects—without 
>>needing intricate knowledge of email header formats. Using 
>>email.policy to generate a 7-bit clean serialization (without having 
>>to muck about with the MIME parts 
>>
>> 
>>).
>>
>>
>> *Concerns & risks*
>>
>> Compatibility and security, of course…
>>
>>- 
>>
>>*Backwards compatibility (for API users):* django.core.mail largely 
>>insulates callers from the underlying Python email package. There are a 
>> few 
>>places where this leaks (e.g., attachmen

Re: Proposal to upgrade django.core.mail to Python's modern email API

[Arthur Pemberton]

Would this be designed to be compatible with "Proposal 14: Background
Workers"?



On Sun, Jun 23, 2024 at 10:46 PM Mike Edmunds  wrote:

> I want to propose updating django.core.mail to replace use of Python's
> legacy email.message.Message (and other legacy email APIs) with
> email.message.EmailMessage (and other modern APIs).
>
> If there's interest, I can put together a more detailed proposal and/or
> ticket, but was hoping to get some initial feedback first. (I searched for
> relevant discussions in django-developers and the issue tracker, and didn't
> find any. Apologies if this has come up before.)
>
> [Note: I maintain django-anymail
> , which implements Django
> integration with several transactional email service providers.]
>
>
> *Background*
>
> Since Python ~3.6, Python's email package has included two largely
> separate implementations:
>
>- a "modern (unicode friendly) API" based on
>email.message.EmailMessage and email.policy.default
>- a "legacy API" providing compatibility with older Python versions,
>based on email.message.Message and email.policy.compat32
>
> (See https://docs.python.org/3/library/email.html, especially toward the
> end.)
>
> django.core.mail currently uses the legacy API.
>
>
> *Why switch?*
>
> There are no plans to deprecate Python's legacy email API, and it's
> working, so why change Django?
>
>-
>
>*Fewer bugs:* The modern API fixes a *lot* of bugs in the legacy API.
>My understanding is legacy bugs will generally not be fixed. (And in fact,
>there are some cases where the legacy API deliberately replicates earlier
>buggy behavior.)
>
>Django #35497  is an
>example of a legacy bug (with a problematic proposed workaround) which is
>just fixed in the modern API.
>-
>
>*Simpler, safer code:* A substantial portion
>
> 
>  of
>django.core.mail's internals implements workarounds and security fixes for
>problems in the legacy API. This would be greatly simplified—maybe
>eliminated completely—using the modern API.
>
>Examples: the modern API prevents CR/NL injections in message headers.
>It serializes and folds address headers properly—even ones with unicode
>names. It enforces single-instance headers where appropriate.
>-
>
>*Easier to work with:* The modern API adds some nice conveniences for
>developers working in django.core.mail, third-party library developers, and
>(depending on what we choose to expose) users of Django's mail APIs.
>
>Examples: populating the "Date" header with a datetime or an address
>header with Address
>
> 
>  objects—without
>needing intricate knowledge of email header formats. Using email.policy to
>generate a 7-bit clean serialization (without having to muck about
>with the MIME parts
>
> 
>).
>
>
> *Concerns & risks*
>
> Compatibility and security, of course…
>
>-
>
>*Backwards compatibility (for API users):* django.core.mail largely
>insulates callers from the underlying Python email package. There are a few
>places where this leaks (e.g., attachments
>
> 
>  allows
>legacy email MIMEBase
> objects), but in
>general the switch should be transparent. (And I have some ideas for
>supporting the other cases.)
>-
>
>*Backwards compatibility (for third-party libraries):* Some libraries
>may use internals I'd propose removing (e.g., SafeMIME and friends);
>we'd handle this through deprecation.
>-
>
>*Backwards compatibility (bug-level):* There's probably some code out
>there that unintentionally depends on legacy email bugs (or the specific
>ways Django works around them). I don't have any examples, but I also don't
>have a good solution for when they surface. Plus, while Python's modern
>email API is pretty mature at this point, there are still new bugs being
>reported against it. Email is complicated.
>-
>
>*Security:* As noted above, the modern API should be more secure than
>the legacy one. But we also have a nice collection of email security
>tests—which *mostly* don't depend on internal implementation. We'd
>keep these.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+

Re: Proposal to upgrade django.core.mail to Python's modern email API

['Mohamed El-Kalioby' via Django developers (Contributions to Django itself)]

+1, I like to help in this

On Mon, 24 Jun 2024, 05:46 Mike Edmunds,  wrote:

> I want to propose updating django.core.mail to replace use of Python's
> legacy email.message.Message (and other legacy email APIs) with
> email.message.EmailMessage (and other modern APIs).
>
> If there's interest, I can put together a more detailed proposal and/or
> ticket, but was hoping to get some initial feedback first. (I searched for
> relevant discussions in django-developers and the issue tracker, and didn't
> find any. Apologies if this has come up before.)
>
> [Note: I maintain django-anymail
> , which implements Django
> integration with several transactional email service providers.]
>
>
> *Background*
>
> Since Python ~3.6, Python's email package has included two largely
> separate implementations:
>
>- a "modern (unicode friendly) API" based on
>email.message.EmailMessage and email.policy.default
>- a "legacy API" providing compatibility with older Python versions,
>based on email.message.Message and email.policy.compat32
>
> (See https://docs.python.org/3/library/email.html, especially toward the
> end.)
>
> django.core.mail currently uses the legacy API.
>
>
> *Why switch?*
>
> There are no plans to deprecate Python's legacy email API, and it's
> working, so why change Django?
>
>-
>
>*Fewer bugs:* The modern API fixes a *lot* of bugs in the legacy API.
>My understanding is legacy bugs will generally not be fixed. (And in fact,
>there are some cases where the legacy API deliberately replicates earlier
>buggy behavior.)
>
>Django #35497  is an
>example of a legacy bug (with a problematic proposed workaround) which is
>just fixed in the modern API.
>-
>
>*Simpler, safer code:* A substantial portion
>
> 
>  of
>django.core.mail's internals implements workarounds and security fixes for
>problems in the legacy API. This would be greatly simplified—maybe
>eliminated completely—using the modern API.
>
>Examples: the modern API prevents CR/NL injections in message headers.
>It serializes and folds address headers properly—even ones with unicode
>names. It enforces single-instance headers where appropriate.
>-
>
>*Easier to work with:* The modern API adds some nice conveniences for
>developers working in django.core.mail, third-party library developers, and
>(depending on what we choose to expose) users of Django's mail APIs.
>
>Examples: populating the "Date" header with a datetime or an address
>header with Address
>
> 
>  objects—without
>needing intricate knowledge of email header formats. Using email.policy to
>generate a 7-bit clean serialization (without having to muck about
>with the MIME parts
>
> 
>).
>
>
> *Concerns & risks*
>
> Compatibility and security, of course…
>
>-
>
>*Backwards compatibility (for API users):* django.core.mail largely
>insulates callers from the underlying Python email package. There are a few
>places where this leaks (e.g., attachments
>
> 
>  allows
>legacy email MIMEBase
> objects), but in
>general the switch should be transparent. (And I have some ideas for
>supporting the other cases.)
>-
>
>*Backwards compatibility (for third-party libraries):* Some libraries
>may use internals I'd propose removing (e.g., SafeMIME and friends);
>we'd handle this through deprecation.
>-
>
>*Backwards compatibility (bug-level):* There's probably some code out
>there that unintentionally depends on legacy email bugs (or the specific
>ways Django works around them). I don't have any examples, but I also don't
>have a good solution for when they surface. Plus, while Python's modern
>email API is pretty mature at this point, there are still new bugs being
>reported against it. Email is complicated.
>-
>
>*Security:* As noted above, the modern API should be more secure than
>the legacy one. But we also have a nice collection of email security
>tests—which *mostly* don't depend on internal implementation. We'd
>keep these.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on t