[issue19662] smtpd.py should not decode utf-8

Arfrever Frehtes Taifersar Arahesis added the comment: New changeset a7d3074fa888 by R David Murray in branch 'default': #19662: Make requirement to support arbitrary keywords explicit. https://hg.python.org/cpython/rev/a7d3074fa888 s/keword/keyword/ -- nosy: +Arfrever

[issue19662] smtpd.py should not decode utf-8

Roundup Robot added the comment: New changeset a3f2b171b765 by R David Murray in branch 'default': #19662: fix typo https://hg.python.org/cpython/rev/a3f2b171b765 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19662

[issue19662] smtpd.py should not decode utf-8

R. David Murray added the comment: Thanks, Arfrever. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19662 ___ ___ Python-bugs-list mailing list

[issue19662] smtpd.py should not decode utf-8

Roundup Robot added the comment: New changeset a7d3074fa888 by R David Murray in branch 'default': #19662: Make requirement to support arbitrary keywords explicit. https://hg.python.org/cpython/rev/a7d3074fa888 -- ___ Python tracker

[issue19662] smtpd.py should not decode utf-8

Roundup Robot added the comment: New changeset 4e22213ca275 by R David Murray in branch 'default': #19662: add decode_data to smtpd so you can get at DATA in bytes form. http://hg.python.org/cpython/rev/4e22213ca275 -- nosy: +python-dev ___ Python

[issue19662] smtpd.py should not decode utf-8

R. David Murray added the comment: Thanks, Maciej. I tweaked the patch a bit, you might want to take a look just for your own information. Mostly I fixed the warning stuff, which I didn't explain very well. The idea is that if the default is used (no value is specified), we want there to

[issue19662] smtpd.py should not decode utf-8

Roundup Robot added the comment: New changeset a6c846ec5fd3 by R David Murray in branch 'default': #19662: Eliminate warnings in other test modules that use smtpd. http://hg.python.org/cpython/rev/a6c846ec5fd3 -- ___ Python tracker

[issue19662] smtpd.py should not decode utf-8

Changes by Milan Oberkirch milan...@oberkirch.org: -- nosy: +jesstess, zvyn ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19662 ___ ___

[issue19662] smtpd.py should not decode utf-8

Maciej Szulik added the comment: I've included Leslie's comments in rst file. The 3rd version is attached in issue19662_v3.patch. -- Added file: http://bugs.python.org/file35409/issue19662_v3.patch ___ Python tracker rep...@bugs.python.org

[issue19662] smtpd.py should not decode utf-8

R. David Murray added the comment: Added review comments. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19662 ___ ___ Python-bugs-list mailing

[issue19662] smtpd.py should not decode utf-8

Maciej Szulik added the comment: I've implemented all your proposed changes, because for most of your changes I was thinking pretty the same way for the whole day today, to make the code more elegant. The current state of work is attached as issue19662_v2.patch -- Added file:

[issue19662] smtpd.py should not decode utf-8

Maciej Szulik added the comment: I'm attaching file issue19662_v1.patch. David please have a look at it and let me know if this is it, if not I'm waiting for your suggestions. -- Added file: http://bugs.python.org/file35390/issue19662_v1.patch ___

[issue19662] smtpd.py should not decode utf-8

R. David Murray added the comment: Yes, this will be fixed in 3.5 one way or another. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19662 ___

[issue19662] smtpd.py should not decode utf-8

Maciej Szulik added the comment: I'll try to take care of this issue in the following few days. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19662 ___

[issue19662] smtpd.py should not decode utf-8

Duke Dougal added the comment: Is this one likely to be included in 3.5? It effectively breaks smtpd so it would be good to see it working again. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19662

[issue19662] smtpd.py should not decode utf-8

Sreepriya Chalakkal added the comment: Hi Maciej, I am travelling now and it might take some delay for me to work on this! I got to know that you are working on RFC 6532. You might take this up and fix it as this is related to your work and I don't want to create delays. --

[issue19662] smtpd.py should not decode utf-8

Maciej Szulik added the comment: Sreepriya, are you still working on this issue? If no I'll be happy to take it over, is yes start with fixing following things: - start with test - this is the most important to have each feautre tested - decode_data, as David mentioned, needs to have default

[issue19662] smtpd.py should not decode utf-8

Sreepriya Chalakkal added the comment: Hi David, The variable decode_data is included to control decoding. But I am not sure what needs to be done while calling the process_message inside found_terminator when it is binary data. How to work around with binary data? Can you tell me what are

[issue19662] smtpd.py should not decode utf-8

Changes by Sreepriya Chalakkal sreepriya1...@gmail.com: Added file: http://bugs.python.org/file34704/switch_while_decode2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19662 ___

[issue19662] smtpd.py should not decode utf-8

R. David Murray added the comment: I propose that we add a new keyword argument to SMTP's __init__, 'decode_data'. This would be set to True by default, and would preserve the current behavior of passing utf-8 decoded data to process_message. Setting it to True would mean that

[issue19662] smtpd.py should not decode utf-8

Sreepriya Chalakkal added the comment: Hi David, I would like to work on this bug. Can you give some more insights about the main issue? As far as I understood, the smtp server is now decoding the incoming bytes as UTF-8. Why do you say that it is not the right way? Can you give some idea

[issue19662] smtpd.py should not decode utf-8

Duke Dougal added the comment: Is there a workaround for this as I'd like to just be receiving binary data from SMTPD. I'm new to this system - is this scheduled for fixing in Python 3.4? -- nosy: +Duke.Dougal ___ Python tracker

[issue19662] smtpd.py should not decode utf-8

R. David Murray added the comment: Unfortunately I did not get to this before the 3.4 beta release, so no, it won't be fixed in 3.4. You can work around it by overriding collect_incoming_data in your subclass and doing data.decode('ascii', 'surrogateescape') instead of str(data, 'utf-8'),

[issue19662] smtpd.py should not decode utf-8

Illirgway added the comment: Here is another patch for fixing this issue: https://github.com/Illirgway/cpython/commit/12d7c59e0564c408a65dd782339f585ab6b14b34 Sorry for my bad english -- nosy: +Illirgway versions: +Python 3.3 -Python 3.5 Added file:

[issue19662] smtpd.py should not decode utf-8

Changes by Antoine Pitrou pit...@free.fr: -- stage: - patch review versions: +Python 3.4 -Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19662 ___

[issue19662] smtpd.py should not decode utf-8

R. David Murray added the comment: As I said, the decoding needs to be controlled by a switch (presumably a keyword argument to SMTPServer) that defaults to the present (incorrect) behavior. -- ___ Python tracker rep...@bugs.python.org

[issue19662] smtpd.py should not decode utf-8

Changes by R. David Murray rdmur...@bitdance.com: -- versions: +Python 3.5 -Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19662 ___ ___

[issue19662] smtpd.py should not decode utf-8

New submission from Leslie P. Polzer: http://hg.python.org/cpython/file/3.3/Lib/smtpd.py#l289 as of now decodes incoming bytes as UTF-8. An SMTP server must not attempt to interpret characters beyond ASCII, however. Originally mail servers were not 8-bit clean, meaning they would only

[issue19662] smtpd.py should not decode utf-8

Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19662 ___ ___ Python-bugs-list

[issue19662] smtpd.py should not decode utf-8

Leslie P. Polzer added the comment: Patch attached. This also adds some more charset clarification to the docs and corrects a minor spelling issue. It is also conceivable that we add a charset attribute to the class. This should have the safe default of latin1, and some notes in the docs that

[issue19662] smtpd.py should not decode utf-8

R. David Murray added the comment: This bug was apparently introduced as part of the work from issue 4184 in python 3.2. My guess, looking at the code, is that the module simply didn't work before that patch, since it would have been attempting to join binary data using a string join

[issue19662] smtpd.py should not decode utf-8

Leslie P. Polzer added the comment: Since this is my first contribution I'm not entirely sure about the fine details of backwards compatibility in Python, so please forgive me if I'm totally missing the mark here. There are facilities in smtpd's parent class asynchat that perform the

[issue19662] smtpd.py should not decode utf-8

R. David Murray added the comment: I think the only backward compatible solution is to add a switch of *some* sort (exact API TBD), whose default is to continue to decode using utf-8, and document it as wrong. Conversion of an email to unicode should be handled by the email package, not by

[issue19662] smtpd.py should not decode utf-8

R. David Murray added the comment: Oh, and to clarify: the backward compatibility is that if code works with X.Y.Z, it should work with X.Y.Z+1. So even though correctly handling binary mail would indeed require someone to reexamine their code, if things happen to be working OK for them (eg: