[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

Serhiy Storchaka added the comment: Python 2.7 is no longer supported. -- nosy: +serhiy.storchaka resolution: -> out of date stage: test needed -> resolved status: open -> closed ___ Python tracker

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

Changes by Éric Araujo mer...@netwok.org: -- nosy: +aronacher ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12398 ___ ___ Python-bugs-list mailing

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

Gregory P. Smith added the comment: I'm running into this on 2.7.3 with code that worked fine on 2.6.5. The problem appears to be caused by a 'Host' http header that has a unicode type for the hostname:port value. Encoding header values makes sense though I haven't yet examined the patch in

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

Adam Cohen a...@seatgeek.com added the comment: I encountered this issue as well. params is simply a bytestring, with no encoding. Workaround/proper solution is to cast the string as a bytearray with bytearray(params). -- nosy: +Adam.Cohen ___

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

STINNER Victor victor.stin...@haypocalc.com added the comment: Here is a patch for httplib encoding HTTP headers to ISO-8859-1, as done in Python 3 (see HTTPConnection.putheader() from http.client). urllib is not affected by this issue because it does already encode Unicode, but encode to

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

Changes by Senthil Kumaran sent...@uthcode.com: -- assignee: - orsenthil nosy: +orsenthil ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12398 ___

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

Changes by Éric Araujo mer...@netwok.org: -- nosy: +eric.araujo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12398 ___ ___ Python-bugs-list

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

Changes by Thijs Triemstra li...@collab.nl: -- nosy: +thijs ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12398 ___ ___ Python-bugs-list mailing

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

sorin sorin.sbar...@gmail.com added the comment: Here is a test file that will replicate the problem, I added it as a gist so it could support contributions ;) Py 2.7 works Py ==2.7 fails Py =3.0 works after minor changes required by py3k https://gist.github.com/1047551 --

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

R. David Murray rdmur...@bitdance.com added the comment: rdmurraypython2.6 py27-str-unicode-bytes.py type(b)=type 'str' Traceback (most recent call last): File py27-str-unicode-bytes.py, line 17, in module unicode_str += b # this line will throw UnicodeDecodeError on Python 2.7

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

Changes by STINNER Victor victor.stin...@haypocalc.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12398 ___ ___

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

R. David Murray rdmur...@bitdance.com added the comment: To clarify: if I convert your program to using strings pre2.6, it still fails with a UnicodeDecodeError, as one would expect. bytes are strings in 2.x. -- ___ Python tracker

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

R. David Murray rdmur...@bitdance.com added the comment: And finally, your program does *not* succeed on Python3, except in the trivial sense that on python3 you never attempt to add the string and bytes data. It is exactly this kind of programming error that Python3 is designed to avoid:

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

sorin sorin.sbar...@gmail.com added the comment: Right, so you have some binary data and you want to sent it to `httplib`. This worked in the past when `msg` was a non-unicode string, but starting with Python 2.7 this became an unicode string, so when you try to append the `message` if will

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

R. David Murray rdmur...@bitdance.com added the comment: But senthil already demonstrated in the previous issue that it does not become a unicode string unless you use unicode input. You also claimed that your test program here succeeded in python2.6, but it does not. This casts a little bit

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

sorin sorin.sbar...@gmail.com added the comment: I updated the gist and made a minimal test https://gist.github.com/1047551 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12398 ___

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

sorin sorin.sbar...@gmail.com added the comment: You are right, I debugged the problem a little more and discovered at least one bug in PyAMF. Still, I was surprised to find out something very strange, it look that BytesIO.getvalue() does return `str` even if the documentation says it does

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

R. David Murray rdmur...@bitdance.com added the comment: No, that's correct. In python 2.x the 'bytes' stuff is just a portability aid. In 2.x, bytes and string are the same type. In Python 3 they aren't, so by using the 'fake' classes in python2 you can often make your code work correctly

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

Terry J. Reedy tjre...@udel.edu added the comment: In 2.7, bytes is an alias for str to aid porting to 3.x. bytes is str True type(bytes()) type 'str' I suspect the doc uses 'bytes' rather than 'str' because it was backported from 3.x. Perhaps it should be changed but I do not know the

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

New submission from sorin sorin.sbar...@gmail.com: It looks that Python 2.7 changes did induce some important bugs into httplib due to to implicit str-unicode encoding/decoding. One clear example is that PyAMF library doesn't work with Python 2.7 because it is not able to generate binary data

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

R. David Murray rdmur...@bitdance.com added the comment: If this worked in 2.6 and fails in 2.7, it would probably be helpful if we can determine what change broke it. I believe hg has some sort of 'bisect' support that might make this not too onerous to do. Senthil (or someone) will

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

Terry J. Reedy tjre...@udel.edu added the comment: A crash is a segfault or equivalent. Python 2.6 only gets security fixes. PyAMF does not run on Python 3. Hence a problem with PyAMF is no evidence of a problem with 3.x. Separate tests/examples would be needed. Changes are not bugs unless

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

Changes by Terry J. Reedy tjre...@udel.edu: -- stage: - test needed type: crash - behavior versions: -Python 3.1, Python 3.2, Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12398

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

Terry J. Reedy tjre...@udel.edu added the comment: Did things like u'a'+'\xf0' work in 2.6- (with implicit latin-1 decoding)? (I do not have 2.6 loaded.) The doc for seq+seq (concatenation) in the language reference section 5.6. Binary arithmetic operations says that both sequences must be

[issue12398] Sending binary data with a POST request in httplib can cause Unicode exceptions

R. David Murray rdmur...@bitdance.com added the comment: Many applications and libraries say Python X.Y or newer, and it is one of the strengths of Python that this will often be true. That's what our backward compatibility policy is about, and that's why the fact that it isn't true for