[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

miss-islington added the comment: New changeset d8576b1d15155688a67baac24c15254700bdd3b7 by Miss Islington (bot) in branch '3.9': bpo-26227: Fixes decoding of host names on Windows from ANSI instead of UTF-8 (GH-25510)

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

miss-islington added the comment: New changeset f7bc44170b4bdd16c46b4b6acf7673ffc24dfb19 by Miss Islington (bot) in branch '3.8': bpo-26227: Fixes decoding of host names on Windows from ANSI instead of UTF-8 (GH-25510)

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Change by Steve Dower : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Steve Dower added the comment: New changeset dc516ef8395d15da0ab225eb0dceb2e0581f51ca by Steve Dower in branch 'master': bpo-26227: Fixes decoding of host names on Windows from ANSI instead of UTF-8 (GH-25510) https://github.com/python/cpython/commit/dc516ef8395d15da0ab225eb0dceb2e0581f51ca

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Change by miss-islington : -- pull_requests: +24231 pull_request: https://github.com/python/cpython/pull/25513 ___ Python tracker ___

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Change by miss-islington : -- nosy: +miss-islington nosy_count: 11.0 -> 12.0 pull_requests: +24230 pull_request: https://github.com/python/cpython/pull/25512 ___ Python tracker

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Change by Steve Dower : -- assignee: -> steve.dower ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Change by Steve Dower : -- keywords: +patch pull_requests: +24228 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/25510 ___ Python tracker

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Change by Eryk Sun : -- components: +Library (Lib) keywords: +3.10regression, 3.6regression, 3.7regression, 3.8regression, 3.9regression -patch stage: test needed -> needs patch type: crash -> behavior versions: +Python 3.10 -Python 3.7 ___ Python

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Steve Dower added the comment: I think PyUnicode_DecodeMBCS(name, strlen(name), "surrogatepass") captures the intention better, and is less likely to break in the future (apart from all the ways it's currently broken :) ) You should be right about the workaround too. --

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

STINNER Victor added the comment: sock_decode_hostname() of socketmodule.c currently uses PyUnicode_DecodeFSDefault() on Windows. PyUnicode_DecodeFSDefault() uses UTF-8 by default (PEP 529). I understand that the ANSI code page should be used instead of UTF-8. Would it work to use

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Steve Dower added the comment: If someone wants to do the extra work to use the native Windows socket functions on Windows instead of the POSIX-ey wrappers, that should happen under a new issue. This bug is for a regression that we ought to fix back as far as we can. -- priority:

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Steve Dower added the comment: Looks like what was actually applied was changed to PyUnicode_DecodeFSDefault, which was later changed on Windows to be always UTF-8. They'll need to be normalised throughout Modules/socketmodule.c (I can't tell at a quick glance which need updating). We

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Владимир Мартьянов added the comment: I have Python 3.7.4 and have the same exception: print (socket.gethostbyaddr("127.0.0.1")) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcf in position 6: invalid continuation byte My OS is Win7 and my computer name contains cyrillic characters.

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

William Dias added the comment: Shouldn't this issue be solved for Python 3.7.5? Or do I have to manually apply the patch? I have a windows 8.1 x64 PC whose hostname contains special characters. When creating a socket, the gethostbyaddr() method raises a UnicodeDecodeError: 'utf-8' codec

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Emanuel Barry added the comment: For future reference, Victor's patch does fix it, I was checking the wrong thing when testing. -- ___ Python tracker

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

STINNER Victor added the comment: He he, no problem. Thanks again for the bug report. I'm surprised that nobody reported it before. -- ___ Python tracker

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Roundup Robot added the comment: New changeset 0681f0a1fe6e by Victor Stinner in branch '3.5': Windows: Decode hostname from ANSI code page https://hg.python.org/cpython/rev/0681f0a1fe6e New changeset 26f6d8cc2749 by Victor Stinner in branch 'default': Merge 3.5: Issue #26227

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

STINNER Victor added the comment: Steve: """ I couldn't remember the names of the alternate functions Windows provides to do the encoding for you, but yes. There are socket APIs there that do encoding and handle memory allocation more safely. Apart from bugs like this, it's not really urgent

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Emanuel Barry added the comment: Yes, it's not all that urgent. And Victor's latest patch doesn't work, either :( I wonder if there's a way to (temporarily) modify the output of ``socket.gethostname()`` to be able to test such weird corner cases. --

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

STINNER Victor added the comment: > Yes, it's not all that urgent. And Victor's latest patch doesn't work, either > :( Could you please elaborate? The patch applies cleanly? You rebuild the socket module? Which error message do you get? -- ___

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

STINNER Victor added the comment: By the way, thanks for your reviews. Code review rocks ;-) -- ___ Python tracker ___

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

STINNER Victor added the comment: > Added comments on Rietveld. Crap. It's easy to miss a compilation error on extensions :-/ I used "make && ./python -m test -v test_socket" to validate gethostbyaddr_encoding-2.patch and it succeded. Maybe we should setup.py to *fail* if an extension

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Eryk Sun added the comment: The patch is missing the "errors" parameter of PyUnicode_DecodeLocale. But it should call PyUnicode_DecodeMBCS instead. In the "C" locale, PyUnicode_DecodeLocale is Latin-1 because the CRT mbstowcs just casts the values to wchar_t. socket_getnameinfo also decodes

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

STINNER Victor added the comment: > The patch is missing the "errors" parameter of PyUnicode_DecodeLocale. Woops, I shouldn't write patch in the middle of the night :-) Hopefully, I didn't push it :-) PyUnicode_DecodeLocale() should only be used when the encoding depends on the *currenet*

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Changes by STINNER Victor : Added file: http://bugs.python.org/file41740/gethostbyaddr_encoding-2.patch ___ Python tracker ___

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Serhiy Storchaka added the comment: Added comments on Rietveld. -- nosy: +serhiy.storchaka ___ Python tracker ___

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Emanuel Barry added the comment: If it worked for you, I assume it's fine and I probably did something wrong on my side. Thanks for the fix! -- ___ Python tracker

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Steve Dower added the comment: I couldn't remember the names of the alternate functions Windows provides to do the encoding for you, but yes. There are socket APIs there that do encoding and handle memory allocation more safely. Apart from bugs like this, it's not really urgent and it

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

New submission from STINNER Victor: On Windows, socket.gethostbyaddr() must decode the hostname from the ANSI code page, not from UTF-8. See for example this issue: https://bugs.python.org/issue26226#msg259077 Attached patch changes the socket module to decode hostnames from the ANSI code

[issue26227] Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname

Changes by STINNER Victor : -- keywords: +patch Added file: http://bugs.python.org/file41734/gethostbyaddr_encoding.patch ___ Python tracker