[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
Andrew Svetlov added the comment: Looks good for me, but please use assertEqual instead assertEquals. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
Chris Jerdonek added the comment: Thank you, Andrew. Here is a patch updated with that change (and also merging with tip). -- Added file: http://bugs.python.org/file26899/issue-15595-4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
Roundup Robot added the comment: New changeset a0f7c2f79bce by Andrew Svetlov in branch '3.2': Issue #15595: Fix subprocess.Popen(universal_newlines=True) http://hg.python.org/cpython/rev/a0f7c2f79bce New changeset aceb820154c3 by Andrew Svetlov in branch 'default': Issue #15595: Fix subprocess.Popen(universal_newlines=True) http://hg.python.org/cpython/rev/aceb820154c3 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
Andrew Svetlov added the comment: Fixed. Thank you. -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
Changes by Chris Jerdonek chris.jerdo...@gmail.com: -- nosy: +asvetlov ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
New submission from Chris Jerdonek:
subprocess.Popen() with universal_newlines=True does not convert line breaks
correctly when the preferred encoding is UTF-16. For example, the following
code--
code = rimport sys; sys.stdout.buffer.write('a\r\nb'.encode('utf-16'))
args = [sys.executable, '-c', code]
popen = Popen(args, universal_newlines=True, stdin=PIPE, stdout=PIPE)
print(popen.communicate(input=''))
yields--
('a\n\nb', None)
instead of--
('a\nb', None)
The reason is that the code attempts to convert newlines before decoding to
unicode instead of after:
http://hg.python.org/cpython/file/85266c6f9ae4/Lib/subprocess.py#l830
I am attaching a failing test case. I will upload a patch shortly.
Also see the related documentation issue 15561.
--
components: Library (Lib)
files: failing-test-case-1.patch
keywords: easy, patch
messages: 167719
nosy: cjerdonek
priority: normal
severity: normal
stage: needs patch
status: open
title: subprocess.Popen(universal_newlines=True) does not work for certain
locales
type: behavior
versions: Python 3.3
Added file: http://bugs.python.org/file26728/failing-test-case-1.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15595
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
STINNER Victor added the comment: when the preferred encoding is UTF-16 What is your operating system? How do you set the locale encoding to UTF-16? Do you mean UTF-16, UTF-16-LE or UTF-16-BE? -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
Chris Jerdonek added the comment: Attaching patch for review. -- stage: needs patch - patch review Added file: http://bugs.python.org/file26729/issue-15595-1.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
Chris Jerdonek added the comment: Victor, I have not personally experienced this issue. I just noticed that the order of operations is wrong or not portable in the _translate_newlines() method when I was looking at the code for another reason. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
Chris Jerdonek added the comment: How do you set the locale encoding to UTF-16? Do you mean UTF-16, UTF-16-LE or UTF-16-BE? I confirmed that the issue occurs for all of these. For testing purposes, you can do-- locale.getpreferredencoding = lambda do_setlocale: 'utf-16' -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: +loewis ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
STINNER Victor added the comment: There is a bug, but I'm not conviced that multibyte encodings are used as locale encoding. About your patch: you should test the 3 types of newlines, so use a string like: '1\r\n2\r3\n4'. +# Popen() defaults to locale.getpreferredencoding(False). +locale.getpreferredencoding = lambda do_setlocale: 'utf-16' FYI it's not directly Popen() which uses the locale encoding, but TextIOWrapper. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
Chris Jerdonek added the comment: Thanks for your comments. FYI it's not directly Popen() which uses the locale encoding, but TextIOWrapper. Yes, I will note that to be more clear. Would you like me to add tests for UTF-16-LE, etc. (via a helper test method), or will UTF-16 alone suffice? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
STINNER Victor added the comment: Would you like me to add tests for UTF-16-LE, etc. (via a helper test method), or will UTF-16 alone suffice? You can use a loop insteadd of an helper function. I fail to see other encoding having the same issue, except the UTF-32 family (utf-32, utf-32-le, utf-32-be). I don't think that writing a test for the 6 codecs is required, you can use utf-16 and utf-32-be. So you test without BOM and with BOM, and utf-16 and utf-32. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
Chris Jerdonek added the comment: Updating patch with Victor's comments. -- Added file: http://bugs.python.org/file26733/issue-15595-2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
Chris Jerdonek added the comment: It looks like this also affects 3.2, but I will need to modify the test slightly because in 3.2, TextIOWrapper calls locale.getpreferredencoding() without any arguments. -- versions: +Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15595] subprocess.Popen(universal_newlines=True) does not work for certain locales
Chris Jerdonek added the comment: Here is a patch also suitable for applying to Python 3.2. -- Added file: http://bugs.python.org/file26734/issue-15595-3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15595 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
