[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset c2cf12857187aa147c268651f10acd6da2c9cb74 by Serhiy Storchaka in 
branch 'master':
bpo-8256: Fixed possible failing or crashing input() (#517)
https://github.com/python/cpython/commit/c2cf12857187aa147c268651f10acd6da2c9cb74


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset a16894ebf8823f0e09036aacde9288c00e8d9058 by Serhiy Storchaka in 
branch '3.5':
[3.5] bpo-8256: Fixed possible failing or crashing input() (#642)
https://github.com/python/cpython/commit/a16894ebf8823f0e09036aacde9288c00e8d9058


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset aac875fa2f03cab61ceeaa2621c4c5534c7bcfc2 by Serhiy Storchaka in 
branch '3.6':
[3.6] bpo-8256: Fixed possible failing or crashing input() (#641)
https://github.com/python/cpython/commit/aac875fa2f03cab61ceeaa2621c4c5534c7bcfc2


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Larry Hastings]

Changes by Larry Hastings :


--
pull_requests: +577

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +531

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +530

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +529

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +427

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
versions: +Python 3.7 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Erik Bray]

Erik Bray added the comment:

I just recently discovered this myself.  In the process of debugging the issue 
I also noticed the same bug that is now fixed via Issue 24402.

While I agree that Issue 24402 mostly mitigates the issue I think this patch is 
still worthwhile, as the current behavior still leads to cryptic, hard to debug 
errors.  For example (although this is not great code, bear with me...) one 
could write a stdout wrapper like:

>>> class WrappedStream:
... encoding = 'utf8'
... errors = None
... def __getattr__(self, attr):
... return getattr(sys.__stdout__, attr)
... 
>>> sys.stdout = WrappedStream()
>>> sys.stdout.fileno()
1
>>> sys.stdout.isatty()
True
>>> sys.stdout.errors
>>> input('Prompt: ')
Traceback (most recent call last):
  File "", line 1, in 
TypeError: bad argument type for built-in operation

This still goes down the path for ttys, but because the 'errors' attribute does 
not defer to the underlying stream it still leads to a hard to debug exception. 
 To be clear, I think the above code *should* break, just not as cryptically.

--
nosy: +erik.bray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Erik Bray]

Erik Bray added the comment:

> I think the above code *should* break

Actually, I see now that Serhiy's patch would allow this example to just pass 
through to the non-interactive fallback.  So I take it back that my example 
should break--I think using the fallback would also be fine.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Here is a patch.

--
stage: needs patch -> patch review
Added file: http://bugs.python.org/file40704/input_fallback.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I would fallback to PyFile_WriteObject(prompt, fout, Py_PRINT_RAW) if the 
stdout has no the encoding attribute or it is not a string.

--
nosy: +serhiy.storchaka
versions: +Python 3.4, Python 3.5, Python 3.6 -Python 3.2, Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Martin Panter]

Martin Panter added the comment:

Serhiy, your patch looks like a worthwhile improvement because it adds proper 
error checking and handling. However I suspect this original bug is actually a 
side effect of Issue 24402. The code in question shouldn’t even be running, 
because sys.stdout is not the original output file descriptor, and is not a 
terminal.

--
dependencies: +input() uses sys.__stdout__ instead of sys.stdout for prompt
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[aliles]

aliles added the comment:

Upload new patch that uses encoding and errors from stderr if stdout values are 
invalid unicode. Includes unit test in test_builtin.py.

With this patch I am no longer able to replicate this issue.

--
Added file: http://bugs.python.org/file27006/p1345978092.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8256
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[aliles]

aliles added the comment:

Replicated this issue on Python 3.3b2. The cause is the 'encoding' and 'errors' 
attributes on io.StringIO() being None. Doctest replaces sys.stdout with a 
StringIO subclass. The exception raised is still a TypeError.

At this point I'm unsure what the fix should be:

1. Should the exception raised be more descriptive of the problem?
2. Should io.StringIO have real values for encoding and errors?
3. Should Doctest's StingIO class provide encoding and errors?

--
nosy: +aliles

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8256
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Antoine Pitrou]

Antoine Pitrou added the comment:

 I suggest that since PyOS_Readline() write the prompt to stderr, the
 conversion uses the encoding of stderr.

Agreed with Amaury.

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8256
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[STINNER Victor]

STINNER Victor victor.stin...@haypocalc.com added the comment:

A patch similar to input_stdout_encoding.patch has been applied to 3.2 and 3.3 
for the issue #6697: see changeset 846866aa0eb6.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8256
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[STINNER Victor]

STINNER Victor victor.stin...@haypocalc.com added the comment:

input_stdout_none_encoding.patch uses UTF-8 if sys.stdout.encoding is None.

--
Added file: http://bugs.python.org/file23608/input_stdout_none_encoding.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8256
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Florent Xicluna]

Florent Xicluna florent.xicl...@gmail.com added the comment:

Confirmed in 3.3.
The patch does not apply cleanly on trunk.

--
stage: test needed - needs patch
versions: +Python 3.3 -Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8256
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc amaur...@gmail.com added the comment:

since the prompt is written to stderr, why is sys.stdout.encoding used instead 
of sys.stderr.encoding?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8256
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[STINNER Victor]

STINNER Victor victor.stin...@haypocalc.com added the comment:

amaury since the prompt is written to stderr, why is sys.stdout.encoding
amaury used instead of sys.stderr.encoding?

input() calls PyOS_Readline() but PyOS_Readline() has multiple 
implementations:
 - PyOS_StdioReadline() if sys_stdin or sys_stdout is not a TTY
 - or PyOS_ReadlineFunctionPointer callback:
   - vms__StdioReadline() (VMS only)
   - PyOS_StdioReadline()
   - call_readline() when readline module is loaded

call_readline() calls rl_callback_handler_install() with the prompt which 
writes the prompt to *stdout* (try ./python 2/dev/null).

I don't think that it really matters that the prompt is written to stderr with 
stdout encoding, because both outputs always use the same encoding.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8256
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8256] input() doesn't catch _PyUnicode_AsString() exception; io.StringIO().encoding is None

[STINNER Victor]

STINNER Victor victor.stin...@haypocalc.com added the comment:

Here is a patch catching the _PyUnicode_AsString() error.

input() uses sys.stdout.encoding to encode the prompt to a byte string, but 
PyOS_StdioReadline() writes the prompt to stderr (it should use sys_stdout).

I don't know which encoding should be used if sys.stdout.encoding is None (eg. 
if sys.stdout is a StringIO() object).

StringIO() of _io module has no encoding because it stores unicode characters, 
not bytes. StringIO() of _pyio module is based on BytesIO() and use utf8 
encoding, but the reference implementation is now _io.

--
Added file: http://bugs.python.org/file17324/input_stdout_encoding.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8256
___Index: Python/bltinmodule.c
===
--- Python/bltinmodule.c(révision 81157)
+++ Python/bltinmodule.c(copie de travail)
@@ -1618,6 +1618,7 @@
 if (promptarg != NULL) {
 PyObject *stringpo;
 PyObject *stdout_encoding;
+char *stdout_encoding_str;
 stdout_encoding = PyObject_GetAttrString(fout,
  encoding);
 if (stdout_encoding == NULL) {
@@ -1630,8 +1631,17 @@
 Py_DECREF(stdout_encoding);
 return NULL;
 }
+if (stdout_encoding != Py_None)
+stdout_encoding_str = _PyUnicode_AsString(stdout_encoding);
+else
+stdout_encoding_str = utf-8;
+if (stdout_encoding_str == NULL) {
+Py_DECREF(stdin_encoding);
+Py_DECREF(stdout_encoding);
+return NULL;
+}
 po = PyUnicode_AsEncodedString(stringpo,
-_PyUnicode_AsString(stdout_encoding), NULL);
+   stdout_encoding_str, NULL);
 Py_DECREF(stdout_encoding);
 Py_DECREF(stringpo);
 if (po == NULL) {
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com