[issue13886] readline-related test_builtin failure

[STINNER Victor]

STINNER Victor  added the comment:

Since nobody managed to fix this issue in 10 years and the test still fails if 
the readline module is loaded, I wrote GH-30631 to skip the test if the 
readline module is loaded.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +28834
pull_request: https://github.com/python/cpython/pull/30631

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[STINNER Victor]

STINNER Victor  added the comment:

Oh, the test_builtin.test_input_tty_non_ascii() fails just if test_readline is 
loaded previously:

$ ./python -E -m test -m test.test_builtin.PtyTests.test_input_tty_non_ascii -v 
test_readline test_builtin
== CPython 3.11.0a4+ (heads/main:7f4b69b9076, Jan 17 2022, 12:28:15) [GCC 
11.2.1 20211203 (Red Hat 11.2.1-7)]
== Linux-5.15.12-200.fc35.x86_64-x86_64-with-glibc2.34 little-endian
== cwd: /home/vstinner/python/main/build/test_python_49429æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 0.48 Run tests sequentially
0:00:00 load avg: 0.48 [1/2] test_readline

--
Ran 0 tests in 0.000s

OK
0:00:00 load avg: 0.48 [2/2] test_builtin -- test_readline ran no tests
test_input_tty_non_ascii (test.test_builtin.PtyTests) ... FAIL

==
FAIL: test_input_tty_non_ascii (test.test_builtin.PtyTests)
--
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2095, in 
test_input_tty_non_ascii
self.check_input_tty("prompté", b"quux\xe9", "utf-8")
^
  File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2086, in 
check_input_tty
self.assertEqual(input_result, expected)

AssertionError: 'quux' != 'quux\udce9'
- quux
+ quux\udce9
? +


--
Ran 1 test in 0.013s

FAILED (failures=1)
test test_builtin failed
test_builtin failed (1 failure)

== Tests result: FAILURE ==

1 test failed:
test_builtin

1 test run no tests:
test_readline

Total duration: 559 ms
Tests result: FAILURE


In just, just importing readline is enough to make the test fails:

$ git diff Lib/test/test_builtin.py
diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py
index 6dc4fa55502..20d3d33d9fb 100644
--- a/Lib/test/test_builtin.py
+++ b/Lib/test/test_builtin.py
@@ -1,3 +1,5 @@
+import readline
+
 # Python test set -- built-in functions
 
 import ast


$ ./python -E -m test -m test.test_builtin.PtyTests.test_input_tty_non_ascii -v 
test_builtin
(...)

==
FAIL: test_input_tty_non_ascii (test.test_builtin.PtyTests)
--
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2097, in 
test_input_tty_non_ascii
self.check_input_tty("prompté", b"quux\xe9", "utf-8")
^
  File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2088, in 
check_input_tty
self.assertEqual(input_result, expected)

AssertionError: 'quux' != 'quux\udce9'
- quux
+ quux\udce9
? +

(...)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[STINNER Victor]

STINNER Victor  added the comment:

On Fedora 35, I still reproduce the initial issue on the main branch of Python:

$ ./python -E -m test -v test_readline test_builtin
(...)

==
FAIL: test_input_tty_non_ascii (test.test_builtin.PtyTests)
--
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2095, in 
test_input_tty_non_ascii
self.check_input_tty("prompté", b"quux\xe9", "utf-8")
^
  File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2086, in 
check_input_tty
self.assertEqual(input_result, expected)

AssertionError: 'quux' != 'quux\udce9'
- quux
+ quux\udce9
? +


==
FAIL: test_input_tty_non_ascii_unicode_errors (test.test_builtin.PtyTests)
--
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2099, in 
test_input_tty_non_ascii_unicode_errors
self.check_input_tty("prompté", b"quux\xe9", "ascii")
^
  File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2086, in 
check_input_tty
self.assertEqual(input_result, expected)

AssertionError: 'quux' != 'quux\udce9'
- quux
+ quux\udce9
? +
(...)


Fedora 35 uses readline 8.1:

$ ldd $(./python -c 'import readline; print(readline.__file__)')
...
libreadline.so.8 => /lib64/libreadline.so.8 (0x7fd00e553000)
...
$ rpm -qf /lib64/libreadline.so.8
readline-8.1-3.fc35.x86_64

$ make pythoninfo|grep readline
readline._READLINE_LIBRARY_VERSION: 8.1
readline._READLINE_RUNTIME_VERSION: 0x801
readline._READLINE_VERSION: 0x801

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Irit Katriel]

Irit Katriel  added the comment:

See also Issue44887.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Irit Katriel]

Irit Katriel  added the comment:

On 3.11 I don't see a problem on windows, but on mac test_input_tty from 
test_builtin hangs when test_readline runs first, while test_builtin passes on 
its own.

--
nosy: +iritkatriel
versions: +Python 3.10, Python 3.11, Python 3.9 -Python 3.5, Python 3.6, Python 
3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Brett Cannon]

Change by Brett Cannon :


--
nosy:  -brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Xavier de Gaye]

Xavier de Gaye  added the comment:

All checks have passed on PR 7133, even on macOs, and the PR is ready to be 
reviewed :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Xavier de Gaye]

Xavier de Gaye  added the comment:

As stated in msg285810, issue 28180 removed the need for an Android workaround. 
After checking this point on the Android API 24 emulator with the current 
master branch, I have opened PR 7133 which is Martin's input-readline.v3.patch 
with the Android workaround removed. The patch is commited with Martin as its 
author.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Xavier de Gaye]

Change by Xavier de Gaye :


--
pull_requests: +6767

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

This old issue still is not fixed. Martin, Xavier, could you open a PR please?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

As well as other 5 bugs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[STINNER Victor]

STINNER Victor  added the comment:

bpo-31703 has been marked as a duplicate of this bug.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Xavier de Gaye]

Xavier de Gaye added the comment:

With input-readline.v3.patch, test_builtin runs with success on Android api 21.

With pep538_coerce_legacy_c_locale_v4.diff that implements PEP 538 in issue 
28180, and with input-readline.v3.patch modified to have 'readline_encoding = 
locale.getpreferredencoding()' even when 'is_android' is True, test_builtin 
runs with success. This means that no specific Android handling would be needed 
if PEP 538 were to be adopted.

The new input-readline.v4.patch is a slight improvement over the previous patch 
and sets readline_encoding to 'UTF-8' on Android when test_builtin is run with 
the environment variable LANG set to 'en_US.UTF-8' and in that case the test 
exercises all the code paths including those with the readline module.  This is 
because locale.getdefaultlocale() returns ('en_US', 'UTF-8') on Android in that 
case and because both getdefaultlocale() and readline scan the environment to 
check for the locale.  This new patch is only useful if tests on Android are 
expected to be run also with one of the locale environment variables set to 
UTF-8 (and PEP 538 is rejected).

I have left some comments on Rietveld.

--
Added file: http://bugs.python.org/file46340/input-readline.v4.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Martin Panter]

Martin Panter added the comment:

V3 of my patch skips the Readline tests in cases involving non-ASCII bytes when 
the locale seen by Readline would be ASCII. Readline may translate the 
non-ASCII bytes to escape sequences, and treat them as special Meta (Alt) key 
combinations. This behaviour depends on Readline configuration (“set 
convert-meta off” in /etc/inputrc in my case).

It also includes a potential workaround for Android, depending on the 
resolution of Issue 28997.

--
Added file: http://bugs.python.org/file46296/input-readline.v3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Xavier de Gaye]

Xavier de Gaye added the comment:

Thanks for waiting for a run of the patch on Android.
Indeed, test_input_tty_non_ascii fails with this patch on Android when LANG is 
not set and is ok when LANG=en_US.UTF-8. As you are suggesting, the test is ok 
when skipping the Readline half of the test. This may be done by adding 
'is_android' in the 'from test.support import ...' statement and updating the 
test with:

if readline and not is_android:
tests.append(True)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Martin Panter]

Martin Panter added the comment:

Hi Xavier, I was about to push input-readline.v2.patch, but I thought it might 
be good to check with you first if this causes problems with Android, based on 
my experience with Issue 28997. With the patch applied, 
test.test_builtin.PtyTests.test_input_tty_non_ascii() will call the input() 
function twice, with and without Readline enabled. It inputs non-ASCII data in 
into a pseudoterminal, and expects to see it in the function return value. 
Perhaps we may need to skip the Readline half of the test on Android.

--
nosy: +xdegaye

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Martin Panter]

Martin Panter added the comment:

input-readline.v2.patch sets stderr=DEVNULL so that the prompt does not come 
out in the test log.

A disadvantage of this is that if there is a failure, any error messages, stack 
trace, etc is also lost. To fix this properly, we would probably have to 
capture stderr using a generalized communicate() loop, like I have done at the 
bottom of 

 (see also Issue 1260171).

--
dependencies:  -Change input() to always prompt to stderr
Added file: http://bugs.python.org/file45771/input-readline.v2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Martin Panter]

Martin Panter added the comment:

Since people keep coming upon this bug, perhaps we should inhibit push my fix 
without fixing that other prompt bug (now a feature change I think). Probably 
have to capture stderr to avoid it coming out in the test output.

--
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Martin Panter]

Martin Panter added the comment:

Here is a possible patch which:

* Runs the three test_input_tty() tests in a fresh child Python interpreter, 
for control over whether Readline is loaded. A pseudoterminal is used for stdin 
and stdout, and stderr is left untouched to easily see error messages.
* Repeats each test with and without Readline enabled
* Only tests the error handling without Readline
* Fixes the non-error test to use proper UTF-8 input

With my patch applied, there are a couple of prompts mixed up in the test log 
output via stderr, due to Issue 1927. It is not a major problem, but perhaps we 
should work on fixing that bug first.

--
dependencies: +raw_input behavior incorrect if readline not enabled
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Martin Panter]

Changes by Martin Panter :


Added file: http://bugs.python.org/file41653/input-readline.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Ned Deily]

Changes by Ned Deily n...@acm.org:


--
nosy: +doko

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

This happens even when run test_builtin twice.

./python -E -m test -v test_builtin test_builtin

or

./python -E -m test -Fv test_builtin

--
nosy: +serhiy.storchaka

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Brett Cannon]

Brett Cannon added the comment:

Issue #18230 is another test_builtin failure related to tty tests.

--
nosy: +brett.cannon

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Nadeem Vawda]

Nadeem Vawda added the comment:

You're right; it breaks backspacing over multibyte characters. I should
have tested it more carefully before committing. I'll revert the changes.

--
resolution: fixed - 
stage: committed/rejected - needs patch
status: closed - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Roundup Robot]

Roundup Robot added the comment:

New changeset e6952acd5a55 by Nadeem Vawda in branch '3.2':
Back out fix for issue #13886; it introduced a new bug in interactive readline 
use.
http://hg.python.org/cpython/rev/e6952acd5a55

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Roundup Robot]

Roundup Robot added the comment:

New changeset 5c7e884b205a by Nadeem Vawda in branch '3.3':
Back out fix for issue #13886; it introduced a new bug in interactive readline 
use.
http://hg.python.org/cpython/rev/5c7e884b205a

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Roundup Robot]

Roundup Robot added the comment:

New changeset 8b8c6abda7e8 by Nadeem Vawda in branch 'default':
Back out fix for issue #13886; it introduced a new bug in interactive readline 
use.
http://hg.python.org/cpython/rev/8b8c6abda7e8

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Roundup Robot]

Roundup Robot added the comment:

New changeset 5bf91dfb1e34 by Nadeem Vawda in branch '2.7':
Back out fix for issue #13886; it introduced a new bug in interactive readline 
use.
http://hg.python.org/cpython/rev/5bf91dfb1e34

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Roundup Robot]

Roundup Robot added the comment:

New changeset cf0f450b3299 by Nadeem Vawda in branch '3.2':
Issue #13886: Fix input() to not strip out supposedly-invalid input bytes.
http://hg.python.org/cpython/rev/cf0f450b3299

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Roundup Robot]

Roundup Robot added the comment:

New changeset 5c8732049dd5 by Nadeem Vawda in branch '3.3':
Issue #13886: Fix input() to not strip out supposedly-invalid input bytes.
http://hg.python.org/cpython/rev/5c8732049dd5

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Roundup Robot]

Roundup Robot added the comment:

New changeset 9774721bfc32 by Nadeem Vawda in branch 'default':
Issue #13886: Fix input() to not strip out supposedly-invalid input bytes.
http://hg.python.org/cpython/rev/9774721bfc32

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Roundup Robot]

Roundup Robot added the comment:

New changeset 12223782031f by Nadeem Vawda in branch '2.7':
Issue #13886: Fix input() to not strip out supposedly-invalid input bytes.
http://hg.python.org/cpython/rev/12223782031f

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Nadeem Vawda]

Changes by Nadeem Vawda nadeem.va...@gmail.com:


--
assignee:  - nadeem.vawda
resolution:  - fixed
stage: patch review - committed/rejected
status: open - closed
versions: +Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Petri Lehtinen]

Petri Lehtinen added the comment:

-setlocale(LC_CTYPE, );
+setlocale(LC_CTYPE, C);

This looks dangerous to me. Are you sure readline's behavior doesn't change 
because of this?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Ezio Melotti]

Ezio Melotti added the comment:

This problem still exists on 3.4.
The attached patch seems to solve it.

--
nosy: +ezio.melotti
versions: +Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Petri Lehtinen]

Changes by Petri Lehtinen pe...@digip.org:


--
nosy: +haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Petri Lehtinen]

Petri Lehtinen pe...@digip.org added the comment:

I had exactly the same error on 3.3b1 when running the test suite with 
randomized order.

--
nosy: +petri.lehtinen

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Nadeem Vawda]

New submission from Nadeem Vawda nadeem.va...@gmail.com:

I've recently come across a strange failure in the tests for the input()
built-in function:

$ ./python -E -m test -v test_readline test_builtin

[... snip ...]

==
FAIL: test_input_tty_non_ascii (test.test_builtin.BuiltinTest)
--
Traceback (most recent call last):
  File /home/nadeem/src/cpython/def/Lib/test/test_builtin.py, line 1079, 
in test_input_tty_non_ascii
self.check_input_tty(prompté, bquux\xe9, utf-8)
  File /home/nadeem/src/cpython/def/Lib/test/test_builtin.py, line 1070, 
in check_input_tty
self.assertEqual(input_result, expected)
AssertionError: 'quux' != 'quux\udce9'
- quux
+ quux\udce9
? +


==
FAIL: test_input_tty_non_ascii_unicode_errors 
(test.test_builtin.BuiltinTest)
--
Traceback (most recent call last):
  File /home/nadeem/src/cpython/def/Lib/test/test_builtin.py, line 1083, 
in test_input_tty_non_ascii_unicode_errors
self.check_input_tty(prompté, bquux\xe9, ascii)
  File /home/nadeem/src/cpython/def/Lib/test/test_builtin.py, line 1070, 
in check_input_tty
self.assertEqual(input_result, expected)
AssertionError: 'quux' != 'quux\udce9'
- quux
+ quux\udce9
? +

The failure only manifests itself if the readline module is loaded before
test_builtin runs (hence the presence of test_readline above). It will
not occur if regrtest is run with either of the -j or -W flags (which is
why it hasn't been seen on the buildbots).

The problem seems to be that readline assumes that its input should use
the locale encoding, and silently strips out any undecodable chars. This
breaks the tests mentioned above, since they set up sys.stdin to use the
surrogateescape error handler, expecting invalid characters to be escaped
rather than discarded.

This problem doesn't crop up if readline is *not* loaded, because in that
case PyOS_Readline() falls back to a stdio-based implementation
(PyOS_StdioReadline()) that preserves invalid characters, allowing them
to be handled properly by sys.stdin's encoding and error handler.

I have been able to fix the test failures with the attached patch, which
stops readline from eating invalid characters, making it consistent with
the stdio-based fallback. Can someone with more knowledge of readline
and/or locale issues advise whether the change is a good idea?

--
components: Extension Modules
files: rl-locale.diff
keywords: patch
messages: 152080
nosy: nadeem.vawda
priority: normal
severity: normal
stage: patch review
status: open
title: readline-related test_builtin failure
type: behavior
versions: Python 3.2, Python 3.3
Added file: http://bugs.python.org/file24337/rl-locale.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Nadeem Vawda]

Nadeem Vawda nadeem.va...@gmail.com added the comment:

Here's another patch that ensures the test always exercises the GNU
readline code path (rather than the stdio fallback). This will cause the
failure to occur when running just test_builtin (no need to also run
test_readline before it).

Ideally we'd want to test both code paths, but I'm not sure how to
accomplish that reliably, short of running the test in a subprocess.

--
Added file: http://bugs.python.org/file24338/rl-test.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13886
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com