[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

STINNER Victor  added the comment:

> 0:00:22 [ 16/419/2] test_unicode crashed (Exit code -11)

Please open a new issue to track this bug.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

Victor, looks good to me: 0:00:26 [ 23/419/3] test_utf8_mode passed.

I don't know wether it is related, but test_unicode crash dumps here:
0:00:22 [ 16/419/2] test_unicode crashed (Exit code -11)
Fatal Python error: Segmentation fault

Current thread 0x0001 (most recent call first):
  File "/var/osipovmi/cpython/Lib/test/test_unicode.py", line 2465 in 
PyUnicode_FromFormat
  File "/var/osipovmi/cpython/Lib/test/test_unicode.py", line 2468 in 
check_format
  File "/var/osipovmi/cpython/Lib/test/test_unicode.py", line 2472 in 
test_from_format
  File "/var/osipovmi/cpython/Lib/unittest/case.py", line 610 in run
  File "/var/osipovmi/cpython/Lib/unittest/case.py", line 658 in __call__
  File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 122 in run
  File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 122 in run
  File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 122 in run
  File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/var/osipovmi/cpython/Lib/test/support/testresult.py", line 162 in run
  File "/var/osipovmi/cpython/Lib/test/support/__init__.py", line 1928 in 
_run_suite
  File "/var/osipovmi/cpython/Lib/test/support/__init__.py", line 2022 in 
run_unittest
  File "/var/osipovmi/cpython/Lib/test/libregrtest/runtest.py", line 175 in 
test_runner
  File "/var/osipovmi/cpython/Lib/test/libregrtest/runtest.py", line 179 in 
runtest_inner
  File "/var/osipovmi/cpython/Lib/test/libregrtest/runtest.py", line 134 in 
runtest
  File "/var/osipovmi/cpython/Lib/test/libregrtest/runtest_mp.py", line 68 in 
run_tests_worker
  File "/var/osipovmi/cpython/Lib/test/libregrtest/main.py", line 587 in _main
  File "/var/osipovmi/cpython/Lib/test/libregrtest/main.py", line 571 in main
  File "/var/osipovmi/cpython/Lib/test/libregrtest/main.py", line 627 in main
  File "/var/osipovmi/cpython/Lib/test/regrtest.py", line 46 in _main
  File "/var/osipovmi/cpython/Lib/test/regrtest.py", line 50 in 
  File "/var/osipovmi/cpython/Lib/runpy.py", line 85 in _run_code
  File "/var/osipovmi/cpython/Lib/runpy.py", line 192 in _run_module_as_main

Is that related to your PEP?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 7d35f79012db89ce9a152a77ac6809eb9c34a35d by Victor Stinner in 
branch '3.7':
bpo-34403: Always implement _Py_GetForceASCII() (GH-10235)
https://github.com/python/cpython/commit/7d35f79012db89ce9a152a77ac6809eb9c34a35d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9547

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 21220bbe65108f5a763ead24a6b572f80d84c9e2 by Victor Stinner in 
branch '3.7':
bpo-34403: Fix initfsencoding() for ASCII (GH-10233)
https://github.com/python/cpython/commit/21220bbe65108f5a763ead24a6b572f80d84c9e2


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

STINNER Victor  added the comment:

Michael Osipov: Oops, my commit b2457efc78b74a1d6d1b77d11a939e886b8a4e2c broke 
the filesystem encoding on HP-UX. It should be fixed by my commit 
905f1ace5f7424e314ca7bed997868a2a3044839.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 905f1ace5f7424e314ca7bed997868a2a3044839 by Victor Stinner in 
branch 'master':
bpo-34523: Fix config_init_fs_encoding() for ASCII (GH-10232)
https://github.com/python/cpython/commit/905f1ace5f7424e314ca7bed997868a2a3044839


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9546

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9545

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

STINNER Victor  added the comment:

> I am the AIX(tools) Michael, Michael O is the HP-UX Michael :p

Oh, I didn't notice that you two have the same first name :-)

> So I was not the one asking. IMHO - as the PEP was new, if I understood
correctly, in 3.7 - would be "nice" to see it back ported. However, like you - 
my goal is to get the tests passing on master, and
worry about backport later.

*My position didn't change since my last comment, same position for AIX and 
HP-UX: msg324289. I also updated my website to write down this policy:
https://pythondev.readthedocs.io/cpython.html#i-want-cpython-to-support-my-platform

By the way, please don't comment issues that are closed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Felt]

Michael Felt  added the comment:

On 28/08/2018 23:14, STINNER Victor wrote:
> STINNER Victor  added the comment:
>
>> Can we backport this to 3.7 at least?
I am the AIX(tools) Michael, Michael O is the HP-UX Michael :p

So I was not the one asking. IMHO - as the PEP was new, if I understood
correctly, in 3.7 - would be "nice" to see it back ported.

However, like you - my goal is to get the tests passing on master, and
worry about backport later.
> My policy is to focus on the master branch to support a new platform. Then 
> add a buildbot and find a core developer to maintain this platform. See the 
> PEP 11 for details.
>
> I would prefer to see a full test suite passing before discussing which 
> changes should or should not be backported.
>
> I would also prefer to first see a more general discussion about who is going 
> to support HP-UX.
>
> IMHO HP-UX is not officially supported today. My list of supported platforms:
> https://pythondev.readthedocs.io/cpython.html#supported-platforms
>
> See the test_utf8_mode now pass on HP-UX, I close the issue. Please open more 
> specific issues for other failures. You might open a meta issue to track all 
> HP-UX issues.
>
> --
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Felt]

Michael Felt  added the comment:

On 28/08/2018 20:43, Michael Felt wrote:
> Attached is the output with LC_ALL=C in the prefix. If you were hoping
> for "dangling processes - your hopes are affirmed.

Previous mail ended with:

== Tests result: FAILURE ==

375 tests OK.

13 tests failed:
    test__xxsubinterpreters test_asyncio test_concurrent_futures
    test_ctypes test_distutils test_embed test_httpservers
    test_multiprocessing_fork test_os test_pkg test_socket
    test_subprocess test_time

30 tests skipped:
    test_curses test_dbm_gnu test_devpoll test_epoll test_gdb
    test_idle test_kqueue test_lzma test_msilib test_ossaudiodev
    test_readline test_smtpnet test_socketserver test_spwd test_sqlite
    test_startfile test_tcl test_timeout test_tix test_tk
    test_ttk_guionly test_ttk_textonly test_turtle test_urllib2net
    test_urllibnet test_winconsoleio test_winreg test_winsound
    test_xmlrpc_net test_zipfile64

Total duration: 14 min 53 sec
Tests result: FAILURE
root@x066:[/data/prj/python/git/python3-3.8]

Without LC_ALL=C summary is (different):

== Tests result: FAILURE ==

376 tests OK.

10 tests failed:
    test__xxsubinterpreters test_asyncio test_ctypes test_distutils
    test_embed test_httpservers test_multiprocessing_forkserver
    test_os test_socket test_time

32 tests skipped:
    test_curses test_dbm_gnu test_devpoll test_epoll test_gdb
    test_idle test_kqueue test_lzma test_msilib test_ossaudiodev
    test_readline test_smtpnet test_socketserver test_spwd test_sqlite
    test_startfile test_tcl test_timeout test_tix test_tk
    test_ttk_guionly test_ttk_textonly test_turtle test_unicode_file
    test_unicode_file_functions test_urllib2net test_urllibnet
    test_winconsoleio test_winreg test_winsound test_xmlrpc_net
    test_zipfile64

Total duration: 11 min 1 sec

And, rather than dangling processes, I see BrokenBarrierErrors

FYI

--
Added file: https://bugs.python.org/file47771/py3.8-default.text

___
Python tracker 

___Script command is started on Tue Aug 28 18:34:49 UTC 2018.
root@x066:[/data/prj/python/git/python3-3.8]script 
../py3.8-default.textm   
maketopas -i 5LC _ALL=C ./python -m 
test -j0 -rC _ALL=C ./python -m test -j0 -r 
 _ALL=C ./python -m test -j0 -r 
ALL=C ./python -m test -j0 -r 
LL=C ./python -m test -j0 -r 
L=C ./python -m test -j0 -r 
=C ./python -m test -j0 -r 
C ./python -m test -j0 -r  
./python -m test -j0 -r ./python -m test -j0 -r 

== CPython 3.8.0a0 (heads/master:d500e5307a, Aug 28 2018, 18:09:17) [C]
== AIX-1-00C291F54C00-powerpc-32bit big-endian
== cwd: /data/prj/python/git/python3-3.8/build/test_python_16777460
== CPU count: 8
== encodings: locale=ISO8859-1, FS=iso8859-1
Using random seed 1054694
Run tests in parallel using 10 child processes
0:00:01 [  1/418] test_dictcomps passed
0:00:01 [  2/418] test_class passed
0:00:01 [  3/418] test_global passed
0:00:01 [  4/418] test_types passed
0:00:02 [  5/418] test_unpack_ex passed
0:00:02 [  6/418] test_sysconfig passed
0:00:02 [  7/418] test_functools passed
0:00:02 [  8/418] test_frozen passed
0:00:02 [  9/418] test_bigmem passed
0:00:03 [ 10/418] test_binascii passed
0:00:03 [ 11/418] test_profile passed
0:00:03 [ 12/418] test_tabnanny passed
0:00:03 [ 13/418] test_pty passed
0:00:03 [ 14/418] test_fstring passed
0:00:03 [ 15/418] test_codecencodings_kr passed
0:00:04 [ 16/418] test_strptime passed
0:00:04 [ 17/418] test_sunau passed
0:00:04 [ 18/418] test_unicode_identifiers passed
0:00:04 [ 19/418] test_longexp passed
0:00:05 [ 20/418] test_fileio passed
0:00:05 [ 21/418] test_repl passed
0:00:06 [ 22/418] test_int passed
0:00:06 [ 23/418] test_posixpath passed
running: test_compile (35 sec 776 ms), test_lib2to3 (36 sec 396 ms), test_queue 
(35 sec 862 ms), test_multiprocessing_spawn (37 sec 717 ms), test_faulthandler 
(35 sec 494 ms), test_compileall (33 sec 460 ms), test_random (37 sec 839 ms), 
test_fork1 (38 sec 549 ms)
0:00:40 [ 24/418] test_xmlrpc passed -- running: test_compile (36 sec 924 ms), 
test_lib2to3 (37 sec 167 ms), test_queue (36 sec 613 ms), 
test_multiprocessing_spawn (38 sec 468 ms), test_faulthandler (36 sec 244 ms), 
test_compileall (34 sec 211 ms), test_random (38 sec 590 ms), test_fork1 (39 
sec 296 ms)
0:00:40 [ 25/418] test_ast passed -- running: test_compile (37 sec 6 ms), 
test_lib2to3 (37 sec 249 ms), test_queue (36 sec 695 ms), 
test_multiprocessing_spawn (38 sec 550 ms), test_faulthandler (36 sec 327 ms), 
test_compileall (34 sec 293 ms), test_random (38 sec 672 

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Felt]

Michael Felt  added the comment:

On 28/08/2018 13:20, STINNER Victor wrote:
> I updated my PR 8969 to implement properly my idea. With this PR, on HP-UX 
> with C or POSIX locale, Python now uses ASCII for its "filesystem encoding": 
> sys.getfilesystemencoding() returns "ascii".
>
> Michael: can you please try my updated PR 8969?
>
> * apply the updated change
> * recompile Python
> * run the test suite using: LC_ALL=C ./python -m test -j0 -r
>
> You may also test with the current locale: ./python -m test -j0 -r

Seems to work well as far as AIX and test_utf8_mode (as you had already
merged I pulled master and built that anew).

Attached is the output with LC_ALL=C in the prefix. If you were hoping
for "dangling processes - your hopes are affirmed.

Perhaps also noteworthy:

root@x066:[/data/prj/python/git/python3-3.8]set | grep LC
LC__FASTMSG=true
MAILCHECK=600
root@x066:[/data/prj/python/git/python3-3.8]set | grep LANG
LANG=en_US

--
Added file: https://bugs.python.org/file47770/py3.8-LC-C.text

___
Python tracker 

___Script command is started on Tue Aug 28 18:19:16 UTC 2018.
root@x066:[/data/prj/python/git/python3-3.8]script 
../py3.8.textrm ../py3.8.textls 
../py3.8.textls ../py3.8.text.rm 
../py3.8.text.make make 
distclean; buildaix --without-computed-gotos 
--with-openssl=/opt/aixto>cd
 *3.8 
py  
   LC _ALL=C ./python -m test -j0 -r
== CPython 3.8.0a0 (heads/master:d500e5307a, Aug 28 2018, 18:09:17) [C]
== AIX-1-00C291F54C00-powerpc-32bit big-endian
== cwd: /data/prj/python/git/python3-3.8/build/test_python_16777456
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Using random seed 2001895
Run tests in parallel using 10 child processes
0:00:02 [  1/418] test_sqlite skipped
test_sqlite skipped -- No module named '_sqlite3'
0:00:02 [  2/418] test_startfile skipped
test_startfile skipped -- object  has no attribute 'startfile'
0:00:02 [  3/418] test_mailcap passed
0:00:02 [  4/418] test_peepholer passed
0:00:02 [  5/418] test_sort passed
0:00:02 [  6/418] test_symbol passed
0:00:02 [  7/418] test_imghdr passed
0:00:02 [  8/418] test_string_literals passed
0:00:02 [  9/418] test_functools passed
0:00:03 [ 10/418] test_crashers passed
0:00:03 [ 11/418] test_unicode_identifiers passed
0:00:03 [ 12/418] test_metaclass passed
0:00:03 [ 13/418] test_normalization passed
0:00:03 [ 14/418] test_turtle skipped
test_turtle skipped -- No module named '_tkinter'
0:00:04 [ 15/418] test_codecmaps_kr passed
0:00:04 [ 16/418] test_pprint passed
0:00:04 [ 17/418] test_bdb passed
0:00:05 [ 18/418] test_pyexpat passed
0:00:05 [ 19/418] test_sched passed
0:00:05 [ 20/418] test_dict passed
0:00:05 [ 21/418] test_setcomps passed
0:00:05 [ 22/418] test_gdb skipped
test_gdb skipped -- Couldn't find gdb on the path
0:00:06 [ 23/418] test_ttk_textonly skipped
test_ttk_textonly skipped -- No module named '_tkinter'
0:00:06 [ 24/418] test_telnetlib passed
0:00:06 [ 25/418] test_plistlib passed
0:00:06 [ 26/418] test_urllib2 passed
0:00:07 [ 27/418] test_popen passed
0:00:07 [ 28/418] test_traceback passed
0:00:08 [ 29/418] test_pipes passed
0:00:08 [ 30/418] test_bisect passed
0:00:08 [ 31/418] test_robotparser passed
0:00:08 [ 32/418] test_statistics passed
0:00:08 [ 33/418] test_dataclasses passed
0:00:09 [ 34/418] test_linecache passed
0:00:09 [ 35/418] test_abstract_numbers passed
0:00:09 [ 36/418] test_profile passed
0:00:09 [ 37/418] test_pdb passed
0:00:09 [ 38/418] test__locale passed
0:00:10 [ 39/418] test_socketserver skipped (resource denied)
test_socketserver skipped -- Use of the 'network' resource not enabled
0:00:10 [ 40/418] test_nntplib passed
0:00:10 [ 41/418] test_xmlrpc_net skipped (resource denied)
test_xmlrpc_net skipped -- Use of the 'network' resource not enabled
0:00:10 [ 42/418] test_script_helper passed
0:00:11 [ 43/418] test_dbm_dumb passed
0:00:11 [ 44/418] test_epoll skipped
test_epoll skipped -- test works only on Linux 2.6
0:00:11 [ 45/418] test_genericpath passed
0:00:12 [ 46/418/1] test_time failed
test test_time failed -- multiple errors occurred; run in verbose mode for 
details
0:00:12 [ 47/418/1] test_ordered_dict passed
0:00:13 [ 48/418/1] test_wait4 passed
0:00:13 [ 49/418/1] test_stat passed
0:00:14 [ 50/418/1] test_file passed
0:00:14 [ 51/418/1] test_getopt passed
0:00:14 [ 52/418/1] test_capi passed
0:00:15 [ 53/418/1] test_code passed
0:00:15 [ 54/418/1] test_cprofile passed
0:00:15 [ 55/418/1] test_codecencodings_jp passed
0:00:16 [ 56/418/1] test_generator_stop passed
0:00:16 [ 

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

STINNER Victor  added the comment:

> Please close, issue fixed. Thank you very much.

You're welcome ;-)

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

Please close, issue fixed. Thank you very much.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

STINNER Victor  added the comment:

> Can we backport this to 3.7 at least?

My policy is to focus on the master branch to support a new platform. Then add 
a buildbot and find a core developer to maintain this platform. See the PEP 11 
for details.

I would prefer to see a full test suite passing before discussing which changes 
should or should not be backported.

I would also prefer to first see a more general discussion about who is going 
to support HP-UX.

IMHO HP-UX is not officially supported today. My list of supported platforms:
https://pythondev.readthedocs.io/cpython.html#supported-platforms

See the test_utf8_mode now pass on HP-UX, I close the issue. Please open more 
specific issues for other failures. You might open a meta issue to track all 
HP-UX issues.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

Can we backport this to 3.7 at least?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

STINNER Victor  added the comment:


New changeset d500e5307aec9c5d535f66d567fadb9c587a9a36 by Victor Stinner in 
branch 'master':
bpo-34403: On HP-UX, force ASCII for C locale (GH-8969)
https://github.com/python/cpython/commit/d500e5307aec9c5d535f66d567fadb9c587a9a36


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

Victor,

this looks good to me:
> osipovmi@blnn724x:/var/osipovmi/cpython []
> $ git fetch vstinner
> remote: Counting objects: 65, done.
> remote: Compressing objects: 100% (18/18), done.
> remote: Total 65 (delta 41), reused 43 (delta 37), pack-reused 10
> Unpacking objects: 100% (65/65), done.
> From https://github.com/vstinner/cpython
>  + 6171da5569...559de620d7 hpux_force_ascii -> vstinner/hpux_force_ascii  
> (forced update)
>  * [new branch]posix_locale37   -> vstinner/posix_locale37
> osipovmi@blnn724x:/var/osipovmi/cpython []
> $ git reset --hard  vstinner/hpux_force_ascii
> HEAD is now at 559de620d7 bpo-34403: On HP-UX, force ASCII for C locale
> $ gmake -j 8
> ...
> osipovmi@blnn724x:/var/osipovmi/cpython []
> $ LC_ALL=C ./python -m test -j0 -r -uall,-network
> ...
> == Tests result: FAILURE ==
> 
> 357 tests OK.
> 
> 35 tests failed:
> test_asyncio test_asyncore test_bytes test_c_locale_coercion
> test_code test_concurrent_futures test_ctypes test_devpoll
> test_distutils test_embed test_faulthandler test_fileio test_gdb
> test_httpservers test_importlib test_io test_mmap
> test_multiprocessing_fork test_multiprocessing_forkserver
> test_multiprocessing_main_handling test_multiprocessing_spawn
> test_normalization test_os test_pkg test_posix test_pty test_re
> test_signal test_socket test_subprocess test_support
> test_threading test_time test_unicode test_zlib
> 
> 26 tests skipped:
> test_curses test_dbm_gnu test_epoll test_idle test_kqueue
> test_lzma test_msilib test_ossaudiodev test_smtpnet
> test_socketserver test_spwd test_startfile test_tcl test_timeout
> test_tix test_tk test_ttk_guionly test_ttk_textonly test_turtle
> test_urllib2net test_urllibnet test_winconsoleio test_winreg
> test_winsound test_xmlrpc_net test_zipfile64
> osipovmi@blnn724x:/var/osipovmi/cpython []
> $ ./python -m test -j0 -r -uall,-network
> ...
> == Tests result: FAILURE ==
> 
> 357 tests OK.
> 
> 35 tests failed:
> test_asyncio test_asyncore test_bytes test_c_locale_coercion
> test_code test_concurrent_futures test_ctypes test_devpoll
> test_distutils test_embed test_faulthandler test_fileio test_gdb
> test_httpservers test_importlib test_io test_mmap
> test_multiprocessing_fork test_multiprocessing_forkserver
> test_multiprocessing_main_handling test_multiprocessing_spawn
> test_normalization test_os test_posix test_pty test_re
> test_readline test_signal test_socket test_subprocess test_support
> test_threading test_time test_unicode test_zlib
> 
> 26 tests skipped:
> test_curses test_dbm_gnu test_epoll test_idle test_kqueue
> test_lzma test_msilib test_ossaudiodev test_smtpnet
> test_socketserver test_spwd test_startfile test_tcl test_timeout
> test_tix test_tk test_ttk_guionly test_ttk_textonly test_turtle
> test_urllib2net test_urllibnet test_winconsoleio test_winreg
> test_winsound test_xmlrpc_net test_zipfile64
> 
> Total duration: 14 min 23 sec
> Tests result: FAILURE

The test_utf8_mode passes. Some other tests likely fail due to this Roman8 
stuff: test_re and friends. I am analyzing the failures step by step and have 
already a few fixes around. Waiting for other PRs to be merged first.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Felt]

Michael Felt  added the comment:

No time to compile for a couple of days. Stress from others wins instead. 

Maybe on Friday. 

Sent from my iPhone

> On 28 Aug 2018, at 13:20, STINNER Victor  wrote:
> 
> 
> STINNER Victor  added the comment:
> 
>> -check('utf8=0', [arg.decode(encoding, 'surrogateescape')],
>> +check('utf8=0', [arg.decode('ascii', 'surrogateescape')],
>>   LC_ALL='C')
>> (...)
>> == Tests result: SUCCESS ==
> 
> Good, it works.
> 
> I updated my PR 8969 to implement properly my idea. With this PR, on HP-UX 
> with C or POSIX locale, Python now uses ASCII for its "filesystem encoding": 
> sys.getfilesystemencoding() returns "ascii".
> 
> Michael: can you please try my updated PR 8969?
> 
> * apply the updated change
> * recompile Python
> * run the test suite using: LC_ALL=C ./python -m test -j0 -r
> 
> You may also test with the current locale: ./python -m test -j0 -r
> 
> If everything is good on your side, I will merge my PR.
> 
> --
> 
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

STINNER Victor  added the comment:

> -check('utf8=0', [arg.decode(encoding, 'surrogateescape')],
> +check('utf8=0', [arg.decode('ascii', 'surrogateescape')],
>LC_ALL='C')
>  (...)
> == Tests result: SUCCESS ==

Good, it works.

I updated my PR 8969 to implement properly my idea. With this PR, on HP-UX with 
C or POSIX locale, Python now uses ASCII for its "filesystem encoding": 
sys.getfilesystemencoding() returns "ascii".

Michael: can you please try my updated PR 8969?

* apply the updated change
* recompile Python
* run the test suite using: LC_ALL=C ./python -m test -j0 -r

You may also test with the current locale: ./python -m test -j0 -r

If everything is good on your side, I will merge my PR.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

Here is the output to your questions:
> osipovmi@blnn724x:/var/osipovmi/cpython []
> $ git checkout hpux_force_ascii
> Branch 'hpux_force_ascii' set up to track remote branch 'hpux_force_ascii' 
> from 'vstinner'.
> Switched to a new branch 'hpux_force_ascii'
> osipovmi@blnn724x:/var/osipovmi/cpython []
> $ git cherry-pick 217af1d38db3e1e875180c6fa160f0fc80e46003
> [hpux_force_ascii 7ce2927185] bpo-34403, bpo-34207: Fix 
> test_utf8_mode.test_cmd_line()
>  Author: Victor Stinner 
>  Date: Tue Aug 28 09:35:25 2018 +0200
>  1 file changed, 20 insertions(+), 11 deletions(-)
> osipovmi@blnn724x:/var/osipovmi/cpython []
> $ export CC=/opt/aCC/bin/cc ; \
>  export CXX=/opt/aCC/bin/aCC ; \
>  export LDFLAGS=-L/usr/local/lib/hpux32 ; \
>  export UNIX_STD=1998 ; \
> ./configure --prefix=/var/osipovmi/python37-testing --without-gcc 
> --with-system-expat --with-pydebug --with-openssl=/opt/openssl
> ...
> osipovmi@blnn724x:/var/osipovmi/cpython []
> $ gmake -j 8
> ...
> osipovmi@blnn724x:/var/osipovmi/cpython []
> $ ./python -m test test_utf8_mode
> Run tests sequentially
> 0:00:00 [1/1] test_utf8_mode
> test test_utf8_mode failed -- Traceback (most recent call last):
>   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 235, in 
> test_cmd_line
> LC_ALL='C')
>   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 214, in check
> self.assertEqual(args, ascii(expected), out)
> AssertionError: "['h\\udca7\\udce9']" != "['h\\xcf\\xd5']"
> - ['h\udca7\udce9']
> + ['h\xcf\xd5']
>  : roman8:['h\udca7\udce9']
> 
> test_utf8_mode failed
> 
> == Tests result: FAILURE ==
> 
> 1 test failed:
> test_utf8_mode
> 
> Total duration: 3 sec 58 ms
> Tests result: FAILURE
> osipovmi@blnn724x:/var/osipovmi/cpython []
> $ git diff
> diff --git a/Lib/test/test_utf8_mode.py b/Lib/test/test_utf8_mode.py
> index 5af35aed61..89c1f92615 100644
> --- a/Lib/test/test_utf8_mode.py
> +++ b/Lib/test/test_utf8_mode.py
> @@ -231,7 +231,7 @@ class UTF8ModeTests(unittest.TestCase):
> 
>  # Check that the command line is decoded from the locale encoding
>  with self.subTest(encoding=encoding):
> -check('utf8=0', [arg.decode(encoding, 'surrogateescape')],
> +check('utf8=0', [arg.decode('ascii', 'surrogateescape')],
>LC_ALL='C')
> 
>  def test_optim_level(self):
> osipovmi@blnn724x:/var/osipovmi/cpython []
> $ ./python -m test test_utf8_mode
> Run tests sequentially
> 0:00:00 [1/1] test_utf8_mode
> 
> == Tests result: SUCCESS ==
> 
> 1 test OK.
> 
> Total duration: 3 sec 65 ms
> Tests result: SUCCESS
> 
> osipovmi@blnn724x:/var/osipovmi/cpython []
> $ LC_ALL=C ./python -X utf8=0
> Python 3.8.0a0 (heads/hpux_force_ascii:7ce2927185, Aug 28 2018, 12:43:04) [C] 
> on hp-ux11
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import locale ; import sys
> >>> sys.getfilesystemencoding() ; locale.getpreferredencoding()
> 'hp-roman8'
> 'roman8'
> >>>
> osipovmi@blnn724x:/var/osipovmi/cpython []
> $

I cannot give a qualified answer on
> Maybe Python should also force ASCII here?
but since you have figured out that the conversion is broken, one must treat is 
as such or use ASCII only and UTF-8 for the rest.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

STINNER Victor  added the comment:

...
> byte 0xA7 decoded to Unicode character U+00A7
...

Well, it confirms what I expected: nl_langinfo(CODESET) announces "roman8", but 
mbstowcs() uses Latin1 encoding in practice.

So I wrote the PR 8969 which forces the ASCII encoding in that case. I'm not 
sure how test_utf8_mode is supposed to be fixed in that case.

Michael: you can try to apply PR 8969, and then apply manually PR 8967 patch:
https://patch-diff.githubusercontent.com/raw/python/cpython/pull/8967.patch

But I expect that with both patches, test_utf8_mode will still fail on 
test_cmd_line(). You can try to modify test_cmd_line() to force encoding to 
"ascii".

What are the values of sys.getfilesystemencoding() and 
locale.getpreferredencoding() with the C locale with PR 8969? I expect "roman8" 
which can cause issue in os.fsencode()/os.fsdecode(). Maybe Python should also 
force ASCII here?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +8442

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

Please see here:
> osipovmi@blnn724x:~ []
> $ uname -a
> HP-UX blnn724x B.11.31 U ia64 HP-UX
> osipovmi@blnn724x:~ []
> $ locale
> LANG=de_DE.utf8
> LC_CTYPE="de_DE.utf8"
> LC_COLLATE="de_DE.utf8"
> LC_MONETARY="de_DE.utf8"
> LC_NUMERIC="de_DE.utf8"
> LC_TIME="de_DE.utf8"
> LC_MESSAGES="de_DE.utf8"
> LC_ALL=
> osipovmi@blnn724x:~ []
> $ cc -o c_locale c_locale.c
> osipovmi@blnn724x:~ []
> $ file c_locale
> c_locale:   ELF-32 executable object file - IA64
> osipovmi@blnn724x:~ []
> $ ./c_locale
> LC_ALL: C C C C C C
> LC_CTYPE: C C C C C C
> nl_langinfo(CODESET): roman8
> byte 0x00 decoded to Unicode character U+
> byte 0x01 decoded to Unicode character U+0001
> byte 0x02 decoded to Unicode character U+0002
> byte 0x03 decoded to Unicode character U+0003
> byte 0x04 decoded to Unicode character U+0004
> byte 0x05 decoded to Unicode character U+0005
> byte 0x06 decoded to Unicode character U+0006
> byte 0x07 decoded to Unicode character U+0007
> byte 0x08 decoded to Unicode character U+0008
> byte 0x09 decoded to Unicode character U+0009
> byte 0x0A decoded to Unicode character U+000A
> byte 0x0B decoded to Unicode character U+000B
> byte 0x0C decoded to Unicode character U+000C
> byte 0x0D decoded to Unicode character U+000D
> byte 0x0E decoded to Unicode character U+000E
> byte 0x0F decoded to Unicode character U+000F
> byte 0x10 decoded to Unicode character U+0010
> byte 0x11 decoded to Unicode character U+0011
> byte 0x12 decoded to Unicode character U+0012
> byte 0x13 decoded to Unicode character U+0013
> byte 0x14 decoded to Unicode character U+0014
> byte 0x15 decoded to Unicode character U+0015
> byte 0x16 decoded to Unicode character U+0016
> byte 0x17 decoded to Unicode character U+0017
> byte 0x18 decoded to Unicode character U+0018
> byte 0x19 decoded to Unicode character U+0019
> byte 0x1A decoded to Unicode character U+001A
> byte 0x1B decoded to Unicode character U+001B
> byte 0x1C decoded to Unicode character U+001C
> byte 0x1D decoded to Unicode character U+001D
> byte 0x1E decoded to Unicode character U+001E
> byte 0x1F decoded to Unicode character U+001F
> byte 0x20 decoded to Unicode character U+0020
> byte 0x21 decoded to Unicode character U+0021
> byte 0x22 decoded to Unicode character U+0022
> byte 0x23 decoded to Unicode character U+0023
> byte 0x24 decoded to Unicode character U+0024
> byte 0x25 decoded to Unicode character U+0025
> byte 0x26 decoded to Unicode character U+0026
> byte 0x27 decoded to Unicode character U+0027
> byte 0x28 decoded to Unicode character U+0028
> byte 0x29 decoded to Unicode character U+0029
> byte 0x2A decoded to Unicode character U+002A
> byte 0x2B decoded to Unicode character U+002B
> byte 0x2C decoded to Unicode character U+002C
> byte 0x2D decoded to Unicode character U+002D
> byte 0x2E decoded to Unicode character U+002E
> byte 0x2F decoded to Unicode character U+002F
> byte 0x30 decoded to Unicode character U+0030
> byte 0x31 decoded to Unicode character U+0031
> byte 0x32 decoded to Unicode character U+0032
> byte 0x33 decoded to Unicode character U+0033
> byte 0x34 decoded to Unicode character U+0034
> byte 0x35 decoded to Unicode character U+0035
> byte 0x36 decoded to Unicode character U+0036
> byte 0x37 decoded to Unicode character U+0037
> byte 0x38 decoded to Unicode character U+0038
> byte 0x39 decoded to Unicode character U+0039
> byte 0x3A decoded to Unicode character U+003A
> byte 0x3B decoded to Unicode character U+003B
> byte 0x3C decoded to Unicode character U+003C
> byte 0x3D decoded to Unicode character U+003D
> byte 0x3E decoded to Unicode character U+003E
> byte 0x3F decoded to Unicode character U+003F
> byte 0x40 decoded to Unicode character U+0040
> byte 0x41 decoded to Unicode character U+0041
> byte 0x42 decoded to Unicode character U+0042
> byte 0x43 decoded to Unicode character U+0043
> byte 0x44 decoded to Unicode character U+0044
> byte 0x45 decoded to Unicode character U+0045
> byte 0x46 decoded to Unicode character U+0046
> byte 0x47 decoded to Unicode character U+0047
> byte 0x48 decoded to Unicode character U+0048
> byte 0x49 decoded to Unicode character U+0049
> byte 0x4A decoded to Unicode character U+004A
> byte 0x4B decoded to Unicode character U+004B
> byte 0x4C decoded to Unicode character U+004C
> byte 0x4D decoded to Unicode character U+004D
> byte 0x4E decoded to Unicode character U+004E
> byte 0x4F decoded to Unicode character U+004F
> byte 0x50 decoded to Unicode character U+0050
> byte 0x51 decoded to Unicode character U+0051
> byte 0x52 decoded to Unicode character U+0052
> byte 0x53 decoded to Unicode character U+0053
> byte 0x54 decoded to Unicode character U+0054
> byte 0x55 decoded to Unicode character U+0055
> byte 0x56 decoded to Unicode character U+0056
> byte 0x57 decoded to Unicode character U+0057
> byte 0x58 decoded to Unicode character U+0058
> byte 0x59 decoded to Unicode character U+0059
> byte 0x5A decoded to Unicode character 

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

STINNER Victor  added the comment:

>   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 214, in check
> self.assertEqual(args, ascii(expected), out)
> AssertionError: "['h\\xa7\\xe9']" != "['h\\xcf\\xd5']"
> - ['h\xa7\xe9']
> + ['h\xcf\xd5']
>  : roman8:['h\xa7\xe9']

Hum, it looks like a bug in the C library of HP-UX. It announces that the 
locale encoding is "roman8", but the mbstowcs() function decodes from the 
Latin1 encoding. The updated test uses the byte string: b'h\xa7\xe9'. The OS 
announces the encoding roman8, so the test expects the Unicode string: 
b'h\xa7\xe9'.decode('roman8') == 'h\xcf\xd5' but it gets 'h\xa7\xe9' which 
looks more like the byte string has been decoded from Latin1: 
b'h\xa7\xe9'.decode('latin1') == 'h\xa7\xe9'.

Michael: would you mind to compile and run the attached c_locale.c test 
program? It sets the LC_ALL locale to C, dump locales (LC_ALL, LC_CTYPE, 
nl_langinfo(CODESET)), and then decode all bytes from the locale encoding 
(LC_CTYPE). The output should help me to understand what is the *effective* 
encoding of HP-UX for the C locale.

You may modify the c_locale.c to replace "C" with "POSIX", to see if the 
behaviour is different.

--
Added file: https://bugs.python.org/file47767/c_locale.c

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

Running off: 217af1d38db3e1e875180c6fa160f0fc80e46003

> $ ./python -m test test_utf8_mode
> Run tests sequentially
> 0:00:00 [1/1] test_utf8_mode
> test test_utf8_mode failed -- Traceback (most recent call last):
>   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 235, in 
> test_cmd_line
> LC_ALL='C')
>   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 214, in check
> self.assertEqual(args, ascii(expected), out)
> AssertionError: "['h\\xa7\\xe9']" != "['h\\xcf\\xd5']"
> - ['h\xa7\xe9']
> + ['h\xcf\xd5']
>  : roman8:['h\xa7\xe9']
> 
> test_utf8_mode failed
> 
> == Tests result: FAILURE ==
> 
> 1 test failed:
> test_utf8_mode
> 
> Total duration: 2 sec 997 ms
> Tests result: FAILURE

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

It unfortunately does not:
> osipovmi@blnn724x:/var/osipovmi/cpython []
> $ git branch
>   3.6
>   3.7
>   bpo-14568
>   bpo-34401
>   bpo-34403
>   bpo-34412
>   bpo-34448
>   bpo-34449
>   bpo-34519
>   master
>   test_c_locale_coercion_hpux
> * utf8_cmd_line
> $ ./python -m test test_utf8_mode
> Run tests sequentially
> 0:00:00 [1/1] test_utf8_mode
> test test_utf8_mode failed -- Traceback (most recent call last):
>   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 231, in 
> test_cmd_line
> check('utf8=0', [c_arg], LC_ALL='C')
>   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 218, in check
> self.assertEqual(args, ascii(expected), out)
> AssertionError: "['h\\xc2\\xa7\\xc3\\xa9']" != "['h\\xf4\\xcf\\xfb\\u02cb']"
> - ['h\xc2\xa7\xc3\xa9']
> + ['h\xf4\xcf\xfb\u02cb']
>  : roman8:['h\xc2\xa7\xc3\xa9']
> 
> test_utf8_mode failed
> 
> == Tests result: FAILURE ==
> 
> 1 test failed:
> test_utf8_mode
> 
> Total duration: 2 sec 921 ms
> Tests result: FAILURE

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

Victor, looking to...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

STINNER Victor  added the comment:

Hi, I'm the author of the UTF-8 Mode PEP (PEP 540) and its implementation. I 
wrote test_utf8_mode. I wasn't sure that it was a good idea to hardcode the 
locale encoding depending on the platform. The fact that AIX and HP-UX use 
different locale encoding confirms that it was a bad choice. My PR 8967 gets 
the locale encoding at runtime instead of hardcoding it. It should fix the test 
on AIX and HP-UX.

To fix the test on HP-UX, I also removed the euro sign (U+20AC: €) from the 
test string. There is no need to test large code point: a single non-ASCII 
character is enough to validate the code.

Michael Osipov: would you mind to test my PR on HP-UX please?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +8441

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Change by Michael Osipov <1983-01...@gmx.net>:


--
keywords: +patch
pull_requests: +8440
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

Now I know why this cannot with Roman 8: it contains chars which are multibyte 
in Unicode (UTF-8) which cannot be mapped into a 7-bit/8-bit encoding. 
Therefore CP1252 does not work because it has Unicode chars too. ISO-8859-1 
solely consists of single byte chars.

This test needs to be skipped on HP-UX. I will provide a patch for that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

Wow, this is pretty surprising. The very same patch for AIX works on HP-UX 
flawlessly:

$ ./python -m test test_utf8_mode
Run tests sequentially
0:00:00 [1/1] test_utf8_mode

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2 sec 769 ms
Tests result: SUCCESS


I still don't really understand why because decode() and ascii() are comparing 
apples and oranges to me.

Michael, since you provided a decent solution would you mind to extend your 
patch for HP-UX? You deserve the credits.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Felt]

Michael Felt  added the comment:

On 27/08/2018 15:22, Michael Osipov wrote:
> Michael Osipov <1983-01...@gmx.net> added the comment:
>
> So I changed the test code to:
>
> diff --git a/Lib/test/test_utf8_mode.py b/Lib/test/test_utf8_mode.py
> index 26e2e13ec5..d9f8a3ed8b 100644
> --- a/Lib/test/test_utf8_mode.py
> +++ b/Lib/test/test_utf8_mode.py
> @@ -208,7 +208,7 @@ class UTF8ModeTests(unittest.TestCase):
>  def test_cmd_line(self):
>  arg = 'h\xe9\u20ac'.encode('utf-8')
>  arg_utf8 = arg.decode('utf-8')
> -arg_ascii = arg.decode('ascii', 'surrogateescape')
> +arg_ascii = arg.decode('roman8', 'surrogateescape')
>  code = 'import locale, sys; print("%s:%s" % 
> (locale.getpreferredencoding(), ascii(sys.argv[1:])))'
>
>  def check(utf8_opt, expected, **kw):
>
> and the output is:
> ==
> FAIL: test_cmd_line (test.test_utf8_mode.UTF8ModeTests)
> --
> Traceback (most recent call last):
>   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 224, in 
> test_cmd_line
> check('utf8=0', [c_arg], LC_ALL='C')
>   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 217, in check
> self.assertEqual(args, ascii(expected), out)
> AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != 
> "['h\\xfb\\u02cb\\xe3\\x82\\u02dc']"
> - ['h\xc3\xa9\xe2\x82\xac']
> + ['h\xfb\u02cb\xe3\x82\u02dc']
>  : roman8:['h\xc3\xa9\xe2\x82\xac']
>
> I still don't understand that.
Something I found helpful was to change:

    check('utf8=0', [c_arg], LC_ALL='C')

to
    check('utf8=0', [c_arg], LC_ALL='C', failure=True )

This also fails, but it shows what is being executed.

Further, my 'understanding' is that ascii(whatever) is much smarter than
whatever.decode('ascii', ...) does. Also, ascii() tends to use the \x
shorthand, while decode('ascii', 'surrogateescape') uses the \udc prefix.

And, while you might still consider it a 'bug', did you try using c_arg
= arg.decode('iso-88859-1') ?

Michael (F)

>
> I believe that surrogate escape only works for ASCII and nothing else. If so, 
> this test must be skipped on HP-UX and AIX.
>
> --
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

Maybe Victor Stinner has some insights here.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

Maybe skipping the test is the best thing:
diff --git a/Lib/test/test_utf8_mode.py b/Lib/test/test_utf8_mode.py
index 26e2e13ec5..d6c4b321be 100644
--- a/Lib/test/test_utf8_mode.py
+++ b/Lib/test/test_utf8_mode.py
@@ -12,7 +12,7 @@ from test.support.script_helper import assert_python_ok, 
assert_python_failure


 MS_WINDOWS = (sys.platform == 'win32')
-
+HPUX = (sys.platform.startswith('hp-ux'))

 class UTF8ModeTests(unittest.TestCase):
 DEFAULT_ENV = {
@@ -205,6 +205,7 @@ class UTF8ModeTests(unittest.TestCase):
 self.assertEqual(out, 'UTF-8 UTF-8')

 @unittest.skipIf(MS_WINDOWS, 'test specific to Unix')
+@unittest.skipIf(HPUX, 'test specific to Unix with ASCII default locale')
 def test_cmd_line(self):
 arg = 'h\xe9\u20ac'.encode('utf-8')
 arg_utf8 = arg.decode('utf-8')

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

So I changed the test code to:

diff --git a/Lib/test/test_utf8_mode.py b/Lib/test/test_utf8_mode.py
index 26e2e13ec5..d9f8a3ed8b 100644
--- a/Lib/test/test_utf8_mode.py
+++ b/Lib/test/test_utf8_mode.py
@@ -208,7 +208,7 @@ class UTF8ModeTests(unittest.TestCase):
 def test_cmd_line(self):
 arg = 'h\xe9\u20ac'.encode('utf-8')
 arg_utf8 = arg.decode('utf-8')
-arg_ascii = arg.decode('ascii', 'surrogateescape')
+arg_ascii = arg.decode('roman8', 'surrogateescape')
 code = 'import locale, sys; print("%s:%s" % 
(locale.getpreferredencoding(), ascii(sys.argv[1:])))'

 def check(utf8_opt, expected, **kw):

and the output is:
==
FAIL: test_cmd_line (test.test_utf8_mode.UTF8ModeTests)
--
Traceback (most recent call last):
  File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 224, in 
test_cmd_line
check('utf8=0', [c_arg], LC_ALL='C')
  File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 217, in check
self.assertEqual(args, ascii(expected), out)
AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != 
"['h\\xfb\\u02cb\\xe3\\x82\\u02dc']"
- ['h\xc3\xa9\xe2\x82\xac']
+ ['h\xfb\u02cb\xe3\x82\u02dc']
 : roman8:['h\xc3\xa9\xe2\x82\xac']

I still don't understand that.

I believe that surrogate escape only works for ASCII and nothing else. If so, 
this test must be skipped on HP-UX and AIX.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

I think you are absoltely right.

> In any case, it seems to be broken for any system that does not have UTF-8 as 
> default.

You likely mean ASCII. Python assumes that LANG=C is ASCII which is not the 
case for AIX and HP-UX.

Your patch looks reasonable, I will try this on Monday. The problem is that 
there is no roman8 codec in Python. Maybe ISO-8859-1 will do it for the test, 
but I am still eager to add one.

> I mention this because it seems neither roman8 nor roman9 have 'official' iso 
> names or alias (correct me if I am wrong).

There are no ISO names because this is not an ISO encoding. This is an HP 
invention aka hp-roman8 (roman8, ibm-1051, r8, Cp1051).

Edit: there is roman8 support: 
https://github.com/python/cpython/blob/e42b705188271da108de42b55d9344642170aa2b/Lib/encodings/hp_roman8.py
 as well as aliases.

There are a few aliases missing: cp1051, ibm1051 and hp-roman8. This needs an 
additonal PR.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Felt]

Michael Felt  added the comment:

As the AIX complaint is (was once the PR merges):
AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != 
"['h\\udcc3\\udca9\\udce2\\udc82\\udcac']"
- ['h\xc3\xa9\xe2\x82\xac']
+ ['h\udcc3\udca9\udce2\udc82\udcac']

And the HP-UX complaint is:
  File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 223, in check
self.assertEqual(args, ascii(expected), out)
AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != 
"['h\\udcc3\\udca9\\udce2\\udc82\\udcac']"
- ['h\xc3\xa9\xe2\x82\xac']
+ ['h\udcc3\udca9\udce2\udc82\udcac']

Maybe a change such as:

--- a/Lib/test/test_utf8_mode.py
+++ b/Lib/test/test_utf8_mode.py
@@ -219,6 +219,8 @@ class UTF8ModeTests(unittest.TestCase):
 check('utf8', [arg_utf8])
 if sys.platform == 'darwin' or support.is_android:
 c_arg = arg_utf8
+elif (platform.system == "AIX") or
+  sys.platform.startswith("hp-ux"):
+c_arg = arg.decode('iso-8859-1')
 else:
 c_arg = arg_ascii
 check('utf8=0', [c_arg], LC_ALL='C')

I mention this because it seems neither roman8 nor roman9 have 'official' iso 
names or alias (correct me if I am wrong).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Felt]

Michael Felt  added the comment:

It might be as simple as what I saw for AIX:

diff --git a/Lib/test/test_utf8_mode.py b/Lib/test/test_utf8_mode.py
index 26e2e13ec5..3e918fd54c 100644
--- a/Lib/test/test_utf8_mode.py
+++ b/Lib/test/test_utf8_mode.py
@@ -219,6 +219,8 @@ class UTF8ModeTests(unittest.TestCase):
 check('utf8', [arg_utf8])
 if sys.platform == 'darwin' or support.is_android:
 c_arg = arg_utf8
+elif sys.platform.startswith("aix"):
+c_arg = arg.decode('iso-8859-1')
 else:
 c_arg = arg_ascii
 check('utf8=0', [c_arg], LC_ALL='C')

so, adding below might be all that is needed:
+elif sys.platform == "hpux":
+c_arg = arg.decode('roman8')

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Łukasz Langa]

Change by Łukasz Langa :


--
keywords: +3.7regression
versions:  -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Łukasz Langa]

Change by Łukasz Langa :


--
dependencies: +test_cmd_line test_utf8_mode test_warnings fail in all FreeBSD 
3.x (3.8) buildbots

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Change by Michael Osipov <1983-01...@gmx.net>:


--
versions: +Python 3.6, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Felt]

Michael Felt  added the comment:

Although the default is different (i.e., roman8 versus latin1 (iso8859-1)) both 
HP-UX and AIX (like Windows, cp1252) this issue and issue 33347 are related.

As I mentioned in https://bugs.python.org/issue34347#msg323319 the string seen 
by self.get_output() is not the same string as "expected".

If I recall, there may be a way to almost get the two be the same - excect 
"expected" is a bytes object and the value returned as CLI output is a regular 
string.

I am thinking, maybe the "easy" way will be to add AIX, HP-UX, and others to 
skip this test. Rather than hard-code, do a query to see what the default is, 
and it it is not UTF-8 - skip the test.

In any case, it seems to be broken for any system that does not have UTF-8 as 
default.

--
nosy: +Michael.Felt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

Thanks, I'll do that. Hopefully I can provide a patch for. Though, I am 
convinced that I have to write a custom codec for roman8 to make all at stuff 
work flawlessly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Terry J. Reedy]

Terry J. Reedy  added the comment:

You might get more information asking questions on python-list.

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34403] test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[Michael Osipov]

New submission from Michael Osipov <1983-01...@gmx.net>:

Running from 3.7 branch on HP-UX 11.31 ia64, 32 bit, big endian.
The test output is:
> Re-running failed tests in verbose mode
> Re-running test 'test_utf8_mode' in verbose mode
> test_cmd_line (test.test_utf8_mode.UTF8ModeTests) ... FAIL
> test_env_var (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_filesystemencoding (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_io (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_io_encoding (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_locale_getpreferredencoding (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_optim_level (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_posix_locale (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_stdio (test.test_utf8_mode.UTF8ModeTests) ... ok
> test_xoption (test.test_utf8_mode.UTF8ModeTests) ... ok
> 
> ==
> FAIL: test_cmd_line (test.test_utf8_mode.UTF8ModeTests)
> --
> Traceback (most recent call last):
>   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 230, in 
> test_cmd_line
> check('utf8=0', [c_arg], LC_ALL='C')
>   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 223, in check
> self.assertEqual(args, ascii(expected), out)
> AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != 
> "['h\\udcc3\\udca9\\udce2\\udc82\\udcac']"
> - ['h\xc3\xa9\xe2\x82\xac']
> + ['h\udcc3\udca9\udce2\udc82\udcac']
>  : roman8:['h\xc3\xa9\xe2\x82\xac']
> 
> --
> Ran 10 tests in 2.595s
> 
> FAILED (failures=1)
> test test_utf8_mode failed
> 1 test failed again:
> test_utf8_mode
> 
> == Tests result: FAILURE then FAILURE ==
> 
> 1 test failed:
> test_utf8_mode
> 
> 1 re-run test:
> test_utf8_mode
> 
> Total duration: 7 sec 265 ms
> Tests result: FAILURE then FAILURE
> Makefile:1066: recipe for target 'test' failed
> gmake: *** [test] Error 2
> 

I tried to understand the issue, but my Python knowledge is too low, especially 
I do not understand by a byte array "arg = 'h\xe9\u20ac'.encode('utf-8')" is 
passed as one arg to the forked process.

I highly assume that this is related to the non-standard, default character 
encoding on HP-UX: https://en.wikipedia.org/wiki/HP_Roman#HP_Roman-8 (roman8).

A stupid 8 bit encoding. The very same snippet on FreeBSD says:
> $ LC_ALL=C python3.6 test_utf8.py
> US-ASCII:[]

Willing to test and modify if someone tells what to do.

--
components: Library (Lib), Tests
messages: 323516
nosy: michael-o
priority: normal
severity: normal
status: open
title: test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com