[issue16414] Add support.NONASCII to test non-ASCII characters

[STINNER Victor]

Changes by STINNER Victor :


--
versions: +Python 3.2, Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[Roundup Robot]

Roundup Robot added the comment:

New changeset 41658a4fb3cc by Victor Stinner in branch '3.2':
Issue #16218, #16414, #16444: Backport FS_NONASCII, TESTFN_UNDECODABLE,
http://hg.python.org/cpython/rev/41658a4fb3cc

New changeset 4d40c1ce8566 by Victor Stinner in branch '3.3':
(Merge 3.2) Issue #16218, #16414, #16444: Backport FS_NONASCII,
http://hg.python.org/cpython/rev/4d40c1ce8566

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[STINNER Victor]

STINNER Victor added the comment:

Handling non-ASCII paths is always a pain. I don't plan to backport 
support.FS_NONASCII to Python 3.3 right now, but I may backport it later.

--
resolution:  -> fixed
status: open -> closed
versions:  -Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[STINNER Victor]

STINNER Victor added the comment:

> Why were you add '- ' suffix to TESTFN_NONASCII?

Oops, the space was a mistake. I add "-" just for the readability of the 
generated filename.

> I don't see U+00A0 and U+20AC in the changeset.

Oh, I forgot to update the patch with the latest results of "brute2.py". It is 
now fixed.

Thanks for the review!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[Roundup Robot]

Roundup Robot added the comment:

New changeset fce9e892c65d by Victor Stinner in branch 'default':
Issue #16414: Fix test_os on Windows, don't test os.listdir() with undecodable
http://hg.python.org/cpython/rev/fce9e892c65d

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[Roundup Robot]

Roundup Robot added the comment:

New changeset 7f90305d9f23 by Victor Stinner in branch 'default':
Issue #16414: Test more characters for support.FS_NONASCII
http://hg.python.org/cpython/rev/7f90305d9f23

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[STINNER Victor]

Changes by STINNER Victor :


Added file: http://bugs.python.org/file27912/brute2.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[Roundup Robot]

Roundup Robot added the comment:

New changeset 55710b8c6670 by Victor Stinner in branch 'default':
Issue #16414: Fix typo in support.TESTFN_NONASCII (useless space)
http://hg.python.org/cpython/rev/55710b8c6670

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I don't see U+00A0 and U+20AC in the changeset.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Why were you add '- ' suffix to TESTFN_NONASCII?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[Roundup Robot]

Roundup Robot added the comment:

New changeset 0e9fbdda3c92 by Victor Stinner in branch 'default':
Issue #16414: Fix support.TESTFN_UNDECODABLE and 
test_genericpath.test_nonascii_abspath()
http://hg.python.org/cpython/rev/0e9fbdda3c92

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[Roundup Robot]

Roundup Robot added the comment:

New changeset de8cf1ece068 by Victor Stinner in branch 'default':
Issue #16414: Add support.FS_NONASCII and support.TESTFN_NONASCII
http://hg.python.org/cpython/rev/de8cf1ece068

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[STINNER Victor]

STINNER Victor added the comment:

> I tested brute.py for all supported in Python encodings:

Oh thanks, interesting result. I completed the encoding list and the character 
list: see brute2.py. I added "joker" characters: U+00A0 and U+20AC which match 
requierements for most locale encodings.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I tested brute.py for all supported in Python encodings:

No character for encoding cp1006:surrogateescape :-(
No character for encoding cp720:surrogateescape :-(
No character for encoding cp864:surrogateescape :-(
No character for encoding iso8859_3:surrogateescape :-(
No character for encoding iso8859_6:surrogateescape :-(
No character for encoding mac_arabic:surrogateescape :-(
No character for encoding mac_farsi:surrogateescape :-(

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[STINNER Victor]

Changes by STINNER Victor :


Removed file: http://bugs.python.org/file27892/support_non_ascii.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[STINNER Victor]

STINNER Victor added the comment:

> I think you should ensure that os.fsdecode(os.fsencode(character)) == 
> character.

Chosen characters respect this property, but it doesn't hurt to add such check.

> Can you use a name that reflects that this is a specific type
> of non-ASCII character having a special property (e.g. FS_NONASCII)?

Done.

--
Added file: http://bugs.python.org/file27905/support_non_ascii-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[STINNER Victor]

STINNER Victor added the comment:

I tested support_non_ascii-2.patch on Windows with cp932 ANSI code page (FS 
encoding), and on Linux with ASCII, ISO-8859-1, ISO-8859-15 and UTF-8 locale 
encodings.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[STINNER Victor]

STINNER Victor added the comment:

> If NONASCII is None I suggest the followed fallback code

I prefer to not "brute force" Unicode because it would slow down any test, even 
tests not using FS_NONASCII. I wrote attached brute.py script to compute an 
exhaustive list of non-ASCII characters encodable to "any" locale encoding. My 
locale encoding list is not complete, but it should be enough for our purpose. 
The list can be completed later.

--
Added file: http://bugs.python.org/file27906/brute.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[Chris Jerdonek]

Chris Jerdonek added the comment:

+# NONASCII: non-ASCII character encodable by os.fsencode(),
+# or None if there is no such character.
+NONASCII = None

Can you use a name that reflects that this is a specific type of non-ASCII 
character having a special property (e.g. FS_NONASCII)?  I think "ASCII" should 
be reserved for a generic non-ASCII character.  Moreover, there may be other 
types of non-ASCII we can add in the future.

--
nosy: +chris.jerdonek

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

If NONASCII is None I suggest the followed fallback code

for i in range(0x80, 0x):
character = chr(i)
if character.isprintable():
try:
if os.fsdecode(os.fsencode(character)) == character:
NONASCII = character
break
except UnicodeError:
pass

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I think you should ensure that os.fsdecode(os.fsencode(character)) == character.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16414] Add support.NONASCII to test non-ASCII characters

[STINNER Victor]

New submission from STINNER Victor:

Attached patch adds support.NONASCII to have a "portable" non-ASCII character 
that can be used to test non-ASCII strings. The patch uses it in some existing 
functions.

I wrote the patch on the default branch, we may start to use it since Python 
3.2.

--
components: Tests
files: support_non_ascii.patch
keywords: patch
messages: 174897
nosy: haypo, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Add support.NONASCII to test non-ASCII characters
versions: Python 3.3, Python 3.4
Added file: http://bugs.python.org/file27892/support_non_ascii.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com