[issue35009] argparse throws UnicodeEncodeError for printing help with unicode choices

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Thanks Serhiy, I have limited knowledge of unicode and hence I thought this to 
be a fix. I am closing this as per msg332337 and the PR I opened for issue24307 
(Feel free to close issue24307 as per Victor's resolution if needed)

--
resolution:  -> wont fix
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35009] argparse throws UnicodeEncodeError for printing help with unicode choices

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Even if you encode default values for output, you can not pass unicode values 
from command line. You can pass a 8-bit string 
'\xe6\x97\xa9\xe4\xb8\x8a\xe5\xa5\xbd' which differs from a Unicode string 
u"早上好". Even after resolving the output issue, unicode choices will never work.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35009] argparse throws UnicodeEncodeError for printing help with unicode choices

[Karthikeyan Singaravelan]

New submission from Karthikeyan Singaravelan :

argparse module uses str() in a few places where passing unicode strings will 
throw UnicodeDecodeError. In Python 3 these scripts run fine since Python 3 has 
unicode strings by default. I am working on this along with finding more places 
where this can throw error along with adding relevant tests for those 
scenarios. I couldn't find any related issues for this in the bug tracker. Feel 
free to close this if it's a duplicate. 

# foo_argparse.py with unicode choices

```
# -*- coding: utf-8 -*-

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--foo', help='foo help', choices=[u"早上好", u"早上好 早上好"])
args = parser.parse_args()

```

# printing help causes error since str is used for choices

$ ./python.exe ../backups/foo_argparse.py --help
Traceback (most recent call last):
  File "../backups/foo_argparse.py", line 5, in 
parser.add_argument('--foo', help='foo help', choices=[u"早上好", u"早上好 早上好"])
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/argparse.py", 
line 1308, in add_argument
self._get_formatter()._format_args(action, None)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/argparse.py", 
line 578, in _format_args
get_metavar = self._metavar_formatter(action, default_metavar)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/argparse.py", 
line 565, in _metavar_formatter
choice_strs = [str(choice) for choice in action.choices]
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: 
ordinal not in range(128)


In case we use unicode() for the above and then also fix the place where 
comparison is done then using the correct choice passes. But using wrong choice 
throws an error where exception string is formed using str that causes error


$ ./python.exe ../backups/foo_argparse_unicode.py --foo 早上好 # passes
$ ./python.exe ../backups/foo_argparse_unicode.py --help # passes
usage: foo_argparse_unicode.py [-h] [--foo {早上好,早上好 早上好}]

optional arguments:
  -h, --help   show this help message and exit
  --foo {早上好,早上好 早上好}  foo help


$ ./python.exe ../backups/foo_argparse_unicode.py --foo 1 # Fails
Traceback (most recent call last):
  File "../backups/foo_argparse_unicode.py", line 7, in 
args = parser.parse_args()
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/argparse.py", 
line 1705, in parse_args
args, argv = self.parse_known_args(args, namespace)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/argparse.py", 
line 1744, in parse_known_args
self.error(str(err))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 49-51: 
ordinal not in range(128)

--
components: Library (Lib)
messages: 327879
nosy: xtreak
priority: normal
severity: normal
status: open
title: argparse throws UnicodeEncodeError for printing help with unicode choices
type: behavior
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com