[issue2931] optparse: various problems with unicode and gettext

[STINNER Victor]

STINNER Victor  added the comment:

This issue was reported in 2008 on Python 2.5. The latest comment is about 
Python 2.

The latest Python version is now Python 3.9 and uses Unicode by default.

I close the issue since there is no activity for 4 years. More tests are always 
welcomed, so someone can still add new tests. Note that the optparse module is 
deprecated since Python 3.2.

--
resolution:  -> out of date
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Irit Katriel]

Irit Katriel  added the comment:

The tests have not been merged yet.

--
nosy: +iritkatriel
versions: +Python 3.10, Python 3.8, Python 3.9 -Python 2.7, Python 3.1, Python 
3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Sean Wang]

Sean Wang added the comment:

This bug still exists in Python 2.7.10 with optparse version 1.5.3.
When the default_value is not ASCII encoded, it would raise 
`UnicodeEncodeError: 'ascii' codec can't encode characters`

this error is due to the `str` usage in `expand_default` method:

def expand_default(self, option):
if self.parser is None or not self.default_tag:
return option.help

default_value = self.parser.defaults.get(option.dest)
if default_value is NO_DEFAULT or default_value is None:
default_value = self.NO_DEFAULT_VALUE

return option.help.replace(self.default_tag, str(default_value))

--
nosy: +Sean.Wang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Sean Wang]

Sean Wang added the comment:

Sorry, missed one condition:
I used `unicode_literals` in Python 2.7.10, example below:

>>> from __future__ import unicode_literals
>>> str('api名称')
Traceback (most recent call last):
  File "", line 1, in 
UnicodeEncodeError: 'ascii' codec can't encode characters in position 3-4: 
ordinal not in range(128)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Sean Wang]

Sean Wang added the comment:

when an unicode option.default_value could not be ascii encoded, it would throw 
exception, detailed logs below:
  File "/Users/seanwang/Documents/dev/foo/bar.py", line 119, in main
parser.print_help()
  File 
"/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py",
 line 1670, in print_help
file.write(self.format_help().encode(encoding, "replace"))
  File 
"/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py",
 line 1650, in format_help
result.append(self.format_option_help(formatter))
  File 
"/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py",
 line 1630, in format_option_help
result.append(OptionContainer.format_option_help(self, formatter))
  File 
"/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py",
 line 1074, in format_option_help
result.append(formatter.format_option(option))
  File 
"/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py",
 line 316, in format_option
help_text = self.expand_default(option)
  File 
"/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py",
 line 288, in expand_default
return option.help.replace(self.default_tag, str(default_value))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 3-4: 
ordinal not in range(128)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Greg Ward]

Greg Ward added the comment:

 I've turned ash's test program into a bunch of test cases against 
 Python 3.5 trunk.  Is it worth committing them?

Yeah, probably. Review comments...

+try:
+self.parser.error(RUSSIAN_TEXT)
+except InterceptedError:
+pass

Why not self.assertRaises()?

Also, when I run the test on its own, it prints

'''
Usage: regrtest.py [options]

regrtest.py: error: Русский текст --unknown
'''

to stderr. Probably need to fiddle with sys.stderr to fix that. Blech.

Finally:

+try:
+import optparse
+old_gettext = optparse._
+optparse._ = dummy_gettext
+
+try:
+OptionParser().parse_args([--unknown])
+except SystemExit:
+pass
+finally:
+optparse._ = old_gettext

This is a lot easier with mock.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[A.M. Kuchling]

A.M. Kuchling added the comment:

I've turned ash's test program into a bunch of test cases against Python 3.5 
trunk.  Is it worth committing them?

--
nosy: +akuchling
Added file: http://bugs.python.org/file38940/issue2931.txt

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Mark Lawrence]

Changes by Mark Lawrence breamore...@yahoo.co.uk:


--
nosy:  -BreamoreBoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Ivan Vilata i Balaguer]

Ivan Vilata i Balaguer ivil...@users.sourceforge.net added the comment:

After so much time I've checked again with the little script I sent and I see 
that it doesn't happen under Python 2.7 (2.7.1+), but it does under 2.6 (2.6.6) 
and 2.5 (2.5.5).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

I’m afraid 2.5 and 2.6 don’t get bug fixes any more, only security fixes.  For 
2.7 and 3.x, even if your bug can’t be reproduced, I think it would be useful 
to add the test to prevent a regression.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Ezio Melotti]

Ezio Melotti ezio.melo...@gmail.com added the comment:

+1

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Steven Bethard]

Steven Bethard steven.beth...@gmail.com added the comment:

Yep, argparse almost certainly has the same kind of problems - I basically 
copied the optparse gettext behavior into argparse because I don't really know 
how that stuff works but figured people must have wanted what was in there. ;-)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

It would be nice to test argparse for the same behavior.

--
nosy: +bethard, eric.araujo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Mark Lawrence]

Mark Lawrence breamore...@yahoo.co.uk added the comment:

Alexy, there would be a much better chance of getting this accepted if you 
could supply a patch file that also included unit tests.

--
nosy: +BreamoreBoy, aronacher
versions: +Python 3.1, Python 3.2 -Python 2.6

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Alexey Shamrin]

Changes by Alexey Shamrin sham...@gmail.com:


--
versions: +Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Alexey Shamrin]

Alexey Shamrin sham...@gmail.com added the comment:

More than a year passed since I reported this... Could someone suggest
how to move this forward? If needed, I can try to improve patch, test or
description of this issue. Should I, for example, split this into
separate issues?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Daniel Diniz]

Changes by Daniel Diniz aja...@gmail.com:


--
nosy: +ezio.melotti, haypo, loewis
priority:  - normal
stage:  - patch review
type:  - behavior
versions: +Python 2.6 -Python 2.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Ivan Vilata i Balaguer]

Ivan Vilata i Balaguer [EMAIL PROTECTED] added the comment:

The attached version of ``optparse_unicode.py`` doensn't depend on a
UTF-8 locale, sorry.

Added file: http://bugs.python.org/file10650/optparse_unicode2.py

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Sam Pablo Kuper]

Sam Pablo Kuper [EMAIL PROTECTED] added the comment:

Using non-ASCII characters in an optparse help string also causes 
UnicodeDecodeErrors. Here's the relevant part of the traceback:

File /home/spk30/opt/ActivePython-2.5/lib/python2.5/optparse.py, line 
1655, in print_help
file.write(self.format_help().encode(encoding, replace))

NB. Adding an encoding declaration at the beginning of the python 
script which used a non-ASCII character in an optparse help string 
didn't solve the problem.

--
nosy: +sampablokuper
versions: +Python 2.5

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Alexey Shamrin]

Alexey Shamrin [EMAIL PROTECTED] added the comment:

sampablokuper, I don't think your problem is relevant to this issue. In
addition to encoding declaration you should use unicode strings: uyour
non-ASCII text. Or wait for Python 3.0, where strings will be unicode
by default.

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Sam Pablo Kuper]

Sam Pablo Kuper [EMAIL PROTECTED] added the comment:

ash, you are correct; my bad. Thanks for the heads-up.

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2931
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2931] optparse: various problems with unicode and gettext

[Alexey Shamrin]

Alexey Shamrin [EMAIL PROTECTED] added the comment:

I've also attached a patch that fixes all these issues and also allows
the word error to be translated with gettext.

Regarding the use of `locale.getpreferredencoding` instead of
`sys.getdefaultencoding`. On my system (Windows XP, Russian) I get:

Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
(Intel)] on win32
Type help, copyright, credits or license for more information.
 import sys, locale
 sys.getdefaultencoding()
'ascii'
 locale.getpreferredencoding()
'cp1251'

Using cp1251 on my system makes much more sense. It's used as a default
encoding everywhere in the system. For example, in Notepad.

--
keywords: +patch
Added file: http://bugs.python.org/file10387/optparse.py.patch

__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2931
__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com