Yes Python Unicode is fun to work with isn't it :)
Grant raises a good point. The \u syntax is the best way to ensure that
what you intended to render actually is correct.
Python provides a means to specify the source character set encoding at
the top of a python file.
If one does that then text will be converted to Unicode from that
character set in the file space.
For example:
# -*- coding: utf-8 -*-
exampleText = u"This is some Unicode with non- ascii character: ü"
print repr(exampleText)
>>> u'This is some Unicode with non- ascii character: \xfc'
However, in the command line interpreter Python does not know what the
source encoding is and
it must be explicitly defined.
>>> exampleText = unicode("This is some Unicode with non- ascii
character: ü", "utf8")
>>> exampleText
>>> u'This is some Unicode with non- ascii character: \xfc'
My example in the i18n Busy Developer Guide should have been the above
and not:
>>> exampleInstance.exampleText = u"This is some Unicode with non-ascii
character: ü"
>>> exampleInstance.exampleText
u"This is some Unicode with non-ascii character: \xc3\xbc"
I have updated the guide to correct the error.
Using the \u syntax is a better choice because no encoding needs to be
explicitly specified in the file or the terminal.
For example:
exampleText = u"This is some Unicode with non- ascii character: \u00FC"
Of course if your terminal uses the ASCII character set then it will not
render correctly :)
--Brian
Brian Kirsch - Cosmo Developer / Chandler Internationalization Engineer
Open Source Applications Foundation
543 Howard St. 5th Floor
San Francisco, CA 94105
http://www.osafoundation.org
Grant Baillie wrote:
I've run across a couple of cases of specifying unicode characters in
Python code that were a little fishy, so I thought I'd send out a
long, rambly email to the list.
The 10-second summary is: If you want to specify a non-ASCII
character in a unicode string, the python \uxxxx escape is your
friend. With anything else, you're playing with fire.
So, to cut a short story long, I was looking at a test case in
Chandler, where we were trying to come up with a non-ASCII path to
use in a Chandler profile directory:
TestCrypto.py:13: u = u"profileDir_(\xc3\xbc)" # u umlaut
This actually succeeds in setting u to be a non-ASCII string, except
that it doesn't contain a "u umlaut". When you specify a u"..." style
string in Python, you're telling the interpreter to assume each
character in the string is a unicode code point. Looking at the list in
<http://www.unicode.org/Public/UNIDATA/NamesList.txt>
you can determine that "u umlaut" is the Unicode character(*)
00FC LATIN SMALL LETTER U WITH DIAERESIS
but in the above, the \xc3 and \xbc are interpreted as:
00C3 LATIN CAPITAL LETTER A WITH TILDE
00BC VULGAR FRACTION ONE QUARTER
Clearly, we don't want any vulgarity in our paths, now do we :) ?
It turns out that the author of the above code was having trouble
entering u umlaut (in a console, or a code editor). As mentioned
above, the easiest and most portable way to do this kind of thing is
to use the \u escape, viz:
u = u"profileDir_(\u00fc)" # u umlaut
In the case of source files, Python has some handy conventions for
specifying what character encoding of a source file is (see <http://
docs.python.org/ref/encodings.html#encodings>). Unfortunately, it
turns out that there's no convention that's adopted by many editors.
Possibly this is a reason to require everyone to use emacs, or vim,
but the resulting religious war would take us well past Chandler 1.0 :).
In the case of entering text in an interactive session, you're
somewhat at the mercy of your terminal program, as well as your
locale. To continue the story, the characters \xc3\xbc above (which
are the UTF-8 encoding of \u00fc), did not come from nowhere. The
developer mentioned earlier copy-and-pasted them from the following
bit of text in the I18n Busy Developers Guide:
>>> exampleInstance.exampleText = u"This is some unicode with non-
ascii character: ü"
>>> exampleInstance.exampleText
u"This is some unicode with non-ascii character: \xc3\xbc"
As we determined above, the printed-out value does not end with ü. In
fact, what happened above was the terminal program was using UTF-8,
but Python had no idea that that was the case, and converted the raw
UTF-8 bytes to unicode characters.
--Grant
(*) It's also representable as the sequence of two characters
0075 LATIN SMALL LETTER U
0308 COMBINING DIAERESIS (Dialytika)
= double dot above, umlaut
= Greek dialytika
= double derivative
x (diaeresis - 00A8)
but that's a whole different can of fish, er crosstown bus.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "chandler-dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/chandler-dev
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "chandler-dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/chandler-dev
