[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)

[Terry Reedy]

On 11/13/2021 4:35 PM, pt...@austin.rr.com wrote:
I’ve not been following the thread, but Steve Holden forwarded me the 

To explore the extreme case, I wrote a pyparsing transformer to convert 
identifiers in a body of Python source to mixed font, equivalent to the 
original source after NFKC normalization. Here are hello.py, and a 
snippet from unittest/utils.py:

def 𝚑𝓮𝖑𝒍𝑜():

     try:

𝔥e𝗅𝕝𝚘︴ = "Hello"

𝕨𝔬r𝓵ᵈ﹎ = "World"

         ᵖ𝖗𝐢𝘯𝓽(f"{𝗵ｅ𝓵𝔩º_}, {𝖜ₒ𝒓lⅆ︴}!")

     except 𝓣𝕪ᵖｅ𝖤𝗿ᵣ𝖔𝚛 as ⅇ𝗑c:

𝒑rℹₙₜ("failed: {}".𝕗𝗼ʳᵐªｔ(ᵉ𝐱𝓬))

if _︴ⁿ𝓪𝑚𝕖__ == "__main__":

𝒉eℓˡ𝗈()

# snippet from unittest/util.py

_𝓟Ⅼ𝖠𝙲𝗘ℋ𝒪Lᴰ𝑬𝕽﹏𝕷𝔼𝗡 = 12

def _𝔰ʰ𝓸ʳ𝕥𝙚𝑛(𝔰, p𝑟𝔢fi𝖝𝕝𝚎𝑛, ｓᵤ𝑓𝗳𝗂𝑥𝗹ₑ𝚗):

     ˢ𝗸ｉ𝗽 = 𝐥ｅ𝘯(𝖘) - ｐr𝚎𝖋𝐢x𝗅ᵉ𝓷 - 𝒔𝙪ffｉ𝘅𝗹𝙚ₙ

     if sｋi𝘱 > _𝐏𝗟𝖠𝘊𝙴H𝕺Ｌ𝕯𝙀𝘙﹏L𝔈𝒩:

𝘴 = '%s[%d chars]%s' % (𝙨[:𝘱𝐫𝕖𝑓𝕚ｘℓ𝒆𝕟], ₛ𝚔𝒊p, 𝓼[𝓁𝒆𝖓(𝚜) - 
𝙨𝚞𝒇fix𝙡ᵉ𝘯:])

     return ₛ

You should able to paste these into your local UTF-8-aware editor or IDE 
and execute them as-is.

Wow.  After pasting the util.py snippet into current IDLE, which on my 
Windows machine* displays the complete text:

>>> dir()
['_PLACEHOLDER_LEN', '__annotations__', '__builtins__', '__doc__', 
'__loader__', '__name__', '__package__', '__spec__', '_shorten']
>>> _shorten('abc', 1, 1)
'abc'
>>> _shorten('abcdefghijklmnopqrw', 2, 2)
'ab[15 chars]rw'

* Does not at all work in CommandPrompt, even after supposedly changing 
to a utf-8 codepage with 'chcp 65000'.

--
Terry Jan Reedy
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NSGBCZQ2R6G2HGPAID4ZI35YCRMF7ERC/
Code of Conduct: http://python.org/psf/codeofconduct/