[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)

[Jim J. Jewett]

Stephen J. Turnbull wrote:
> Jim J. Jewett writes:
> > At the time, we considered it, and we also considered a narrower
> > restriction on using multiple scripts in the same identifier, or at
> > least the same identifier portion (so it was OK if separated by
> > _).

> > This would ban "παν語", aka "pango".  That's arguably a good idea
> (IMO, 0.9 wink), but might make some GTK/GNOME folks sad.

I am not quite motivated enough to search the archives, but I'm pretty sure the 
examples actually found were less prominent than that.  There seemed to be at 
least one or two fora where it was something of a local idiom.

>... I don't recall ever seeing
> an identifier with ASCII and Japanese glommed together without a
> separator.  It was almost always of the form "English verb - Japanese
> lexical component". 

The problem was that some were written without a "-" or "_" to separate the 
halves.  It looked fine -- the script change was obvious to even someone who 
didn't speak the non-English language.  But having to support that meant any 
remaining restriction on mixed scripts would be either too weak to be 
worthwhile, or too complicated to write into the python language specification.

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CUTMZG55WY3CLNNKB6VTPCOUXJ22EEZY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] 3.9.8 and 3.11a2 temporarily on hold due to Tcl/Tk

[Łukasz Langa]

As you might have noticed, 3.9.8 was scheduled for release on Monday. This 
didn't happen yet.

There's a bunch of ongoing work fixing Tcl/Tk problems. macOS Monterey got 
released with a new incompatible Tcl/Tk version, some fixes were required for 
tkinter compatibility. Details in https://bugs.python.org/issue44828 
. This is now a fixed issue (thanks, Ned!) 
but we're going through some more ttk fixes discovered in failing buildbots, 
which were sadly unrelated to the Tcl/Tk version bump. In particular, there's a 
ttk refleak that was discovered by the Windows 8.1 refleak buildbot which 
turned out to be a widespread issue (other refleak buildbots were simply 
skipping GUI tests). Example failure: 
https://buildbot.python.org/all/#/builders/6/builds/168 


I'm fixing the macOS ttk test failures and the refleak now to unblock the 3.9.8 
release. I should be done with this tomorrow. We'll go ahead with the release 
then. Pablo decided to wait for me with 3.11a2 since the problems are fixing 
are also happening in `main` (and 3.10).

We'll keep you posted, cheers!

--
Łukasz Langa
CPython Developer in Residence
Python Software Foundation



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TRTOK4ZNLFOL6TAYKM3FE7MMYXMOK6CL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 663:

[Brett Cannon]

Rendered versions can be found at the following links:

https://www.python.org/dev/peps/pep-0663/
https://python.github.io/peps/pep-0663/

On Tue, Nov 2, 2021 at 8:41 PM Ethan Furman  wrote:

> See the latest changes, which are mostly a (hopefully) improved abstract,
> better tables, and some slight rewordings.
>
> Feedback welcome!
>
> ---
>
> PEP: 663
> Title: Standardizing Enum str(), repr(), and format() behaviors
> Version: $Revision$
> Last-Modified: $Date$
> Author: Ethan Furman 
> Discussions-To: python-dev@python.org
> Status: Draft
> Type: Informational
> Content-Type: text/x-rst
> Created: 23-Feb-2013
> Python-Version: 3.11
> Post-History: 20-Jul-2021, 02-Nov-2021
> Resolution:
>
>
> Abstract
> 
>
> Update the ``repr()``, ``str()``, and ``format()`` of the various Enum
> types
> to better match their intended purpose.  For example, ``IntEnum`` will have
> its ``str()`` change to match its ``format()``, while a user-mixed int-enum
> will have its ``format()`` match its ``str()``.  In all cases, an enum's
> ``str()`` and ``format()`` will be the same (unless the user overrides
> ``format()``).
>
> Add a global enum decorator which changes the ``str()`` and ``repr()``
> (and
> ``format()``) of the decorated enum to be a valid global reference: i.e.
> ``re.IGNORECASE`` instead of .
>
>
> Motivation
> ==
>
> Having the ``str()`` of ``IntEnum`` and ``IntFlag`` not be the value causes
> bugs and extra work when replacing existing constants.
>
> Having the ``str()`` and ``format()`` of an enum member be different can be
> confusing.
>
> The addition of ``StrEnum`` with its requirement to have its ``str()`` be
> its
> ``value`` is inconsistent with other provided Enum's ``str``.
>
> The iteration of ``Flag`` members, which directly affects their
> ``repr()``, is
> inelegant at best, and buggy at worst.
>
>
> Rationale
> =
>
> Enums are becoming more common in the standard library; being able to
> recognize
> enum members by their ``repr()``, and having that ``repr()`` be easy to
> parse, is
> useful and can save time and effort in understanding and debugging code.
>
> However, the enums with mixed-in data types (``IntEnum``, ``IntFlag``, and
> the new
> ``StrEnum``) need to be more backwards compatible with the constants they
> are
> replacing -- specifically, ``str(replacement_enum_member) ==
> str(original_constant)``
> should be true (and the same for ``format()``).
>
> IntEnum, IntFlag, and StrEnum should be as close to a drop-in replacement
> of
> existing integer and string constants as is possible.  Towards that goal,
> the
> ``str()`` output of each should be its inherent value; e.g. if ``Color``
> is an
> ``IntEnum``::
>
>  >>> Color.RED
>  
>  >>> str(Color.RED)
>  '1'
>  >>> format(Color.RED)
>  '1'
>
> Note that ``format()`` already produces the correct output, only ``str()``
> needs
> updating.
>
> As much as possible, the ``str()``, ``repr()``, and ``format()`` of enum
> members
> should be standardized across the standard library.  However, up to Python
> 3.10
> several enums in the standard library have a custom ``str()`` and/or
> ``repr()``.
>
> The ``repr()`` of Flag currently includes aliases, which it should not;
> fixing that
> will, of course, already change its ``repr()`` in certain cases.
>
>
> Specification
> =
>
> There a three broad categories of enum usage:
>
> - simple: ``Enum`` or ``Flag``
>a new enum class is created with no data type mixins
>
> - drop-in replacement: ``IntEnum``, ``IntFlag``, ``StrEnum``
>a new enum class is created which also subclasses ``int`` or ``str``
> and uses
>``int.__str__`` or ``str.__str__``
>
> - user-mixed enums and flags
>the user creates their own integer-, float-, str-, whatever-enums
> instead of
>using enum.IntEnum, etc.
>
> There are also two styles:
>
> - normal: the enumeration members remain in their classes and are accessed
> as
>``classname.membername``, and the class name shows in their ``repr()``
> and
>``str()`` (where appropriate)
>
> - global: the enumeration members are copied into their module's global
>namespace, and their module name shows in their ``repr()`` and ``str()``
>(where appropriate)
>
> Some sample enums::
>
>  # module: tools.py
>
>  class Hue(Enum):  # or IntEnum
>  LIGHT = -1
>  NORMAL = 0
>  DARK = +1
>
>  class Color(Flag):  # or IntFlag
>  RED = 1
>  GREEN = 2
>  BLUE = 4
>
>  class Grey(int, Enum):  # or (int, Flag)
> BLACK = 0
> WHITE = 1
>
> Using the above enumerations, the following two tables show the old and new
> output (blank cells indicate no change):
>
>
> +++-++---+
> | style  | category   | enum repr() | enum str() | enum
> format() |
>
> 

[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)

[Chris Jerdonek]

On Tue, Nov 2, 2021 at 7:21 AM Petr Viktorin  wrote:

> That brings us to possible changes in Python in this  area, which is an
> interesting topic.


Is there a use case or need for allowing the comment-starting character “#”
to occur when text is still in the right-to-left direction? Disallowing
that would prevent Petr’s examples in which active code is displayed after
the comment mark, which to me seems to be one of the more egregious
examples. Or maybe this case is no worse than others and isn’t worth
singling out.

—Chris




>
> As for \0, can we ban all ASCII & C1 control characters except
> whitespace? I see no place for them in source code.
>
>
> For homoglyphs/confusables, should there be a SyntaxWarning when an
> identifier looks like ASCII but isn't?
>
> For right-to-left text: does anyone actually name identifiers in
> Hebrew/Arabic? AFAIK, we should allow a few non-printing
> "joiner"/"non-joiner" characters to make it possible to use all Arabic
> words. But it would be great to consult with users/teachers of the
> languages.
> Should Python run the bidi algorithm when parsing and disallow reordered
> tokens? Maybe optionally?
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/TGB377QWGIDPUWMAJSZLT22ERGPNZ5FZ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3OTEOUYXN6H7BCLFVJ4QV3PL5RPIXFNL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Serhiy Storchaka]

03.11.21 15:14, Stephen J. Turnbull пише:
> So the only
> time that wouldn't be true is if escape sequences are allowed to
> represent characters.  I believe unicode_escape is the only codec
> that does.

Also raw_unicode_escape and utf_7. And maybe punycode or idna, I am not
sure.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XRORKXTTV55YOSMP7Z7MAL4AG2UQRXHK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)

[Serhiy Storchaka]

03.11.21 14:31, Petr Viktorin пише:
> For example: should the parser emit a lightweight audit event if it
> finds a non-ASCII identifier? (See below for why ASCII is special.)
> Or for encoding declarations?

There are audit events for import and compile. You can also register
import hooks if you want more fanny preprocessing than just
unicode-encoding. I do not think we need to add more specific audit
events, they were not designed for this.

And I think it is too late to detect suspicious code at the time of its
execution. It should be detected before adding that code to the code
base (review tools, pre-commit hooks).

> I don't think this would actually ban Cyrillic/Greek.
> (My suggestion is not vanilla confusables detection; it might require
> careful reading: "should there be a [linter] warning when an identifier
> looks like ASCII but isn't?")

Yes, but it should be optional and configurable and not be the part of
the Python compiler. This is not our business as Python core developers.

> I am not a native speaker, but I did try a bit to find an actual
> ASCII-like word in a language that uses Cyrillic. I didn't succeed; I
> think they might be very rare.

With simple script I have found 62 words common between English and
Ukrainian: гасу/racy, горе/rope, рима/puma, міх/mix, etc. But there are
much more English and Ukrainian words which contains only letters which
can be confused with letters from other script. And identifiers can
contains abbreviations and shortening, they are not all can be found in
dictionaries.

> Even if there was such a word -- or a one-letter abbreviation used as a
> variable name -- it would be confusing to use. Removing the possibility
> of confusion could *help* Cyrillic users. (I can't speak for them; this
> is just a brainstorming idea.)

I never used non-Latin identifiers in Python, but I guess that where
they are used (in schools?) there is a mix of English and non-English
identifiers, and identifiers consisting of parts of English and
non-English words without even an underscore between them. I know
because in other languages they just use inconsistent transliteration.
Emitting any warning by default is a discrimination of non-English
users. It would be better to not add support of non-ASCII identifiers at
first place.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XHHXRWGKTDTZIYGS6AB3DKEVFH5D6BHV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Stephen J. Turnbull]

Chris Angelico writes:

 > Ah, okay, so much for that, then. What about the weaker sense:
 > Characters below 128 are always and only represented by those byte
 > values? So if you find byte value 39, it might not actually be an
 > apostrophe, but if you're looking for an apostrophe, you know for sure
 > that it'll be represented by byte value 39?

1.  The apostrophe that Python considers a string delimiter is always
represented by byte value 39 in the compiler input.  So the only
time that wouldn't be true is if escape sequences are allowed to
represent characters.  I believe unicode_escape is the only codec
that does.

2.  There's always eval which will accept a string containing escape
sequences.

 > Yes. I'm sure someone will come along and say "but I have to have an
 > all-ASCII source file, directly runnable, with non-ASCII variable
 > names", because XKCD 1172, but I don't have enough sympathy for that
 > obscure situation to want the mess that unicode_escape can give.

It's not an obscure situation to me.  As I wrote earlier, been there,
done that, made my own T-shirt.  I don't *think* it matters today, but
the number of DOS machines and Windows 98 machines left in Japan is
not zero.  Probably they can't run Python 3, but that's not something
I can testify to.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RNCM3QNGBRRM5GW6SL3Q6FP6R55F5CHU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Petr Viktorin]

On 03. 11. 21 12:33, Serhiy Storchaka wrote:

03.11.21 12:36, Petr Viktorin пише:

On 03. 11. 21 2:58, Kyle Stanley wrote:

I'd suggest both: briefer, easier to read write up for average user in
docs, more details/semantics in informational PEP. Thanks for working
on this, Petr!


Well, this is the brief write-up :)
Maybe it would work better if the  info was integrated into the relevant
parts of the docs, rather than be a separate HOWTO.

I went with an informational PEP because it's quicker to publish.


What is the supposed target audience of this document?


Good question! At this point it looks like it's linter authors.


If it is core
Python developers only, then PEP is the right place to publish it. But I
think that it rather describes potential issues in arbitrary Python
project, and as such, it will be more accessible as a part of the Python
documentation (as a HOW-TO article perhaps). AFAIK all other
informational PEPs are about developing Python, not developing in Python
(even if they are (mis)used (e.g. PEP 8) outside their scope).


There's a bunch of packaging PEPs, or a PEP on what the the 
/usr/bin/python command should be. I think PEP 672 is in good company 
for now.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UTNIZZVWL56G7KSYSS67PYYZ2YPE7NX3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Petr Viktorin]

On 03. 11. 21 12:37, Chris Angelico wrote:

On Wed, Nov 3, 2021 at 10:22 PM Steven D'Aprano  wrote:


On Wed, Nov 03, 2021 at 11:21:53AM +1100, Chris Angelico wrote:


TBH, I'm not entirely sure how valid it is to talk about *security*
considerations when we're dealing with Python source code and variable
confusions, but that's a term that is well understood.


It's not like Unicode is the only way to write obfuscated code,
malicious or otherwise.



But to the extent that it is a security concern, it's not one that
linters can really cope with. I'm not sure how a linter would stop
someone from publishing code on PyPI that causes confusion by its
character encoding, for instance.


Do we require that PyPI prevents people from publishing code that causes
confusion by its poorly written code and obfuscated and confusing
identifiers?

The linter is to *flag the issue* during, say, code review or before
running the code, like other code quality issues.

If you're just running random code you downloaded from the internet
using pip, then Unicode confusables are the least of your worries.

I'm not really sure why people get so uptight about Unicode confusables,
while being blasé about the opportunities to smuggle malicious code into
pure ASCII code.



Right, which is why I was NOT talking about confusables. I don't
consider them to be a particularly Unicode-related threat, although
the larger range of available characters does make it more plausible
than in ASCII.

But I do see a problem with code where most editors misrepresent the
code, where abuse of a purely ASCII character encoding for purely
ASCII code can cause all kinds of tooling issues. THAT is a more
viable attack vector, since code reviewers will be likely to assume
that their syntax highlighting is correct.

And yes, I'm aware that Python can't be expected to cope with poor
tools, but when *many* well-known tools have the same problem, one
must wonder who should be solving the issue.


This is a very good point. Let's not point fingers, but figure out how 
to make users' lives easier together :)



This was the first time I was "in" on an embargoed "issue", and let me 
tell you, I was surprised by the amount of time spent on polishing the 
messaging. Now, you can't reasonably twist all this into a "Python is 
insecure" or "Company X products are insecure" headline, which is good, 
but with that out of the way we can focus on *what* could be improved 
over *where* the improvement could be and who should do it.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FNUZCNDF7K2LLHRYRDYY3ZZYISRCI4XJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)

[Petr Viktorin]

We seem to agree that this is work for linters. That's reasonable; I'd 
generalize it to "tools and policies". But even so, discussing what we'd 
expect linters to do is on topic here.
Perhaps we can even find ways for the language to support linters -- 
type checking is also for external tools, but has language support.


For example: should the parser emit a lightweight audit event if it 
finds a non-ASCII identifier? (See below for why ASCII is special.)

Or for encoding declarations?

On 03. 11. 21 6:26, Stephen J. Turnbull wrote:

Serhiy Storchaka writes:

  > All control characters except CR, LF, TAB and FF are banned outside
  > comments and string literals. I think it is worth to ban them in
  > comments and string literals too.

+1

  > > For homoglyphs/confusables, should there be a SyntaxWarning when an
  > > identifier looks like ASCII but isn't?
  >
  > It would virtually ban Cyrillic.

+1 (for the comment and for the implied -1 on SyntaxWarning, let's
keep the Cyrillic repertoire in Python!)


I don't think this would actually ban Cyrillic/Greek.
(My suggestion is not vanilla confusables detection; it might require 
careful reading: "should there be a [linter] warning when an identifier 
looks like ASCII but isn't?")


I am not a native speaker, but I did try a bit to find an actual 
ASCII-like word in a language that uses Cyrillic. I didn't succeed; I 
think they might be very rare.
Even if there was such a word -- or a one-letter abbreviation used as a 
variable name -- it would be confusing to use. Removing the possibility 
of confusion could *help* Cyrillic users. (I can't speak for them; this 
is just a brainstorming idea.)


Steven adds:
Let's not enshrine as a language "feature" that non Western European 
languages are dangerous second-class citizens.


That would be going too far, yes, but the fact is that non-English 
languages *are* second-class citizens. Code that uses Python keywords 
and stdlib must use English, and possibly another language. It is the 
mixing of languages that can be dangerous/confusing, not the languages 
themselves.





  > It is a work for linters,

+1

Aside from the reasons Serhiy presents, I'd rather not tie
this kind of rather ambiguous improvement in Unicode handling to the
release cycle.

It might be worth having a pep module/script in Python (perhaps
more likely, PyPI but maintained by whoever does the work to make
these improvements + Petr or somebody Petr trusts to do it), that
lints scripts specifically for confusables and other issues.


If I have any say in it, the name definitely won't include a PEP number ;)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LB4O3YVDNVVNLYPMNH236QXGGUYG4BVI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Chris Angelico]

On Wed, Nov 3, 2021 at 10:22 PM Steven D'Aprano  wrote:
>
> On Wed, Nov 03, 2021 at 11:21:53AM +1100, Chris Angelico wrote:
>
> > TBH, I'm not entirely sure how valid it is to talk about *security*
> > considerations when we're dealing with Python source code and variable
> > confusions, but that's a term that is well understood.
>
> It's not like Unicode is the only way to write obfuscated code,
> malicious or otherwise.
>
>
> > But to the extent that it is a security concern, it's not one that
> > linters can really cope with. I'm not sure how a linter would stop
> > someone from publishing code on PyPI that causes confusion by its
> > character encoding, for instance.
>
> Do we require that PyPI prevents people from publishing code that causes
> confusion by its poorly written code and obfuscated and confusing
> identifiers?
>
> The linter is to *flag the issue* during, say, code review or before
> running the code, like other code quality issues.
>
> If you're just running random code you downloaded from the internet
> using pip, then Unicode confusables are the least of your worries.
>
> I'm not really sure why people get so uptight about Unicode confusables,
> while being blasé about the opportunities to smuggle malicious code into
> pure ASCII code.
>

Right, which is why I was NOT talking about confusables. I don't
consider them to be a particularly Unicode-related threat, although
the larger range of available characters does make it more plausible
than in ASCII.

But I do see a problem with code where most editors misrepresent the
code, where abuse of a purely ASCII character encoding for purely
ASCII code can cause all kinds of tooling issues. THAT is a more
viable attack vector, since code reviewers will be likely to assume
that their syntax highlighting is correct.

And yes, I'm aware that Python can't be expected to cope with poor
tools, but when *many* well-known tools have the same problem, one
must wonder who should be solving the issue.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WMSJLYG5YQ7SMNHXKSXNEMM7UKKIARCN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Serhiy Storchaka]

03.11.21 12:36, Petr Viktorin пише:
> On 03. 11. 21 2:58, Kyle Stanley wrote:
>> I'd suggest both: briefer, easier to read write up for average user in
>> docs, more details/semantics in informational PEP. Thanks for working
>> on this, Petr!
> 
> Well, this is the brief write-up :)
> Maybe it would work better if the  info was integrated into the relevant
> parts of the docs, rather than be a separate HOWTO.
> 
> I went with an informational PEP because it's quicker to publish.

What is the supposed target audience of this document? If it is core
Python developers only, then PEP is the right place to publish it. But I
think that it rather describes potential issues in arbitrary Python
project, and as such, it will be more accessible as a part of the Python
documentation (as a HOW-TO article perhaps). AFAIK all other
informational PEPs are about developing Python, not developing in Python
(even if they are (mis)used (e.g. PEP 8) outside their scope).

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TM7EU4QHHXTJMXGQT2EJRKZYZ764HNAD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Steven D'Aprano]

On Wed, Nov 03, 2021 at 11:11:00AM +0100, Marc-Andre Lemburg wrote:

> Coming back to the thread topic, many of the Unicode security
> considerations don't apply to non-Unicode encodings, since those
> usually don't support e.g. changing the bidi direction within a
> stream of text or other interesting features you have in Unicode
> such as combining code points, invisible (space) code points, font
> rendering hint code points, etc.
> 
> So in a sense, those non-Unicode encodings are safer than
> using UTF-8 :-)

Thank you MAL for that timely reminder that most encodings are not 
Unicode. I have to admit that I often forget that there is a whole 
universe of non-Unicode, non-ASCII encodings.


> Please also note that most character lookalikes are not encoding
> issues, but instead font issues, which then result in the characters
> looking similar.

+1


-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NJFO5C7367F4NLLQTJRNNNUCRRLA6BES/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Steven D'Aprano]

On Wed, Nov 03, 2021 at 11:21:53AM +1100, Chris Angelico wrote:

> TBH, I'm not entirely sure how valid it is to talk about *security*
> considerations when we're dealing with Python source code and variable
> confusions, but that's a term that is well understood.

It's not like Unicode is the only way to write obfuscated code, 
malicious or otherwise.


> But to the extent that it is a security concern, it's not one that
> linters can really cope with. I'm not sure how a linter would stop
> someone from publishing code on PyPI that causes confusion by its
> character encoding, for instance.

Do we require that PyPI prevents people from publishing code that causes 
confusion by its poorly written code and obfuscated and confusing 
identifiers?

The linter is to *flag the issue* during, say, code review or before 
running the code, like other code quality issues.

If you're just running random code you downloaded from the internet 
using pip, then Unicode confusables are the least of your worries.

I'm not really sure why people get so uptight about Unicode confusables, 
while being blasé about the opportunities to smuggle malicious code into 
pure ASCII code.

https://en.wikipedia.org/wiki/Underhanded_C_Contest

Is it unfamiliarity? Worse? "Real programmers write identifiers in 
English." And the ironic thing is, while it is very difficult indeed for 
automated checkers to detect underhanded code in ASCII, it is trivially 
easier for editors, linters and other tools to spot the sort of Unicode 
confusables we're talking about here. But we spend all our energy 
worrying about the minor issue, and almost none on the broader problem 
of malicious code in general.

I'm pretty sure I could upload a library to PyPI that included

os.system('rm -rf .')

and nobody would blink an eye, but if I write:

A = 1
А = 2
Α = 3
print(A, А, Α)

everyone goes insane. Let's keep the threat in perspective. Writing an 
informational PEP for the education of people is a great idea. Rushing 
into making wholesale changes to the interpreter, not so much.


-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YGPSWZL4Z7LKTUHC25JVMHA5LUSLLQEL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)

[Steven D'Aprano]

On Tue, Nov 02, 2021 at 05:55:55PM +0200, Serhiy Storchaka wrote:

> All control characters except CR, LF, TAB and FF are banned outside
> comments and string literals. I think it is worth to ban them in
> comments and string literals too. In string literals you can use
> backslash-escape sequences, and comments should be human readable, there
> are no reason to include control characters in them. There is a
> precedence of emitting warnings for some superficial escapes in strings.

Agreed. I don't think there is any good reason for including control 
characters (apart from whitespace) in comments.

In strings, I would consider allowing VT (vertical tab) as well, that is 
whitespace.

>>> '\v'.isspace()
True

But I don't have a strong opinion on that.


[Petr]
> > For homoglyphs/confusables, should there be a SyntaxWarning when an
> > identifier looks like ASCII but isn't?

Let's not enshrine as a language "feature" that non Western European 
languages are dangerous second-class citizens.


> It would virtually ban Cyrillic. There is a lot of Cyrillic letters
> which look like Latin letters, and there are complete words written in
> Cyrillic which by accident look like other words written in Latin.

Agreed.


> It is a work for linters, which can have many options for configuring
> acceptable scripts, use spelling dictionaries and dictionaries of
> homoglyphs, etc.

Linters and editors. I have no objection to people using editors that 
highlight non-ASCII characters in blinking red letters, so long as I can 
turn that option off :-)



-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RWE5FIWHUM5PSOJ6BI2PAO5TDE3KLC5D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Petr Viktorin]

On 03. 11. 21 2:58, Kyle Stanley wrote:
I'd suggest both: briefer, easier to read write up for average user in 
docs, more details/semantics in informational PEP. Thanks for working on 
this, Petr!


Well, this is the brief write-up :)
Maybe it would work better if the  info was integrated into the relevant 
parts of the docs, rather than be a separate HOWTO.


I went with an informational PEP because it's quicker to publish.



On Tue, Nov 2, 2021 at 2:07 PM David Mertz, Ph.D. > wrote:


This is an amazing document, Petr. Really great work!

I think I agree with Marc-André that putting it in the actual Python
documentation would give it more visibility than in a PEP.

On Tue, Nov 2, 2021, 1:06 PM Marc-Andre Lemburg mailto:m...@egenix.com>> wrote:

On 01.11.2021 13:17, Petr Viktorin wrote:
 >> PEP: 
 >> Title: Unicode Security Considerations for Python
 >> Author: Petr Viktorin mailto:encu...@gmail.com>>
 >> Status: Active
 >> Type: Informational
 >> Content-Type: text/x-rst
 >> Created: 01-Nov-2021
 >> Post-History:

Thanks for writing this up. I'm not sure whether a PEP is the
right place
for such documentation, though. Wouldn't it be more visible in
the standard
Python documentation ?

-- 
Marc-Andre Lemburg

eGenix.com

Professional Python Services directly from the Experts (#1, Nov
02 2021)
 >>> Python Projects, Coaching and Support ...
https://www.egenix.com/ 
 >>> Python Product Development ...
https://consulting.egenix.com/ 


::: We implement business ideas - efficiently in both time and
costs :::

    eGenix.com Software, Skills and Services GmbH 
Pastor-Loeh-Str.48

     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
            Registered at Amtsgericht Duesseldorf: HRB 46611
https://www.egenix.com/company/contact/

https://www.malemburg.com/ 

___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/FSFG2B3LCWU5PQWX3WRIOJGNV2JFW4AU/


Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/6PHPDZRCYNA44NHSHXPBL7QMWXMHXWGU/


Code of Conduct: http://python.org/psf/codeofconduct/



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6OET4CKEZIA34PAXIJR7BUDKT2DPX2DG/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZNANXZ7VP6CVDAGWEFXHKYFO6AR3MZXQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Paul Moore]

On Wed, 3 Nov 2021 at 10:11, Marc-Andre Lemburg  wrote:
> I don't think limiting the source code encoding is the right approach
> to making code more secure. Instead, tooling has to be used to detect
> potentially malicious code points in code.

+1

Discussing "making code more secure" without being clear on what the
threat model is, is always going to be inconclusive. In this case, I
believe the threat model is "an untrusted 3rd party submitting a PR
which potentially contains malicious code to a Python project". For
that threat, I think the correct approach is for core Python to
promote awareness (via this PEP and maybe something in the docs
themselves) and for projects to implement appropriate code checks that
are run against all PRs to flag this sort of issue.

What threat can't be addressed at a per-project level, but *can* be
addressed in core Python (without triggering so many false positives
that people are trained to ignore the warnings or work around the
prohibitions, defeating the purpose of the change)?

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FQ42C66BVCE6AQFSP4J6V6ERS4VV44MK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Marc-Andre Lemburg]

On 03.11.2021 01:21, Chris Angelico wrote:
> On Wed, Nov 3, 2021 at 11:09 AM Steven D'Aprano  wrote:
>>
>> On Wed, Nov 03, 2021 at 03:03:54AM +1100, Chris Angelico wrote:
>>> On Wed, Nov 3, 2021 at 1:06 AM Petr Viktorin  wrote:
 Let me know if it's clear in the newest version, with this note:

> Here, ``encoding: unicode_escape`` in the initial comment is an encoding
> declaration. The ``unicode_escape`` encoding instructs Python to treat
> ``\u0027`` as a single quote (which can start/end a string), ``\u002c`` as
> a comma (punctuator), etc.

>>>
>>> Huh. Is that level of generality actually still needed? Can Python
>>> deprecate all but a small handful of encodings?
>>
>> To be clear, are you proposing to deprecate the encodings *completely*
>> or just as the source code encoding?
> 
> Only source code encodings. Obviously we still need to be able to cope
> with all manner of *data*, but Python source code shouldn't need to be
> in bizarre, weird encodings.
> 
> (Honestly, I'd love to just require that Python source code be UTF-8,
> but that would probably cause problems, so mandating that it be one of
> a small set of encodings would be a safer option.)

Most Python code will be written in UTF-8 going forward, but there's
still a lot of code out there in other encodings. Limiting this
to some reduced set doesn't really make sense, since it's not
clear where to draw the line.

Coming back to the thread topic, many of the Unicode security
considerations don't apply to non-Unicode encodings, since those
usually don't support e.g. changing the bidi direction within a
stream of text or other interesting features you have in Unicode
such as combining code points, invisible (space) code points, font
rendering hint code points, etc.

So in a sense, those non-Unicode encodings are safer than
using UTF-8 :-)

Please also note that most character lookalikes are not encoding
issues, but instead font issues, which then result in the characters
looking similar.

There are fonts which are designed to avoid this
and it's no surprise that source code fonts typically do make
e.g. 0 and O, as well as 1 and l look sufficiently different to be
able to notice the difference.

Things get a lot harder when dealing with combining characters, since
it's not always easy to spot the added diacritics, e.g. try
this:

>>> print ('a\u0348bc') # strong articulation
a͈bc
>>> print ('a\u034Fbc') # combining grapheme joiner
a͏bc

The latter is only "visible" in the unicode_escape encoding:

>>> print ('a\u034Fbc'.encode('unicode_escape'))
b'a\\u034fbc'

Projects wanting to limit code encoding settings, disallow using
bidi markers and other special code points in source code, can easily
do this via e.g. pre-commit hooks, special editor settings, code
linters or security scanners.

I don't think limiting the source code encoding is the right approach
to making code more secure. Instead, tooling has to be used to detect
potentially malicious code points in code.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 03 2021)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MBWBY47ILPL3E6733W4XAZXF2M6RKFH6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Chris Angelico]

On Wed, Nov 3, 2021 at 8:01 PM Stephen J. Turnbull
 wrote:
>
> Chris Angelico writes:
>
>  > But I was surprised to find that Python would let you use
>  > unicode_escape for source code.
>
> I'm not surprised.  Today it's probably not necessary, but I've
> exchanged a lot of code (not Python, though) with folks whose editors
> were limited to 8 bit codes or even just ASCII.  It wasn't frequent
> that I needed to discuss non-ASCII code with them (that they needed to
> run) but it would have been painful to do without some form of codec
> that encoded Japanese using only ASCII bytes.

Bearing in mind that string literals can always have their own
escapes, this feature is really only important to the source code
tokens themselves.

>  > Maybe the phrase "a small handful" was a bit too hopeful, but would it
>  > be possible to mandate (after, obviously, a deprecation period) that
>  > source encodings be ASCII-compatible?
>
> Not sure what you mean there.  In the usual sense of ASCII-compatible
> (the ASCII bytes always mean the corresponding character in the ASCII
> encoding), I think there are at least two ASCII-incompatible encodings
> that would cause a lot of pain if they were prohibited, specifically
> Shift JIS and Big5.  (In certain contexts in those encodings an ASCII
> byte frequently is a trailing byte in a multibyte character.)

Ah, okay, so much for that, then. What about the weaker sense:
Characters below 128 are always and only represented by those byte
values? So if you find byte value 39, it might not actually be an
apostrophe, but if you're looking for an apostrophe, you know for sure
that it'll be represented by byte value 39?

> It might make sense to prohibit unicode_escape nowadays -- I think
> almost all systems now can handle Unicode properly, but I don't think
> we can go farther than that.
>

Yes. I'm sure someone will come along and say "but I have to have an
all-ASCII source file, directly runnable, with non-ASCII variable
names", because XKCD 1172, but I don't have enough sympathy for that
obscure situation to want the mess that unicode_escape can give.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4JUWQJRMPCSPY3CCJCXLJKBVZ2UFW56F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Serhiy Storchaka]

03.11.21 11:01, Stephen J. Turnbull пише:
>  And of
> course UTF-16 is incompatible in that sense, although I don't know if
> anybody actually saves Python code in UTF-16.

CPython does not currently support UTF-16 for source files.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KN4MPLKSRKQOJM2DUFQNO4UGGOJN5YNU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Stephen J. Turnbull]

Chris Angelico writes:

 > But I was surprised to find that Python would let you use
 > unicode_escape for source code.

I'm not surprised.  Today it's probably not necessary, but I've
exchanged a lot of code (not Python, though) with folks whose editors
were limited to 8 bit codes or even just ASCII.  It wasn't frequent
that I needed to discuss non-ASCII code with them (that they needed to
run) but it would have been painful to do without some form of codec
that encoded Japanese using only ASCII bytes.

 > Maybe the phrase "a small handful" was a bit too hopeful, but would it
 > be possible to mandate (after, obviously, a deprecation period) that
 > source encodings be ASCII-compatible?

Not sure what you mean there.  In the usual sense of ASCII-compatible
(the ASCII bytes always mean the corresponding character in the ASCII
encoding), I think there are at least two ASCII-incompatible encodings
that would cause a lot of pain if they were prohibited, specifically
Shift JIS and Big5.  (In certain contexts in those encodings an ASCII
byte frequently is a trailing byte in a multibyte character.)  I'm sure
there is a ton of legacy Python code in those encodings in East Asia,
some of which is still maintained in the original encoding.  And of
course UTF-16 is incompatible in that sense, although I don't know if
anybody actually saves Python code in UTF-16.

It might make sense to prohibit unicode_escape nowadays -- I think
almost all systems now can handle Unicode properly, but I don't think
we can go farther than that.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ESIU62AXASWUDX7MSPMTFIDONIAI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)

[Serhiy Storchaka]

02.11.21 18:49, Jim J. Jewett пише:
> If escape sequences were also allowed in comments (or at least in strings 
> within comments), this would make sense.  I don't like banning them 
> otherwise, since odd characters are often a good reason to need a comment, 
> but it is definitely a "mention, not use" situation.

If you mean backslash-escaped sequences like \u, there is no reason
to ban them in comments. Unlike to Java they do not have special meaning
outside of string literals. But if you mean terminal control sequences
(which change color or move cursor) they should not be allowed in comments.

> At the time, we considered it, and we also considered a narrower restriction 
> on using multiple scripts in the same identifier, or at least the same 
> identifier portion (so it was OK if separated by _).

I implemented this restrictions in one of my projects. The character set
was limited, and even this did not solve all issues with homoglyphs.

I think that we should not introduce such arbitrary limitations at the
parser level and left it to linters.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2TT3VX4D4FMRSEOV4O2ICYTC7VC5M2J4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Chris Angelico]

On Wed, Nov 3, 2021 at 5:12 PM Stephen J. Turnbull
 wrote:
>
> Chris Angelico writes:
>
>  > Huh. Is that level of generality actually still needed? Can Python
>  > deprecate all but a small handful of encodings?
>
> I think that's pointless.  With few exceptions (GB18030, Big5 has a
> couple of code point pairs that encode the same very rare characters,
> ISO 2022 extensions) you're not going to run into the confuseables
> problem, and AFAIK the only generic BIDI solution is Unicode (the ISO
> 8859 encodings of Hebrew and Arabic do not have direction markers).
>
> What exactly are you thinking?

You'll never eliminate confusables (even ASCII has some, depending on
font). But I was surprised to find that Python would let you use
unicode_escape for source code.



# coding: unicode_escape

x = '''

Code example:

\u0027\u0027\u0027 # format in monospaced on the web site

print("Did you think this would be executed?")

\u0027\u0027\u0027 # end monospaced

Surprise!
'''

print("There are %d lines in x." % len(x.split(chr(10



With some carefully-crafted comments, a lot of human readers will
ignore the magic tokens. It's not uncommon to put example code into
triple-quoted strings, and it's also not all that surprising when
simplified examples do things that you wouldn't normally want done
(like monkeypatching other modules), since it's just an example, after
all.

I don't have access to very many editors, but SciTE, VS Code, nano,
and the GitHub gist display all syntax-highlighted this as if it were
a single large string. Only Idle showed it as code in between, and
that's because it actually decoded it using the declared character
coding, so the magic lines showed up with actual apostrophes.

Maybe the phrase "a small handful" was a bit too hopeful, but would it
be possible to mandate (after, obviously, a deprecation period) that
source encodings be ASCII-compatible?

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QQM7HLRMVKBELRRYBJYGR356QVSOLKKZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Stephen J. Turnbull]

Chris Angelico writes:

 > Huh. Is that level of generality actually still needed? Can Python
 > deprecate all but a small handful of encodings?

I think that's pointless.  With few exceptions (GB18030, Big5 has a
couple of code point pairs that encode the same very rare characters,
ISO 2022 extensions) you're not going to run into the confuseables
problem, and AFAIK the only generic BIDI solution is Unicode (the ISO
8859 encodings of Hebrew and Arabic do not have direction markers).

What exactly are you thinking?

The only thing I'd like to see is to rearrange the codec aliases so
that the "common names" would denote the maximal repertoires in each
family (gb denotes gb18030, sjis denotes shift_jisx0213, etc) as in
the WhatWG recommendations for web browsers.  But that's probably too
backward incompatible to fly.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W4RJJVAJN7FB24R52YSCU2Y3QZQE3BEL/
Code of Conduct: http://python.org/psf/codeofconduct/