[Python-Dev] Re: What to do about invalid escape sequences

[Serhiy Storchaka]

08.08.19 07:55, Toshio Kuratomi пише:

Like the Ansible feature, though, the problem is that over time we've
discovered that it is hard to educate users about the exact
characteristic of the feature (\k == k but \n == newline;


No, \k == \\k. This differs from most other programming languages.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7OTMWGJOMXT6F6NONVSL2WLFG3VPP4B6/

[Python-Dev] Re: What to do about invalid escape sequences

[Toshio Kuratomi]

On Mon, Aug 5, 2019 at 6:47 PM  wrote:
>
> I wish people with more product management experience would chime in; 
> otherwise, 3.8 is going to ship with an intentional hard-to-ignore annoyance 
> on the premise that we don't like the way people have been programming and 
> that they need to change their code even if it was working just fine.
>

I was resisting weighing in since I don't know the discussion around
deprecating this language feature in the first place (other than
what's given in this thread).  However, in the product I work on we
made a very similar change in our last release so I'll throw it out
there for people to take what they will from it.

We have a long standing feature which allows people to define groups
of hosts and give them a name.  In the past that name could include
dashes, dots, and other characters which are not legal as Python
identifiers.  When users use those group names in our "DSL" (not truly
a DSL but close enough), they can do it using either dictionary-lookup
syntax (groupvars['groupname']) or using dotted attribute notation
groupvars.groupname.  We also have a longstanding problem where users
will try to do something like groupvars.group-name using the
dotted attribute notation with group names that aren't proper python
identifiers.  This causes problems as the name then gets split on the
characters that aren't legal in identifiers and results in something
unexpected (undefined variable, an actual subtraction operation, etc).
In our last release we decided to deprecate and eventually make it
illegal to use non-python-identifiers for the group names.

At first, product management *did* let us get away with this.  But
after some time and usage of the pre-releases, they came to realize
that this was a major problem.  User's had gotten used to being able
to use these characters in their group names.  They had defined their
group names and gotten used to typing their group names and built up a
whole body of playbooks that used these group names

Product management still let us get away with this.. sort of. The
scope of the change was definitely modified.  Users were now allowed
to select whether invalid group names were disallowed (so they could
port their installations), allowed with a warning (presumably so they
could do work but also see that they were affected) or allow without a
warning (presumably because they knew not to use these group names
with dotted attribute notation) .  This feature was also no longer
allowed to be deprecated... We could have a warning that said "Don't
do this" but not remove the feature in the future.

Now... I said this was a config option So what we do have in the
release is that the config option allows but warns by default and *the
config option* has a deprecation warning.  You see... we're planning
on changing from warn by default now to disallowing by default in the
future so the deprecation is flagging the change in config value.

And you know what?  User's absolutely hate this.  They don't like the
warning.  They don't like the implication that they're doing something
wrong by using a long-standing feature.  They don't like that we're
going to change the default so that they're current group names will
break.  They dislike that it's being warned about because of
attribute-lookup-notation which they can just learn not to use with
their group names.  They dislike this so much that some of us have
talked about abandoning this idea... instead, having a public group
name that users use when they write in the "DSL" and an internal group
name that we use when evaluating the group names. Perhaps that works,
perhaps it doesn't, but I think that's where my story starts being
specific to our feature and no longer applicable to Python and escape
sequences

Now like I said, I don't know the discussions that lead to invalid
escape sequences being deprecated so I don't know whether there's more
compelling reasons for doing it but it seems to me that there's even
less to gain by doing this than what we did in Ansible.  The thing
Ansible is complaining about can do the wrong thing when used in
conjunction with certain other features of our "DSL".  The thing that
the python escape sequences is complaining about are never invalid (As
was pointed out, it's complaining when a sequence of two characters
will do what the user intended rather than complaining when a sequence
of two characters will do something that the user did not intend).
Like the Ansible feature, though, the problem is that over time we've
discovered that it is hard to educate users about the exact
characteristic of the feature (\k == k but \n == newline;
groupvars['group-name']  works but groupvars.group-name does not) so
we've both given up on continuing to educate the users in favor of
attempting to nanny the user into not using the feature.  That most
emphatically has not worked for us and has spent a bunch of goodwill
with our users but the python userbase is not 

[Python-Dev] Re: typing: how to use names in result-tuples?

[Guido van Rossum]

Alas, we didn't think of struct sequences when we designed PEP 484. It
seems they are a hybrid of Tuple and NamedTuple; both of these are
currently special-cased in mypy in ways that cannot easily be combined.

Do you really need anonymous fields?

I see an example in typeshed/stdlib/3/os/__init__.pyi (in
github.com/python/typeshed), for stat_result. It defines names for all the
fields, plus a __getitem__() method that indicates that indexing returns an
int. This doesn't help if anonymous fields could have different types, not
does it teach the type checker about the number of anonymous fields.

--Guido

On Wed, Aug 7, 2019 at 1:51 AM Christian Tismer 
wrote:

> Hi all,
>
> Ok, I am about to implement generation of such structures
> automatically using the struct sequence concept.
>
>
> One more question:
> --
>
> Struct sequences are not yet members of the typing types.
> I would like to add that, because a major use case is also to
> show nice .pyi files with all the functions and types.
>
> * namedtuple has made the transition to NamedTuple
>
> * What would I need to do that for StructSequence as well?
>
> Things get also a bit more complicated since struct sequence
> objects can contain unnamed fields.
>
> Any advice would be appreciated, I am no typing expert (yet :-)
>
> cheers -- Chris
>
>
> On 30.07.19 17:10, Guido van Rossum wrote:
> > I think I have to agree with Petr. Define explicit type names.
> >
> > On Tue, Jul 30, 2019 at 2:45 AM Paul Moore  > > wrote:
> >
> > On Tue, 30 Jul 2019 at 09:33, Christian Tismer  > > wrote:
> > > >>> typing.NamedTuple("__f", x=int, y=int)
> > > 
> > > >>> typing.NamedTuple("__f", x=int, y=int) is
> typing.NamedTuple("__f",
> > > x=int, y=int)
> > > False
> >
> > This appears to go right back to collections.namedtuple:
> >
> > >>> from collections import namedtuple
> > >>> n1 = namedtuple('f', ['a', 'b', 'c'])
> > >>> n2 = namedtuple('f', ['a', 'b', 'c'])
> > >>> n1 is n2
> > False
> >
> > I found that surprising, as I expected the named tuple type to be
> > cached based on the declared name 'f'. But it's been that way forever
> > so obviously my intuition here is wrong. But maybe it would be useful
> > for this case if there *was* a way to base named tuple identity off
> > the name/fields? It could be as simple as caching the results:
> >
> > >>> from functools import lru_cache
> > >>> cached_namedtuple = lru_cache(None)(namedtuple)
> > >>> n1 = cached_namedtuple('f', ('a', 'b', 'c')) # A tuple rather
> > than a list of field names, as lists aren't hashable
> > >>> n2 = cached_namedtuple('f', ('a', 'b', 'c'))
> > >>> n1 is n2
> > True
> >
> > Paul
> >
> >
> >
> > --
> > --Guido van Rossum (python.org/~guido )
> > /Pronouns: he/him/his //(why is my pronoun here?)/
> > <
> http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/
> >
> >
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/GIFRTFWPEGKZ33PTW63YXKGXHHAQJ35I/
> >
>
>
> --
> Christian Tismer :^)   tis...@stackless.com
> Software Consulting  : http://www.stackless.com/
> Karl-Liebknecht-Str. 121 : https://github.com/PySide
> 14482 Potsdam: GPG key -> 0xFB7BEE0E
> phone +49 173 24 18 776  fax +49 (30) 700143-0023
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QZ44KHLLNL54ZCHEAQ5WBWCOV7AU2HGW/

[Python-Dev] Re: What to do about invalid escape sequences

[raymond . hettinger]

This isn't about me.  As a heavy user of the 3.8 beta, I'm just the canary in 
the coal mine.

After many encounters with these warnings, I'm starting to believe that 
Python's long-standing behavior was convenient for users.  Effectively, "\-" 
wasn't an error, it was just a way of writing "\-". For the most part, that 
worked out fine. Sure, we all seen interactive prompt errors from having \t in 
a pathname but not in production (likely because a FileNotFoundError would 
surface immediately).
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4YNZYCOBWGMLC6BDXQFJJWLXEK47I5PU/

[Python-Dev] Re: What to do about invalid escape sequences

[MRAB]

On 2019-08-07 23:43, Steve Holden wrote:
This whole thread would be an excellent justification for following 3.9 
with 4.0. It's as near as we ever want to get to a breaking change, and 
a major version number would indicate the need to review. If increasing 
strictness of escape code interpretation in string literals is the only 
incompatibility there would surely be general delight.



I can think of another possible one: import * requires __all__.

[snip]
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WQEHMFMR7IRWYDXDSCZUGJKGDI5HNEDK/

[Python-Dev] Re: What to do about invalid escape sequences

[Chris Angelico]

On Thu, Aug 8, 2019 at 8:58 AM  wrote:
>
> For me, these warnings are continuing to arise almost daily.  See two recent 
> examples below.  In both cases, the code previously had always worked without 
> complaint.
>
> - Example from yesterday's class 
>
> ''' How old-style formatting works with positional placeholders
>
> print('The answer is %d today, but was %d yesterday' % (new, old))
>  \o
>   \o
> '''
>
> SyntaxWarning: invalid escape sequence \-

I've no idea why this is even a string literal, but if it absolutely
has to be, then you could use a character other than backslash.

> - Example from today's class 
>
> # Cut and pasted from:
> # https://en.wikipedia.org/wiki/VCard#vCard_2.1
> vcard = '''
> ...
> LABEL;WORK;PREF;ENCODING=QUOTED-PRINTABLE;CHARSET=UTF-8:100 Waters Edge=0D=
>  =0ABaytown\, LA 30314=0D=0AUnited States of America
> ...
> '''
>
> SyntaxWarning: invalid escape sequence \,

When you take a text string and create a string literal to represent
it, sometimes you have to modify it to become syntactically valid.
This is exactly the sort of thing that SHOULD be being warned about,
because it's sometimes going to work and sometimes not, depending on
the exact data you're working with. Please don't teach people the
habit of pretending that the backslash isn't significant.

If the warning were changed to be silent for 3.8, what would you do
differently? How would having extra time to solve this problem help
you?

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WCNY3C7VBLCP5RDKKVMMEMN7R26GK2FI/

[Python-Dev] Re: What to do about invalid escape sequences

[raymond . hettinger]

For me, these warnings are continuing to arise almost daily.  See two recent 
examples below.  In both cases, the code previously had always worked without 
complaint.

- Example from yesterday's class 

''' How old-style formatting works with positional placeholders

print('The answer is %d today, but was %d yesterday' % (new, old))
 \o
  \o
'''
   
SyntaxWarning: invalid escape sequence \-

- Example from today's class 

# Cut and pasted from: 
# https://en.wikipedia.org/wiki/VCard#vCard_2.1
vcard = '''
BEGIN:VCARD
VERSION:2.1
N:Gump;Forrest;;Mr.
FN:Forrest Gump
ORG:Bubba Gump Shrimp Co.
TITLE:Shrimp Man
PHOTO;GIF:http://www.example.com/dir_photos/my_photo.gif
TEL;WORK;VOICE:(111) 555-1212
TEL;HOME;VOICE:(404) 555-1212
ADR;WORK;PREF:;;100 Waters Edge;Baytown;LA;30314;United States of America
LABEL;WORK;PREF;ENCODING=QUOTED-PRINTABLE;CHARSET=UTF-8:100 Waters Edge=0D=
 =0ABaytown\, LA 30314=0D=0AUnited States of America
ADR;HOME:;;42 Plantation St.;Baytown;LA;30314;United States of America
LABEL;HOME;ENCODING=QUOTED-PRINTABLE;CHARSET=UTF-8:42 Plantation St.=0D=0A=
 Baytown, LA 30314=0D=0AUnited States of America
EMAIL:forrestg...@example.com
REV:20080424T195243Z
END:VCARD
'''

SyntaxWarning: invalid escape sequence \,
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OYGRL5AWSJZ34MDLGIFTWJXQPLNSK23S/

[Python-Dev] Re: What to do about invalid escape sequences

[Steve Holden]

This whole thread would be an excellent justification for following 3.9
with 4.0. It's as near as we ever want to get to a breaking change, and a
major version number would indicate the need to review. If increasing
strictness of escape code interpretation in string literals is the only
incompatibility there would surely be general delight.

Kind regards,
Steve Holden


On Wed, Aug 7, 2019 at 8:19 PM eryk sun  wrote:

> On 8/7/19, Steve Dower  wrote:
> >
> > * change the PyErr_SetExcFromWindowsErrWithFilenameObjects function to
> > append (or chain) an extra message when either of the filenames contains
> c
> > control characters (or change OSError to do it, or the default
> > sys.excepthook)
>
> On a related note for Windows, if the error is specifically
> ERROR_INVALID_NAME, we could extend this to look for and warn about
> the five reserved wildcard characters (asterisk, question mark, double
> quote, less than, greater than), pipe, and colon. It's only sometimes
> the case for colon because it's allowed in device names and used as
> the name and type delimiter for stream names.
>
> Kernel object names don't reserve wildcard characters, pipe, and
> colon. So I wouldn't want anything but the control-character warning
> if it's say ERROR_FILE_NOT_FOUND. An example would be
> SharedMemory(name='Global\test'), or a similar error for registry key
> and value names such as OpenKey(hkey, 'spam\test'), that is if winreg
> were updated to include the name in the exception. Note that forward
> slash is just a name character in these cases, not a path separator,
> so we have to use backslash, even if just via replace('/', '\\').
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/UFMVFL4QDUXLZFBWVW4YLAKPHQ6LTPDK/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KM2IVRWN5QPLCFHJ5FUWZ6XB7DW2VONS/

[Python-Dev] Re: How to extricate large set of diffs from hg.python.org/sandbox?

[Skip Montanaro]

> I think this might work:
>
> $ hg diff -r fb80df16c4ff -r tip
>
> Not sure fb80df16c4ff is the correct base revision.  It seems to be
> the base of Victor's work.  I put the resulting patch file here:
>
> http://python.ca/nas/python/registervm-victor.txt

Thanks, Neil. I barely remembered anything about Mercurial (not even
installed on my current device). It didn't occur to me that the
necessary precursor to that big diff might be as simple as

hg clone https://hg.python.org/sandbox/registervm/

Skip
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IAMCZ3APJ2LOGWXCH7PQOV67E52KMBZV/

[Python-Dev] Re: What to do about invalid escape sequences

[eryk sun]

On 8/7/19, Steve Dower  wrote:
>
> * change the PyErr_SetExcFromWindowsErrWithFilenameObjects function to
> append (or chain) an extra message when either of the filenames contains c
> control characters (or change OSError to do it, or the default
> sys.excepthook)

On a related note for Windows, if the error is specifically
ERROR_INVALID_NAME, we could extend this to look for and warn about
the five reserved wildcard characters (asterisk, question mark, double
quote, less than, greater than), pipe, and colon. It's only sometimes
the case for colon because it's allowed in device names and used as
the name and type delimiter for stream names.

Kernel object names don't reserve wildcard characters, pipe, and
colon. So I wouldn't want anything but the control-character warning
if it's say ERROR_FILE_NOT_FOUND. An example would be
SharedMemory(name='Global\test'), or a similar error for registry key
and value names such as OpenKey(hkey, 'spam\test'), that is if winreg
were updated to include the name in the exception. Note that forward
slash is just a name character in these cases, not a path separator,
so we have to use backslash, even if just via replace('/', '\\').
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UFMVFL4QDUXLZFBWVW4YLAKPHQ6LTPDK/

[Python-Dev] Re: Can someone please finish merging GH-13482?

[Tim Peters]

[Brett Cannon ]
> We probably need to update https://devguide.python.org/committing/ to
> have a step-by-step list of how to make a merge works and how to
> handle backports instead of the wall of text that we have. (It's already
> outdated anyway, e.g. `Misc/ACKS` really isn't important as git itself
> records the author of the commit and so that can be used with
> `Misc/ACKS` for old commits to gather the list of folks who have"
> contributed.)

Don't put too much weight on my screwups ;-)  I was appalled to hear
that the OP's contribution was still sitting unmerged, and was in a
hurry to resolve that at a time I _had_ no significant time to give to
it.

Mariatta and Terry Reedy finished up what I left undone, so in the end
it's all good :-)

And there's a problem with the GitHub workflow docs that may be unique
to me:  we have helpful layers of automation, but they're shortening a
git workflow I don't understand even without the automation.  With the
automation, I'm just doubly clueless.

That's because I'm old.  My capacity to give a rip about source
control system quirks was apparently entirely used up when I spent
several weeks mastering the intricacies of Mercurial.  Try as I might,
I just haven't been able to force myself to become competent with git.
It's not that I disapprove of git!  It's apparently more that we're
each born with a finite capacity for being _able_ to learn Yet Another
New Source Control System, and mine was used up on YANSCS #8 ;-)

aging-isn't-for-the-optimistic-ly y;rs  - tim
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ACAJVCPHE2N5RNI4BNL2JK6INMJRPZ2N/

[Python-Dev] Re: Can someone please finish merging GH-13482?

[Brett Cannon]

We probably need to update https://devguide.python.org/committing/ to have a 
step-by-step list of how to make a merge works and how to handle backports 
instead of the wall of text that we have. (It's already outdated anyway, e.g. 
`Misc/ACKS` really isn't important as git itself records the author of the 
commit and so that can be used with `Misc/ACKS` for old commits to gather the 
list of folks who have contributed.)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WXJY3B5ZBX7X3IVV3NOPLK6R52E6RL5P/

[Python-Dev] Re: How to extricate large set of diffs from hg.python.org/sandbox?

[Neil Schemenauer]

On 2019-08-07, Skip Montanaro wrote:
> Victor's experiments into a register-based virtual machine live here:
> 
> https://hg.python.org/sandbox/registervm
> 
> I'd like to revive them, if for no other reason to understand what he
> did. I see no obvious way to collect them all as a massive diff.

I think this might work:

$ hg diff -r fb80df16c4ff -r tip

Not sure fb80df16c4ff is the correct base revision.  It seems to be
the base of Victor's work.  I put the resulting patch file here:

http://python.ca/nas/python/registervm-victor.txt

If you are actively working on the register VM idea, shoot me an
email.  I'm interested to collaborate.

Regards,

  Neil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XHB66YDONTOL2ERDMLN4Y6HPIFRMY6BD/

[Python-Dev] Re: What to do about invalid escape sequences

[brian . skinn]

Steven D'Aprano wrote:

> Because our processes don't work the way we assumed, it turns out that 
> in practice we haven't given developers the deprecation period we 
> thought we had. Read Nathaniel's post, if you haven't already done so:
> https://mail.python.org/archives/list/python-dev@python.org/message/E7QCC74O...
> He makes a compelling case that while we might have had the promised 
> deprecation period by the letter of the law, in practice most developers 
> will have never seen it, and we will be breaking the spirit of the 
> promise if we continue with the unmodified plan.
> ...
> I'm sure that the affected devs will understand why it was their fault 
> they couldn't see the warnings, when even people from a first-class 
> library like SymPy took four iterations to do it right.
> > Currently it
> > requires some extra steps or flags, which are not well known. What
> > change are you proposing for 3.8 that will ensure that this actually
> > gets solved?
> > Absolutely nothing. I don't have to: we're an entire community, this 
> doesn't have to fall only on my shoulders. I'm not even the messenger: 
> that's Raymond. I'm just (partly) agreeing with him.
> Just because I don't have a solution for this problem doesn't mean the 
> problem doesn't exist.

As the SymPy team has figured out the right pytest incantation to expose these 
warnings, perhaps a feature request on pytest to encapsulate that mix of 
options into a single flag would be a good idea?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WOJQTOXMYKHLQO4KICEIZH3PDEMQLMBL/

[Python-Dev] Re: What to do about invalid escape sequences

[Steve Dower]

On 07Aug2019 0247, Chris Angelico wrote:

On Wed, Aug 7, 2019 at 7:33 PM Steven D'Aprano  wrote:

What's the rush? Let's be objective here: what benefit are we going to
get from this change? Is there anyone hanging out desperately for "\d"
and "\-" to become SyntaxErrors, so they can... do what?


So that problems can start to be detected. Time and again, Python
users on Windows get EXTREMELY confused by the way their code worked
perfectly with one path, then bizarrely fails with another. That is a
very real problem, and the problem is that it appeared to work when
actually it was wrong.
[...]
If you can offer a better plan, then by all means, do so. But
deferring without a change is of no real value, and it means ANOTHER
eighteen months added onto the time before novice programmers get to
be told about string literal problems.


Allow me to offer one:

* change the SyntaxWarning into a default-silenced one that fires every 
time a .pyc is loaded (this is the hard part, but it's doable)
* change pathlib.PureWindowsPath, os.fsencode and os.fsdecode to 
explicitly warn when the path contains control characters
* change the PyErr_SetExcFromWindowsErrWithFilenameObjects function to 
append (or chain) an extra message when either of the filenames contains 
control characters (or change OSError to do it, or the default 
sys.excepthook)


I don't care whether the changes are applied to all platforms rather 
than just Windows, but since Windows developers hit the problem and 
(some) Linux developers like to use control characters in filenames, I 
can see a justification for only warning on Windows.


Long term we can still deprecate and eventually block unrecognized 
escape sequences, but the long standing behaviour can stand for a few 
more years without creating more harm.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2ICLCF5T53DBPVZPVHMT2XTXL64QF7WW/

[Python-Dev] Re: What to do about invalid escape sequences

[Joao S. O. Bueno]

For what I can see, the majority of new users in an interactive environment
seeing the warning will do so because the incorrect string will be
in _their_ code. The benefits are immediate, as people change to either
using raw-strings or using forward-slashes for file paths.

The examples in the beggining of this thread, where one changing
a file path to "C:\users" sudden have broken code speaks for themselves:
this is a _fix_ . Broken libraries will be fixed within weeks of a Py 3.8
release. People
will either be using an old install, with Python 3.7, or they keep
everything up to date,
 and for those after 2 months max, the library warnings will be all but
gone.

In the meantime, what is possible is to publicize more how to disable these
warnings on
end-users side, since we all agree that few people know how to that.

On Wed, 7 Aug 2019 at 06:51, Chris Angelico  wrote:

> On Wed, Aug 7, 2019 at 7:33 PM Steven D'Aprano 
> wrote:
> > What's the rush? Let's be objective here: what benefit are we going to
> > get from this change? Is there anyone hanging out desperately for "\d"
> > and "\-" to become SyntaxErrors, so they can... do what?
>
> So that problems can start to be detected. Time and again, Python
> users on Windows get EXTREMELY confused by the way their code worked
> perfectly with one path, then bizarrely fails with another. That is a
> very real problem, and the problem is that it appeared to work when
> actually it was wrong.
>
> Python has a history of fixing these problems. It used to be that
> b"\x61\x62\x63\x64" was equal to u"abcd", but now Python sees these as
> fundamentally different. Data-dependent bugs caused by a syntactic
> oddity are a language flaw that needs to be fixed.
>
> > Because our processes don't work the way we assumed, it turns out that
> > in practice we haven't given developers the deprecation period we
> > thought we had. Read Nathaniel's post, if you haven't already done so:
> >
> >
> https://mail.python.org/archives/list/python-dev@python.org/message/E7QCC74OBYEY3PVLNQG2ZAVRO653LD5K/
> >
> > He makes a compelling case that while we might have had the promised
> > deprecation period by the letter of the law, in practice most developers
> > will have never seen it, and we will be breaking the spirit of the
> > promise if we continue with the unmodified plan.
>
> Yes, that's a fair complaint. But merely pushing the deprecation back
> by a version is not solving it. There has to be SOMETHING done
> differently.
>
> > And yet here we are rushing through a breaking change in an accelerated
> > manner, for a change of marginal benefit.
>
> It's not a marginal benefit. For people who try to teach Python on
> multiple operating systems, this is a very very real benefit. Just
> because YOU don't see the benefit doesn't mean it isn't there.
>
> > > Otherwise, all you're doing is saying "I wish this
> > > problem would just go away".
> >
> > No, I'm saying we don't have to rush this into 3.8. Let's keep the
> > warning silent and push everything back a release.
> >
> > Now is better than never.
> > Although never is often better than *right* now.
>
> Not sure how the Zen supports what you're saying there, since you're
> specifically saying "not never, not now, just later". But what do you
> actually mean by not rushing this into 3.8?
>
> > Right now, we're looking at a seriously compromised user-experience for
> > 3.8. People are going to hate these warnings, many of them won't know
> > what to do with them and will be sure that Python is buggy, and for very
> > little benefit.
>
> Then the problem is that people blame Python for these warnings. That
> is a problem to be solved; we need people to understand that a warning
> emitted by a library is a *library bug* not a language flaw.
>
> > > Library authors can start _right now_ fixing their code so it's more
> > > 3.8 compatible.
> >
> > Provided that (1) they are aware that this is a problem that needs to be
> > fixed, and (2) they have the round tuits to actually fix it by 3.8.0.
> > Neither are guaranteed.
>
> (1) Yes it is, see above; (2) fair point, but this is restricted to
> string literals and can be detected simply by compiling the code, so
> it's a reasonably findable problem.
>
> > > ("More" because 3.8 doesn't actually break anything.)
> > > What is actually gained by waiting longer
> >
> > We gain the avoidance of a painful experience in 3.8 for a significant
> > number of users and third-party devs.
> >
> > The question we haven't had answered is what we gain by pushing through
> > with the original plan. Plenty of people have said "Let's just do it"
> > but as far as I can see not one has explained *why* we should put end-
> > users and library developers through this frustrating and annoying
> > rushed deprecation period.
>
> And unless you have a plan to do something different in 3.8 that
> ensures that library devs see the warnings, there's no justification
> for the delay. All you'll do is defer the 

[Python-Dev] How to extricate large set of diffs from hg.python.org/sandbox?

[Skip Montanaro]

Victor's experiments into a register-based virtual machine live here:

https://hg.python.org/sandbox/registervm

I'd like to revive them, if for no other reason to understand what he
did. I see no obvious way to collect them all as a massive diff. For
the moment, I downloaded each commit and am applying them oldest to
newest (against 3.3, which I think was Victor's base), correcting
issues as I go along. Still, that is going to take a good long while.
If there's an easier way to do this, I'm all ears.

Skip
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BR2V4XHEBMCLNWU2TMUIJBEYU2UNQBRD/

[Python-Dev] Re: What to do about invalid escape sequences

[Chris Angelico]

On Wed, Aug 7, 2019 at 7:33 PM Steven D'Aprano  wrote:
> What's the rush? Let's be objective here: what benefit are we going to
> get from this change? Is there anyone hanging out desperately for "\d"
> and "\-" to become SyntaxErrors, so they can... do what?

So that problems can start to be detected. Time and again, Python
users on Windows get EXTREMELY confused by the way their code worked
perfectly with one path, then bizarrely fails with another. That is a
very real problem, and the problem is that it appeared to work when
actually it was wrong.

Python has a history of fixing these problems. It used to be that
b"\x61\x62\x63\x64" was equal to u"abcd", but now Python sees these as
fundamentally different. Data-dependent bugs caused by a syntactic
oddity are a language flaw that needs to be fixed.

> Because our processes don't work the way we assumed, it turns out that
> in practice we haven't given developers the deprecation period we
> thought we had. Read Nathaniel's post, if you haven't already done so:
>
> https://mail.python.org/archives/list/python-dev@python.org/message/E7QCC74OBYEY3PVLNQG2ZAVRO653LD5K/
>
> He makes a compelling case that while we might have had the promised
> deprecation period by the letter of the law, in practice most developers
> will have never seen it, and we will be breaking the spirit of the
> promise if we continue with the unmodified plan.

Yes, that's a fair complaint. But merely pushing the deprecation back
by a version is not solving it. There has to be SOMETHING done
differently.

> And yet here we are rushing through a breaking change in an accelerated
> manner, for a change of marginal benefit.

It's not a marginal benefit. For people who try to teach Python on
multiple operating systems, this is a very very real benefit. Just
because YOU don't see the benefit doesn't mean it isn't there.

> > Otherwise, all you're doing is saying "I wish this
> > problem would just go away".
>
> No, I'm saying we don't have to rush this into 3.8. Let's keep the
> warning silent and push everything back a release.
>
> Now is better than never.
> Although never is often better than *right* now.

Not sure how the Zen supports what you're saying there, since you're
specifically saying "not never, not now, just later". But what do you
actually mean by not rushing this into 3.8?

> Right now, we're looking at a seriously compromised user-experience for
> 3.8. People are going to hate these warnings, many of them won't know
> what to do with them and will be sure that Python is buggy, and for very
> little benefit.

Then the problem is that people blame Python for these warnings. That
is a problem to be solved; we need people to understand that a warning
emitted by a library is a *library bug* not a language flaw.

> > Library authors can start _right now_ fixing their code so it's more
> > 3.8 compatible.
>
> Provided that (1) they are aware that this is a problem that needs to be
> fixed, and (2) they have the round tuits to actually fix it by 3.8.0.
> Neither are guaranteed.

(1) Yes it is, see above; (2) fair point, but this is restricted to
string literals and can be detected simply by compiling the code, so
it's a reasonably findable problem.

> > ("More" because 3.8 doesn't actually break anything.)
> > What is actually gained by waiting longer
>
> We gain the avoidance of a painful experience in 3.8 for a significant
> number of users and third-party devs.
>
> The question we haven't had answered is what we gain by pushing through
> with the original plan. Plenty of people have said "Let's just do it"
> but as far as I can see not one has explained *why* we should put end-
> users and library developers through this frustrating and annoying
> rushed deprecation period.

And unless you have a plan to do something different in 3.8 that
ensures that library devs see the warnings, there's no justification
for the delay. All you'll do is defer the exact same problem by
another eighteen months. If the warning remains silent in 3.8, how
will library devs get any indication that they need to fix something?

If you can offer a better plan, then by all means, do so. But
deferring without a change is of no real value, and it means ANOTHER
eighteen months added onto the time before novice programmers get to
be told about string literal problems.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RISO4KSTHBMQZJT5XFS34GCB2PB66WNV/

[Python-Dev] Re: What to do about invalid escape sequences

[Paul Moore]

On Wed, 7 Aug 2019 at 10:32, Steven D'Aprano  wrote:

> No, I'm saying we don't have to rush this into 3.8. Let's keep the
> warning silent and push everything back a release.
>
> Now is better than never.
> Although never is often better than *right* now.
>
> Right now, we're looking at a seriously compromised user-experience for
> 3.8. People are going to hate these warnings, many of them won't know
> what to do with them and will be sure that Python is buggy, and for very
> little benefit.
>
> Let's slow down and put it off for another release, giving us time to
> solve the warnings problem, and library authors the deprecation period
> promised.

+1 from me. The arguments made here are pretty compelling to me, and I
agree that we should take a breath and not rush this warning into 3.8,
given what we now know.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OXGY2MPRTK3BJAXCRVLFKKKQNREKO7O4/

[Python-Dev] Re: What to do about invalid escape sequences

[Steven D'Aprano]

On Wed, Aug 07, 2019 at 02:33:51PM +1000, Chris Angelico wrote:
> On Wed, Aug 7, 2019 at 1:54 PM Steven D'Aprano  wrote:
> > Don't think of this as a failure. Think of it as an opportunity: we've
> > identified a weakness in our deprecation process. Let's fix that
> > process, make sure that *developers* will see the warning in 3.8 or 3.9,
> > and not raise an exception until 4.0 or 4.1.
> >
> 
> So HOW are you going to make sure developers see it?

I've only just started thinking about it, give me a couple of minutes! *wink*

What's the rush? Let's be objective here: what benefit are we going to 
get from this change? Is there anyone hanging out desperately for "\d" 
and "\-" to become SyntaxErrors, so they can... do what?

Because our processes don't work the way we assumed, it turns out that 
in practice we haven't given developers the deprecation period we 
thought we had. Read Nathaniel's post, if you haven't already done so:

https://mail.python.org/archives/list/python-dev@python.org/message/E7QCC74OBYEY3PVLNQG2ZAVRO653LD5K/

He makes a compelling case that while we might have had the promised 
deprecation period by the letter of the law, in practice most developers 
will have never seen it, and we will be breaking the spirit of the 
promise if we continue with the unmodified plan.

Quite frankly, if we continue with the unmodified plan, third-party devs 
who are affected will have the right to feel mightly pissed off at us. 
We make an implicit, if not explicit, promise that we won't break 
backswards compatibility lightly, but if we do, we will give them plenty 
of notice except under the most dire circumstances (such as a serious 
security vulnerability).

And yet here we are rushing through a breaking change in an accelerated 
manner, for a change of marginal benefit. Sure, we can say that 
*technically* we gave them all the notice promised, it was at the bottom 
of a locked filing cabinet stuck in a disused lavatory with a sign on 
the door saying "Beware of The Leopard".

https://www.goodreads.com/quotes/40705-but-the-plans-were-on-display-on-display-i-eventually

I'm sure that the affected devs will understand why it was *their* fault 
they couldn't see the warnings, when even people from a first-class 
library like SymPy took four iterations to do it right.


> Currently it
> requires some extra steps or flags, which are not well known. What
> change are you proposing for 3.8 that will ensure that this actually
> gets solved? 

Absolutely nothing. I don't have to: we're an entire community, this 
doesn't have to fall only on my shoulders. I'm not even the messenger: 
that's Raymond. I'm just (partly) agreeing with him.

Just because I don't have a solution for this problem doesn't mean the 
problem doesn't exist.


> Otherwise, all you're doing is saying "I wish this
> problem would just go away".

No, I'm saying we don't have to rush this into 3.8. Let's keep the 
warning silent and push everything back a release.

Now is better than never.
Although never is often better than *right* now.

Right now, we're looking at a seriously compromised user-experience for 
3.8. People are going to hate these warnings, many of them won't know 
what to do with them and will be sure that Python is buggy, and for very 
little benefit.

Let's slow down and put it off for another release, giving us time to 
solve the warnings problem, and library authors the deprecation period 
promised.


> Library authors can start _right now_ fixing their code so it's more
> 3.8 compatible.

Provided that (1) they are aware that this is a problem that needs to be 
fixed, and (2) they have the round tuits to actually fix it by 3.8.0. 
Neither are guaranteed.

Its not a big fix, but people have other priorities, like work, family, 
a life, etc. That's why we normally give developers *multiple years* of 
warnings to fix problems, not weeks. This change is not so important 
that we have to push it through in an accelerated time frame.



> ("More" because 3.8 doesn't actually break anything.)
> What is actually gained by waiting longer

We gain the avoidance of a painful experience in 3.8 for a significant 
number of users and third-party devs.

The question we haven't had answered is what we gain by pushing through 
with the original plan. Plenty of people have said "Let's just do it" 
but as far as I can see not one has explained *why* we should put end- 
users and library developers through this frustrating and annoying 
rushed deprecation period.




-- 
Steven
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/I6DFONZPRHL4VYUYICAXIMUTR4KVVHV6/

[Python-Dev] Re: typing: how to use names in result-tuples?

[Christian Tismer]

Hi all,

Ok, I am about to implement generation of such structures
automatically using the struct sequence concept.


One more question:
--

Struct sequences are not yet members of the typing types.
I would like to add that, because a major use case is also to
show nice .pyi files with all the functions and types.

* namedtuple has made the transition to NamedTuple

* What would I need to do that for StructSequence as well?

Things get also a bit more complicated since struct sequence
objects can contain unnamed fields.

Any advice would be appreciated, I am no typing expert (yet :-)

cheers -- Chris


On 30.07.19 17:10, Guido van Rossum wrote:
> I think I have to agree with Petr. Define explicit type names.
> 
> On Tue, Jul 30, 2019 at 2:45 AM Paul Moore  > wrote:
> 
> On Tue, 30 Jul 2019 at 09:33, Christian Tismer  > wrote:
> > >>> typing.NamedTuple("__f", x=int, y=int)
> > 
> > >>> typing.NamedTuple("__f", x=int, y=int) is typing.NamedTuple("__f",
> > x=int, y=int)
> > False
> 
> This appears to go right back to collections.namedtuple:
> 
> >>> from collections import namedtuple
> >>> n1 = namedtuple('f', ['a', 'b', 'c'])
> >>> n2 = namedtuple('f', ['a', 'b', 'c'])
> >>> n1 is n2
> False
> 
> I found that surprising, as I expected the named tuple type to be
> cached based on the declared name 'f'. But it's been that way forever
> so obviously my intuition here is wrong. But maybe it would be useful
> for this case if there *was* a way to base named tuple identity off
> the name/fields? It could be as simple as caching the results:
> 
> >>> from functools import lru_cache
> >>> cached_namedtuple = lru_cache(None)(namedtuple)
> >>> n1 = cached_namedtuple('f', ('a', 'b', 'c')) # A tuple rather
> than a list of field names, as lists aren't hashable
> >>> n2 = cached_namedtuple('f', ('a', 'b', 'c'))
> >>> n1 is n2
> True
> 
> Paul
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido )
> /Pronouns: he/him/his //(why is my pronoun here?)/
> 
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/GIFRTFWPEGKZ33PTW63YXKGXHHAQJ35I/
> 


-- 
Christian Tismer :^)   tis...@stackless.com
Software Consulting  : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FG6BG5DENW4TTD24XJIHXWS37RW7X5JY/

[Python-Dev] Re: What to do about invalid escape sequences

[Serhiy Storchaka]

07.08.19 03:57, Gregory P. Smith пише:
People distribute code via pypi.  if we reject uploads of packages with 
these problems and link to fixers (modernize can be taught what to do), 
we prevent them from spreading further.


How can we check that there are such problems in the package? Pass all 
*.py files through a linter? But the package can contain "incorrect" 
files, for example files for Python 2 or earlier Python 3 versions. Even 
the CPython testsuite contains bad Python files for testing purpose.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JX7IDIGFLAZIF2YQIR5IYNNHLLHGRA4T/

[Python-Dev] Re: What to do about invalid escape sequences

[Serhiy Storchaka]

07.08.19 03:31, Rob Cliffe via Python-Dev пише:

How about: whenever a third-party library uses a potentially-wrong
escape sequence, it creates a message on the console. Then when
someone sees that message, they can post a bug report against the
package.


Would not it just increase the amount of a noise? The main complain 
about new warnings is a noise.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VZHWNZ4X7PXXE4Y4XIZCLMWSYGNJ5WPY/

[Python-Dev] Re: What to do about invalid escape sequences

[Serhiy Storchaka]

07.08.19 01:37, Brett Cannon пише:

I think this is a good example of how the community is not running tests with 
warnings on and making sure that their code is warnings-free. This warning has 
existed for at least one full release and fixing it doesn't require some crazy 
work-around for backwards compatibility, and so this tells me people are simply 
either ignoring the warnings or they are not aware of them.


There are several PRs for fixing warnings on GitHub every month. And 
seems a deprecation warning about importing ABCs from collections is at 
least so common (if not more) as a warning about "invalid escape 
sequences". The former is more visible to end users because is emitted 
at every run, not only at the first bytecode compilation.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7TYOKDS3D5YXKJFBJO6G6OVKVRYKRCHO/

[Python-Dev] Re: What to do about invalid escape sequences

[Serhiy Storchaka]

06.08.19 20:37, Paul Moore пише:

I don't see issues reported in the bug trackers for docutils and
bottle. Maybe as a start, someone could raise issues there?


The warning in docutils was fixed.
https://sourceforge.net/p/docutils/code/8255/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JPKBMJ6WH5HQGUDND3JZCLGRQ2KKSEPN/