[Python-Dev] Re: Can someone please finish merging GH-13482?

[Tim Peters]

[Mariatta ]
- Since this is a 1st-time contributor, does it need a change to the ACKS
file?

>
> I think the change is trivial enough, the misc/acks is not necessary.
>
> - Anything else?
>
>
> 1. Does it need to be backported? If so, please add the "needs backport to
> .." label.
>
> 2. Add the "烙 automerge" label. The bot will take care of merging. It
> will use the PR title and description as commit message.
> If the PR title/ description is not good enough as commit message, edit it
> first before adding the "烙 automerge" label.
>
> So I'll watch and do it myself next time ;-)
>
>
> Hopefully the above instructions are clear and simple enough, so I'll let
> you trigger the automerge this time.
>

So I tried ;-)  No idea what did and didn't work.  I got several of these
messages:

"""
Sorry, I can't merge this PR. Reason: Base branch was modified. Review and
try the merge again..
"""

I'm guessing (don't know) those are about failed auto-backports.  I don't
have time now to try to figure it all out - It seemed to have spawned a
number of new GH PRs for the backports, and I'm thoroughly lost now :-(

The change did show up when I pulled from upstream master, so I expect that
part did work.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SHIDYDSL43D6ANHGBCGVPMHSGCU5XK2P/

[Python-Dev] Re: What to do about invalid escape sequences

[Chris Angelico]

On Wed, Aug 7, 2019 at 1:54 PM Steven D'Aprano  wrote:
> Don't think of this as a failure. Think of it as an opportunity: we've
> identified a weakness in our deprecation process. Let's fix that
> process, make sure that *developers* will see the warning in 3.8 or 3.9,
> and not raise an exception until 4.0 or 4.1.
>

So HOW are you going to make sure developers see it? Currently it
requires some extra steps or flags, which are not well known. What
change are you proposing for 3.8 that will ensure that this actually
gets solved? Otherwise, all you're doing is saying "I wish this
problem would just go away".

Library authors can start _right now_ fixing their code so it's more
3.8 compatible. ("More" because 3.8 doesn't actually break anything.)
What is actually gained by waiting longer, and how do you propose to
make this transition easier?

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UGDPMFKXJRU2CPKPAI5NLHDNH3VG6BWN/

[Python-Dev] Re: What to do about invalid escape sequences

[Steven D'Aprano]

On Tue, Aug 06, 2019 at 07:58:12PM -0700, Nathaniel Smith wrote:

> For example, all my projects run tests with deprecation warnings
> enabled and warnings turned into errors, but I never saw any of these
> warnings. What happens is: the warning is issued when the .py file is
> byte-compiled; but at this point, deprecation warnings probably aren't
> visible. Later on, when pytest imports the file, it has warnings
> enabled... but now the warning isn't issued.

This!

If Nathaniel's analysis is correct, and I think it is, we've identified 
a flaw in our deprecation process. We've assumed that library devs will 
see the warnings long before end users.

Since the benefit of this breaking change is quite small, let's delay it 
long enough to fix the deprecation process.


-- 
Steven
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/D4BLHVLQL4KR5QGAUMOONVH7MJ4ZMR2L/

[Python-Dev] Re: What to do about invalid escape sequences

[Steven D'Aprano]

On Wed, Aug 07, 2019 at 10:14:08AM +1000, Chris Angelico wrote:
> On Wed, Aug 7, 2019 at 10:03 AM Steven D'Aprano  wrote:
> > - Keep the SyntaxWarning silent by default for 3.8. That gives us
> > another year or more to gently pressure third-party libraries to fix
> > their code, and to find ways to encourage developers to run with
> > warnings enabled.
> 
> How do you propose to apply this pressure?

We already have some good information about the offending libraries.

(Remember, the libraries here haven't done anything wrong. They were
using a documented feature. We've just changed our mind about that
feature.)

Raymond mentioned two, docutils and bottle, and Matt did a scan of the 
top 100 downloads on PyPI. We can start by reporting this as a bug to 
them.

 
> How about: whenever a third-party library uses a potentially-wrong
> escape sequence, it creates a message on the console. Then when
> someone sees that message, they can post a bug report against the
> package.

You're right, of course, and if we were talking about one or two 
warnings a week, affecting a handful of users, I don't think Raymond 
would have said anything. But apparently this is a widespread problem 
with common third party libraries. That means its going to affect lots 
of people.

We have a few problems:

- The people affected will mostly be the end users, not the developers.

- These sorts of SyntaxWarnings are scary and intimdating to beginners,
  even when they are harmless. Many of them will not know how to silence 
  warnings, or who to report it as a bug to.

- Since end users rarely search for existing bug reports before 
  adding a new one, or upgrade to the latest version, we're 
  effectively sentencing the third-party library authors to be
  flooded with potentially dozens of identical bug reports long after
  they have fixed the issue.

- I expect that many end users will report it as a *Python* bug, so
  we're going to share some of that pain too.

- The benefit of the desired change is relatively low.

The intention was for the developers, not end users, to see the warning. 
If end users see more than a tiny number of these warnings, our plan 
failed. That's okay: since the benefit of the breaking change is small, 
we can rethink the plan, delay the breaking change, and try to come up 
with a better system that ensures developers see these warnings before 
their users do.

We're not fixing a major security issue here, or adding a new feature 
that will make people's code enormously better. We're breaking people's 
code to force them to write "better" code, so that *maybe* some day in 
the future we can add new escape sequences.

That's a really small benefit for breaking backwards compatibility. 
We don't break backwards compatibility lightly because of the knock on 
effects of code churn, libraries that stop working, frustrated users, 
obsoleted blog posts and books, questions asked on Stackoverflow etc. 
When the benefit is small, we require the pain to be correspondingly 
small. That's not going to be the case if we continue with the plan.

Don't think of this as a failure. Think of it as an opportunity: we've 
identified a weakness in our deprecation process. Let's fix that 
process, make sure that *developers* will see the warning in 3.8 or 3.9, 
and not raise an exception until 4.0 or 4.1.

I know people just want to get it over and done with, I do too. But we 
have responsibilities to the community, and we've lived with the current 
behaviour for 25+ years, another 2-3 years won't kill us.


-- 
Steven
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NY5XTN2QYKYAZFDKSX5TSEO3SY4WOXYU/

[Python-Dev] Re: Can someone please finish merging GH-13482?

[Mariatta]

>
> - Does something special need to be done for doc changes?


Nothing special.

- Since this is a 1st-time contributor, does it need a change to the ACKS
> file?


I think the change is trivial enough, the misc/acks is not necessary.

- Anything else?


1. Does it need to be backported? If so, please add the "needs backport to
.." label.

2. Add the "烙 automerge" label. The bot will take care of merging. It will
use the PR title and description as commit message.
If the PR title/ description is not good enough as commit message, edit it
first before adding the "烙 automerge" label.

So I'll watch and do it myself next time ;-)


Hopefully the above instructions are clear and simple enough, so I'll let
you trigger the automerge this time.


ᐧ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SP4POPZH4VSBIVEJIG4KZ6AYIC3K26R4/

[Python-Dev] Re: What to do about invalid escape sequences

[Nathaniel Smith]

On Tue, Aug 6, 2019 at 3:44 PM Brett Cannon  wrote:
> I think this is a good example of how the community is not running tests with 
> warnings on and making sure that their code is warnings-free. This warning 
> has existed for at least one full release and fixing it doesn't require some 
> crazy work-around for backwards compatibility, and so this tells me people 
> are simply either ignoring the warnings or they are not aware of them.
>
> If it's the case that people are choosing to ignore warnings then that's on 
> them and there's not much we can do there.
>
> But my suspicion is it's the latter case of people simply not thinking about 
> running with warnings on and making sure to check for them. For instance, are 
> people running their CI with warnings turned on? How about making sure to 
> check the output of their CI to make sure there are no warnings? Or even 
> better, how many people are running CI with warnings turned into exceptions? 
> My guess is all of this is rather low because people are probably just doing 
> `pytest` without thinking of turning on warnings as exceptions to trigger a 
> CI failure and are only looking for CI passing versus checking its output.

There's an important point here that I think has been missed.

These days deprecation warnings are much more visible in general,
because all the major test systems enable them by default. BUT, this
SPECIFIC warning almost completely circumvented all those systems, so
almost no-one saw it.

For example, all my projects run tests with deprecation warnings
enabled and warnings turned into errors, but I never saw any of these
warnings. What happens is: the warning is issued when the .py file is
byte-compiled; but at this point, deprecation warnings probably aren't
visible. Later on, when pytest imports the file, it has warnings
enabled... but now the warning isn't issued.

Quoting Aaron Meurer from the bpo thread:

> As an anecdote, for SymPy's CI, we went through five (if I am counting 
> correctly) iterations of trying to test this. Each of the first four were 
> subtly incorrect, until we finally managed to find the correct one (for 
> reference, 'python -We:invalid -m compileall -f -q module/').  So most 
> library authors who will attempt to add tests against this will get it wrong.

Since folks don't seem to be reading that thread, I'll re-post my
comment from it as well:

> I think we haven't *actually* done a proper DeprecationWarning period for 
> this. We tried, but because of the issue with byte-compiling, the warnings 
> were unconditionally suppressed for most users -- even the users who are 
> diligent enough to enable warnings and look at warnings in their test suites.
> I can see a good argument for making the change, but if we're going to do it 
> then it's obviously the kind of change that requires a proper deprecation 
> period, and that hasn't happened.
> Maybe .pyc files need to be extended to store a list of syntax-related 
> DeprecationWarnings and SyntaxWarnings, that are re-issued every time the 
> .pyc is loaded? Then we'd at least have the technical capability to deprecate 
> this properly.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/E7QCC74OBYEY3PVLNQG2ZAVRO653LD5K/

[Python-Dev] Can someone please finish merging GH-13482?

[Tim Peters]

https://github.com/python/cpython/pull/13482

is a simple doc change for difflib, which I approved some months ago.
But I don't know the current workflow well enough to finish it myself.
Like:

- Does something special need to be done for doc changes?

- Since this is a 1st-time contributor, does it need a change to the ACKS file?

- Anything else?

I'm sure this is all obvious after you've done it once, but I haven't.
So I'll watch and do it myself next time ;-)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TFTQCJ7IZNCE3XUWEC4IVHPXBDZIKVEI/

[Python-Dev] Re: What to do about invalid escape sequences

[Glenn Linderman]

On 8/6/2019 5:57 PM, Gregory P. Smith wrote:
People distribute code via pypi.  if we reject uploads of packages 
with these problems and link to fixers (modernize can be taught what 
to do), we prevent them from spreading further.  A few years after 
doing that, we can revisit how much pain and for whom making this a 
SyntaxWarning or even SyntaxError would actually be.


it isn't useful to tell /users/ of packages to spend time figuring out 
who to complain to that some packages code that they somehow depend on 
(often transitively) is not modern enough.




People also distribute code other ways. A few years after cleaning up 
pypi, you still wouldn't know how much pain and for whom making this a 
SyntaxWarning or SyntaxError would actually be.


Just do it and get it over with.

Glenn
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UZLAFCN64OQ3IQ6GPIVFIELSK43DXYXE/

[Python-Dev] Re: What to do about invalid escape sequences

[Gregory P. Smith]

People distribute code via pypi.  if we reject uploads of packages with
these problems and link to fixers (modernize can be taught what to do), we
prevent them from spreading further.  A few years after doing that, we can
revisit how much pain and for whom making this a SyntaxWarning or even
SyntaxError would actually be.

it isn't useful to tell *users* of packages to spend time figuring out who
to complain to that some packages code that they somehow depend on (often
transitively) is not modern enough.

On Tue, Aug 6, 2019 at 5:39 PM Rob Cliffe via Python-Dev <
python-dev@python.org> wrote:

>
>
> On 07/08/2019 01:14:08, Chris Angelico wrote:
> > On Wed, Aug 7, 2019 at 10:03 AM Steven D'Aprano 
> wrote:
> >> - Keep the SyntaxWarning silent by default for 3.8. That gives us
> >> another year or more to gently pressure third-party libraries to fix
> >> their code, and to find ways to encourage developers to run with
> >> warnings enabled.
> > How do you propose to apply this pressure?
> >
> > How about: whenever a third-party library uses a potentially-wrong
> > escape sequence, it creates a message on the console. Then when
> > someone sees that message, they can post a bug report against the
> > package.
> >
> > In other words, a non-silent warning.
> >
> > ChrisA
> > ___
> >
> The interpreter knows which module contains the questionable string.
> So: is it feasible for the warning message to include something like
> "...  If you are not the maintainer of xxxmodule.py, please contact
> them, or post a bug report on ..."
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/JJULDRHOTETV2LDLJE6RWKZ6HCCPHTUP/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MK5SN5OCCRFHYBNUXL5BDUIN2Q7FV4JB/

[Python-Dev] Re: What to do about invalid escape sequences

[Rob Cliffe via Python-Dev]

On 07/08/2019 01:14:08, Chris Angelico wrote:

On Wed, Aug 7, 2019 at 10:03 AM Steven D'Aprano  wrote:

- Keep the SyntaxWarning silent by default for 3.8. That gives us
another year or more to gently pressure third-party libraries to fix
their code, and to find ways to encourage developers to run with
warnings enabled.

How do you propose to apply this pressure?

How about: whenever a third-party library uses a potentially-wrong
escape sequence, it creates a message on the console. Then when
someone sees that message, they can post a bug report against the
package.

In other words, a non-silent warning.

ChrisA
___


The interpreter knows which module contains the questionable string.
So: is it feasible for the warning message to include something like
"...  If you are not the maintainer of xxxmodule.py, please contact 
them, or post a bug report on ..."

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JJULDRHOTETV2LDLJE6RWKZ6HCCPHTUP/

[Python-Dev] Re: What to do about invalid escape sequences

[Chris Angelico]

On Wed, Aug 7, 2019 at 10:03 AM Steven D'Aprano  wrote:
> - Keep the SyntaxWarning silent by default for 3.8. That gives us
> another year or more to gently pressure third-party libraries to fix
> their code, and to find ways to encourage developers to run with
> warnings enabled.

How do you propose to apply this pressure?

How about: whenever a third-party library uses a potentially-wrong
escape sequence, it creates a message on the console. Then when
someone sees that message, they can post a bug report against the
package.

In other words, a non-silent warning.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LPXFAMMIW2WZLT5EBENMI2PDFPSW6L2O/

[Python-Dev] Re: What to do about invalid escape sequences

[Steven D'Aprano]

This really is a hairy one.

The current behaviour encourages people to use a single backslash when 
they should be using a double, but that works only sometimes. Should you 
include an escape or not? Sometimes the backslash stays and sometimes it 
disappears:

py> "abc \d \' xyz"
"abc \\d ' xyz"

To be honest, I'm kind of surprised that I haven't seen a bug report for 
the backslash disappearing when followed by a quote.

This behaviour also prevents us from adding new kinds of escape 
sequences in the future without a deprecation period. For example, 
there's a long-missing escape sequence often found in C, \e for ESC, but 
we can't add it now even if we wanted it because it would break code 
that relies on "\e" being the same as "\\e".

So I remain +1 on changing the behaviour.

On the other hand, I see Raymond's point. Having tried it with a few 
contrived modules, I agree that it would be intimidating and annoying 
for many users.

As an educator, Raymond could easily teach people how to silence the 
warnings. But people outside the classroom are going to be hit with 
these warnings too, and many of them are not going to know how to 
silence them or even that it is possible to silence them.

I see Raymond's point about not breaking ASCII art, but I think that 
there are work-arounds for that, using raw strings and string 
concatenation. Besides, its 2019 and we use UTF-8 by default. Unicode 
provides alternative ways to draw ASCII art than pure ASCII:

Raymond's example:
'\---> special case'

Unicode:
'└───▷ special case'

So I'm coming around to the position that while we should continue with 
the plan to make invalid escape sequences an error, we should slow down 
a bit. This isn't a critical problem that needs to be fixed soonest. 
Let's push everything back one release:

- Keep the SyntaxWarning silent by default for 3.8. That gives us 
another year or more to gently pressure third-party libraries to fix 
their code, and to find ways to encourage developers to run with 
warnings enabled.

- In 3.9, we can try making the warnings visible again.

- And aim to make it an error in 4.0/3.10.



-- 
Steven
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LNGLOCHOFWKAWW4AI2WYYBIWGJXB3LNY/

[Python-Dev] PEP 588 updates

[Mariatta]

I've updated the text of PEP 588 (https://www.python.org/dev/peps/pep-0588/)
with several items under "Migration Plan" section:

- Hire a professional project manager to help ensure the success of the
project.
- Create a playground issue tracker on GitHub to experiment and test the
new workflow
- The work on new CLA host is currently stalled. (more details:
https://discuss.python.org/t/cla-assistant-is-no-go/2066)
- We've created a new Python Triage team, taking advantage of GitHub's beta
Triage role. We need a better guideline/criteria on how to promote people
to the Python triage team.
- A test repo containing proposed GitHub labels have been created by Carol
Willing. It can be reviewed at:
https://github.com/willingc/test-581/labels Feedback
are welcome. I would suggest discussing the labels in Discourse
 under core-workflow category.
- we should create a separate branch on GitHub with info about the new
workflow. This should be done ahead of the actual migration, not after.

Please re-read PEP 588 in its entirety. Any questions or concerns about PEP
588 can be discussed here in python-dev, or Discourse under core-workflow
category. (https://discuss.python.org/c/core-workflow)

Thanks.

Mariatta
ᐧ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PFKR477KFKLTHJKLTQRS7D64SMFJV6EA/

[Python-Dev] Re: What to do about invalid escape sequences

[Steven D'Aprano]

On Tue, Aug 06, 2019 at 08:14:35AM +0200, Michael wrote:

> For "filenames" you could, perhaps, make an exception in the calls that
> use them. e.g., when they are hard-coded in something such as
> open("..\training\new_memo.doc").

Functions such as open cannot tell whether their argument was provided 
as a string literal or a variable or other expression. So if you are 
thinking we change open() to be something like this:

if filename contains control characters, and filename is a literal:
fix filename

that's not going to work.

Also, it is important that open() be able to work with filenames which 
contain control characters, because some files actually do have control 
characters in them. They are super-hard to deal with in typical POSIX 
shells, but easy to work with in Python. Unless Python tries to be 
"helpful" and auto-corrects (auto-corrupts) filenames for us.

I don't want the interpreter trying to guess what I meant and running 
that instead of what I actually wrote.


-- 
Steven
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NKBDAPMCSEUG5TBXJ3GOBU5SVNXMRWNR/

[Python-Dev] Re: What to do about invalid escape sequences

[Steven D'Aprano]

On Wed, Aug 07, 2019 at 10:41:25AM +1200, Greg Ewing wrote:
> Rob Cliffe via Python-Dev wrote:
> >
> >Sorry, that won't work.  Strings are parsed at compile time, open() is 
> >executed at run-time.
> 
> It could check for control characters, which are probably the result
> of a backslash accident. Maybe even auto-correct them...

http://www.catb.org/jargon/html/D/DWIM.html



-- 
Steven
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OPMEBXN3NJQJRTPOWMPO7YPKNEWQKZKJ/

[Python-Dev] Re: What to do about invalid escape sequences

[Gregory P. Smith]

On Tue, Aug 6, 2019 at 10:06 AM Neil Schemenauer 
wrote:

>
> Making it an error so soon would be mistake, IMHO.  That will break
> currently working code for small benefit.  When Python was a young
> language with a few thousand users, it was easier to make these
> kinds of changes.  Now, we should be much more conservative and give
> people a long time and a lot of warning.  Ideally, we should provide
> tools to fix code if possible.
>
> Could PyPI and pip gain the ability to warn and even fix these
> issues?  Having a warning from pip at install time could be better
> than a warning at import time.  If linting was built into PyPI, we
> could even do a census to see how many packages would be affected by
> turning it into an error.
>

PyPI could warn on or reject packages at upload time when they contain
these kinds code compileall -> pyc time of issues.  that'd force the issue
to be in front of the package owners rather than anyone else.

Similarly, automation could proactively scan existing pypi packages
claiming py3 compatibility for the issue and reach out to owners.

But the issue becomes one of versions: we shouldn't be complaining about
older versions of packages to the package owners.  yet people are free to
list old versions or < constraints in their own packages requirements
regardless of what python version they're running on.

-gps


>
> On 2019-08-05, raymond.hettin...@gmail.com wrote:
> > P.S. In the world of C compilers, I suspect that if the relatively
> > new compiler warnings were treated as errors, the breakage would
> > be widespread. Presumably that's why they haven't gone down this
> > road.
>
> The comparision with C compilers is relevant.  C and C++ represent a
> fairly extreme position on not breaking working code.   E.g. K & R
> style functional declarations were supported for decades.  I don't
> think we need to go quite that far but also one or two releases is
> not enough time.
>
> Regards,
>
>   Neil
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/V2EDFDJGXRIDMKJU3FKIWC2NDLMUZA2Y/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IH62W73S3WFSW6MUHJJ7XBTGFO4PXAWE/

[Python-Dev] Re: What to do about invalid escape sequences

[Greg Ewing]

Rob Cliffe via Python-Dev wrote:


Sorry, that won't work.  Strings are parsed at compile time, open() is 
executed at run-time.


It could check for control characters, which are probably the result
of a backslash accident. Maybe even auto-correct them...

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IQ62S3MI2T2RMQXT7UNQT3SKA7URHWSZ/

[Python-Dev] Re: What to do about invalid escape sequences

[Brett Cannon]

Paul Moore wrote:
> On Tue, 6 Aug 2019 at 17:39, Matt Billenstein m...@vazor.com wrote:
> >
> > On Mon, Aug 05, 2019 at 04:22:50AM -,
> > raymond.hettin...@gmail.com wrote:
> > This once seemed like a reasonable and innocuous
> > idea to me; however, I've
> > been using the 3.8 beta heavily for a month and no longer think it is a good
> > idea.  The warning crops up frequently, often due to third-party packages
> > (such as docutils and bottle) that users can't easily do anything about.
> > Perhaps those packages could be flagged now via pylint and problems raised 
> > with
> > the respective package maintainers before the actual 3.8 release?  Checking 
> > the
> > top 100 or top 1000 packages on PyPI?
> > I don't see issues reported in the bug trackers for docutils and
> bottle. Maybe as a start, someone could raise issues there? And any
> other projects Raymond encountered issues with? If nothing else, it's
> polite to give these projects a warning now that they should be
> stricter about how they use escape sequences, because the core devs
> intend to deprecate and ultimately remove the current permissive
> behaviour. That's what the 3.8 betas are for, after all.
> If the feedback from bug reports like this is that projects consider
> it an unacceptable burden to change, then maybe we would then rethink
> the timescales of the deprecation, or even whether we should do it at
> all. If they just release a quick fix, maybe we're worrying over the
> wrong thing here?

I think this is a good example of how the community is not running tests with 
warnings on and making sure that their code is warnings-free. This warning has 
existed for at least one full release and fixing it doesn't require some crazy 
work-around for backwards compatibility, and so this tells me people are simply 
either ignoring the warnings or they are not aware of them.

If it's the case that people are choosing to ignore warnings then that's on 
them and there's not much we can do there.

But my suspicion is it's the latter case of people simply not thinking about 
running with warnings on and making sure to check for them. For instance, are 
people running their CI with warnings turned on? How about making sure to check 
the output of their CI to make sure there are no warnings? Or even better, how 
many people are running CI with warnings turned into exceptions? My guess is 
all of this is rather low because people are probably just doing `pytest` 
without thinking of turning on warnings as exceptions to trigger a CI failure 
and are only looking for CI passing versus checking its output.

Quick, how many pytest users here know what CLI arg to pass to pytest to have 
it turn warnings into exceptions? The answer is `-W` with the same type of 
argument as `python` takes and I had to look that up myself. 

So this seems to show a communication issue. In this specific instance I'm torn 
because this isn't a new thing and maybe we need to shock people into starting 
to care about warnings? Or maybe we should make a concerted effort in this beta 
cycle to get people to really test their code by telling them they probably 
have a warning hiding in their code that they don't know about and if we don't 
see better uptake we hold off on a release with the SyntaxWarning and make a 
**really** big push for people to run their tests under Python 3.8 with 
warnings flipped on in some fashion and we don't back down in 3.9?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QWTKSPXTPQN6OTOBEWHCWW6EYAACXJR2/

[Python-Dev] Re: What to do about invalid escape sequences

[Matt Billenstein]

On Tue, Aug 06, 2019 at 04:32:04PM +, Matt Billenstein wrote:
> Perhaps those packages could be flagged now via pylint and problems raised 
> with
> the respective package maintainers before the actual 3.8 release?  Checking 
> the
> top 100 or top 1000 packages on PyPI?

fwiw, ran pylint on the top 100 pypi pkgs from:

https://hugovk.github.io/top-pypi-packages/top-pypi-packages-30-days.json

The list of packages is pretty small:

https://gist.github.com/mattbillenstein/ad862d032b8575f8d6e08384850f2223

but some have quite a few errors...

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TKDVLMRD4NRNQD3X3GYKSG5D2I3AKPBO/

[Python-Dev] Re: What to do about invalid escape sequences

[Paul Moore]

On Tue, 6 Aug 2019 at 17:39, Matt Billenstein  wrote:
>
> On Mon, Aug 05, 2019 at 04:22:50AM -, raymond.hettin...@gmail.com wrote:
> > This once seemed like a reasonable and innocuous idea to me; however, I've
> > been using the 3.8 beta heavily for a month and no longer think it is a good
> > idea.  The warning crops up frequently, often due to third-party packages
> > (such as docutils and bottle) that users can't easily do anything about.
>
> Perhaps those packages could be flagged now via pylint and problems raised 
> with
> the respective package maintainers before the actual 3.8 release?  Checking 
> the
> top 100 or top 1000 packages on PyPI?

I don't see issues reported in the bug trackers for docutils and
bottle. Maybe as a start, someone could raise issues there? And any
other projects Raymond encountered issues with? If nothing else, it's
polite to give these projects a warning now that they should be
stricter about how they use escape sequences, because the core devs
intend to deprecate and ultimately remove the current permissive
behaviour. That's what the 3.8 betas are for, after all.

If the feedback from bug reports like this is that projects consider
it an unacceptable burden to change, then maybe we would then rethink
the timescales of the deprecation, or even whether we should do it at
all. If they just release a quick fix, maybe we're worrying over the
wrong thing here?

I remain ambivalent about the change myself. The point that it's the
false positives we *want* to address, but this change hits the false
negatives, is a telling one for me. I don't think I'd support the
change now if we were discussing it for the first time. But on the
other hand, the discussion has already happened, and the decision was
made, and while I'm OK with responding to the evidence that having
loud user-visible warnings is a bad UX, I don't think there's any new
evidence here that significantly changes the facts on which the
decision to eventually make invalid escapes an error was made - just
evidence that the way we chose to introduce that change may be a
problem. (And honestly, to my knowledge, we've never yet found a case
where there *hasn't* been controversy about warnings from libraries
being triggered by user code, so it's not like this is a new problem).

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/47CHDF7UOJUPCSCBZVYZP6JVLPMCYZED/

[Python-Dev] Re: What to do about invalid escape sequences

[Steve Dower]

On 06Aug2019 0959, Neil Schemenauer wrote:

Could PyPI and pip gain the ability to warn and even fix these
issues?  Having a warning from pip at install time could be better
than a warning at import time.  If linting was built into PyPI, we
could even do a census to see how many packages would be affected by
turning it into an error.


If the "generate .pyc" step is used, then it should trigger warnings at 
install time. Hopefully pip does not suppress these (nor treat them as 
fatal), though that would be one satisfactory way of hiding third-party 
warnings from end users. As Serhiy said, once the .pyc exists, you won't 
see this warning again.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ESLQDFXVNQRE2RV72V6GHTMBXRTI3M6B/

[Python-Dev] Re: What to do about invalid escape sequences

[MRAB]

On 2019-08-06 17:32, Matt Billenstein wrote:

On Mon, Aug 05, 2019 at 04:22:50AM -, raymond.hettin...@gmail.com wrote:

This once seemed like a reasonable and innocuous idea to me; however, I've
been using the 3.8 beta heavily for a month and no longer think it is a good
idea.  The warning crops up frequently, often due to third-party packages
(such as docutils and bottle) that users can't easily do anything about.


Perhaps those packages could be flagged now via pylint and problems raised with
the respective package maintainers before the actual 3.8 release?  Checking the
top 100 or top 1000 packages on PyPI?

Or it could be deferred until Python 4.0. Are there any other issues 
where we could say that from Python 4.0 you shouldn't do X?

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LKNAPPAYOMXVQJI2TPDCKV277ZWGSOWR/

[Python-Dev] Re: What to do about invalid escape sequences

[Neil Schemenauer]

Making it an error so soon would be mistake, IMHO.  That will break
currently working code for small benefit.  When Python was a young
language with a few thousand users, it was easier to make these
kinds of changes.  Now, we should be much more conservative and give
people a long time and a lot of warning.  Ideally, we should provide
tools to fix code if possible.

Could PyPI and pip gain the ability to warn and even fix these
issues?  Having a warning from pip at install time could be better
than a warning at import time.  If linting was built into PyPI, we
could even do a census to see how many packages would be affected by
turning it into an error.

On 2019-08-05, raymond.hettin...@gmail.com wrote:
> P.S. In the world of C compilers, I suspect that if the relatively
> new compiler warnings were treated as errors, the breakage would
> be widespread. Presumably that's why they haven't gone down this
> road.

The comparision with C compilers is relevant.  C and C++ represent a
fairly extreme position on not breaking working code.   E.g. K & R
style functional declarations were supported for decades.  I don't
think we need to go quite that far but also one or two releases is
not enough time.

Regards,

  Neil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/V2EDFDJGXRIDMKJU3FKIWC2NDLMUZA2Y/

[Python-Dev] Re: What to do about invalid escape sequences

[Matt Billenstein]

On Mon, Aug 05, 2019 at 04:22:50AM -, raymond.hettin...@gmail.com wrote:
> This once seemed like a reasonable and innocuous idea to me; however, I've
> been using the 3.8 beta heavily for a month and no longer think it is a good
> idea.  The warning crops up frequently, often due to third-party packages
> (such as docutils and bottle) that users can't easily do anything about.

Perhaps those packages could be flagged now via pylint and problems raised with
the respective package maintainers before the actual 3.8 release?  Checking the
top 100 or top 1000 packages on PyPI?

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZS5B2EOCBTJGSHMVTL4ZXNEEGGBL2RN6/

[Python-Dev] Re: What to do about invalid escape sequences

[Rob Cliffe via Python-Dev]

On 06/08/2019 07:14:35, Michael wrote:


For "filenames" you could, perhaps, make an exception in the calls that
use them. e.g., when they are hard-coded in something such as
open("..\training\new_memo.doc").

Sorry, that won't work.  Strings are parsed at compile time, open() is 
executed at run-time.

And the use of open might be masked by using a synonym for it;
open might be shadowed by the program assigning to it;
the argument to open() might be an expression such as 
("..training\new_memo" + extn)    etc., etc.

Rob Cliffe
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5G2JKQKSLAA5CQRUHCN7JFKY2KPIGYYX/

[Python-Dev] Re: What to do about invalid escape sequences

[eryk sun]

On 8/5/19, Steve Dower  wrote:
>
> though I do also see many people bitten by FileNotFoundError
> because of a '\n' in their filename.

Thankfully the common filesystems used in Windows reserve ASCII
control characters in filenames (except not in stream names or
named-pipe names). So a mistaken string literal usually fails with a
more obvious ERROR_INVALID_NAME or C EINVAL instead of a mysterious
ERROR_FILE_NOT_FOUND or C ENOENT.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HTOH6MOHYIDD2UX7YSM2ZVY4BP32ATYL/

[Python-Dev] Re: Is "%zd" format is portable now?

[Michael]

On 05/08/2019 11:16, Inada Naoki wrote:
> https://github.com/python/cpython/blob/1213123005d9f94bb5027c0a5256ea4d3e97b61d/Include/pyport.h#L158-L168
>
> This can be changed to this:
>
> #ifndef PY_FORMAT_SIZE_T
> /* "z" is defined C99 and portable enough.  We can use "%zd" instead of
>"%" PY_FORMAT_SIZE_T "d" for now.
> */
> # define PY_FORMAT_SIZE_T "z"
> #endif
>
Just in case you are curious - I made the change manually in pyport.h
and AIX passes all tests.

Hope this helps!




signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WU74QPST7A42UE25F7OPR6B5LRBTVKDL/

[Python-Dev] Re: What to do about invalid escape sequences

[Random832]

On Mon, Aug 5, 2019, at 00:29, raymond.hettin...@gmail.com wrote:
> The warning crops up frequently, often due to 
> third-party packages (such as docutils and bottle) that users can't 
> easily do anything about.  And during live demos and student workshops, 
> it is especially distracting.

Maybe what we really need is a general solution for warning visibility in 
third-party packages. Isn't that the whole reason DeprecationWarning isn't 
visible by default, for that matter?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GUULUXKYZN3UGWPEUPFKKEKNIZOOYJTJ/

[Python-Dev] Re: What to do about invalid escape sequences

[Ivan Pozdeev via Python-Dev]

On 05.08.2019 7:22, raymond.hettin...@gmail.com wrote:

We should revisit what we want to do (if anything) about invalid escape 
sequences.

For Python 3.8, the DeprecationWarning was converted to a SyntaxWarning which 
is visible by default.  The intention is to make it a SyntaxError in Python 3.9.


I don't see a problem with the DeprecationWarning approach. It's our standard procedure for removing features. Just make it an error in a 
later release -- whoever didn't heed the warning can't say we didn't warn them well in advance that this will stop working in the future.


It doesn't matter if this was a "perfectly working code": we change Python over time and what worked in the past may stop in the future. 
That's not something unexpected and we don't guarantee compatibility between minor releases.


It looks like either the core team has no unifying vision on what the deprecation process is and/or what each warning is for, or that this 
was a user experience experiment and Raymond's complaint is a part of its feedback (so the Mysterious Omniscient Council is probably going 
to incorporate it into other feedback and decide what they want to do next).




This once seemed like a reasonable and innocuous idea to me; however, I've been 
using the 3.8 beta heavily for a month and no longer think it is a good idea.  
The warning crops up frequently, often due to third-party packages (such as 
docutils and bottle) that users can't easily do anything about.  And during 
live demos and student workshops, it is especially distracting.

I now think our cure is worse than the disease.  If code currently has a non-raw 
string with '\latex', do we really need Python to yelp about it (for 3.8) or 
reject it entirely (for 3.9)?   If someone can't remember exactly which special 
characters need to be escaped, do we really need to stop them in their tracks 
during a data analysis session?  Do we really need to reject ASCII art in 
docstrings: ` \---> special case'?

IIRC, the original problem to be solved was false positives rather than false 
negatives:  filename = '..\training\new_memo.doc'.  The warnings and errors 
don't do (and likely can't do) anything about this.

If Python 3.8 goes out as-is, we may be punching our users in the nose and 
getting almost no gain from it.  ISTM this is a job best left for linters.  For 
a very long time, Python has been accepting the likes of 'more \latex markup' 
and has been silently converting it to 'more \\latex markup'.  I now think it 
should remain that way.  This issue in the 3.8 beta releases has been an almost 
daily annoyance for me and my customers. Depending on how you use Python, this 
may not affect you or it may arise multiple times per day.


Raymond

P.S.  Before responding, it would be a useful exercise to think for a moment 
about whether you remember exactly which characters must be escaped or whether 
you habitually put in an extra backslash when you aren't sure.  Then see:  
https://bugs.python.org/issue32912


I use raw strings whenever I have literals with backslashes. Very rarely, I 
need to insert \n-as-newline -- which I do into a regular string.
Since the first one appears in regexes and parameters that are Windows paths 
and the second one in output messages, they do not intersect.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZX2JLOZDOXWVBQLKE4UCVTU5JABPQSLB/


--
Regards,
Ivan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LQWWXWY7OJP7PKHP7NMO27PKXUIF4WSL/

[Python-Dev] Re: What to do about invalid escape sequences

[Michael]

On 05/08/2019 06:22, raymond.hettin...@gmail.com wrote:
I have read through (most) of the thread, and visited the issue referenced.
> We should revisit what we want to do (if anything) about invalid escape 
> sequences.
IMHO - revisit is okay, generally - but that was actually done a long
time ago. Now it is, to me, just another example of "Python" being
indecisive.
>
> For Python 3.8, the DeprecationWarning was converted to a SyntaxWarning which 
> is visible by default.  The intention is to make it a SyntaxError in Python 
> 3.9.
Sounds like this has been discussed in depth - and decided.
>
> This once seemed like a reasonable and innocuous idea to me; however, I've 
> been using the 3.8 beta heavily for a month and no longer think it is a good 
> idea.  The warning crops up frequently, often due to third-party packages 
> (such as docutils and bottle) that users can't easily do anything about.  And 
> during live demos and student workshops, it is especially distracting. 

Because it is not innocuous? My experience with developers (you mention
3rd party) - is that they are lazy. If something is not up there, "in
the face", they will always have a reason to say - tomorrow. Or,
perhaps, since this has been a silent issue (and they are too lazy to
read "What's new" they do not even know. The "head buried in the sand"
sort of thing.

As to demo's and workshops - YOU know this - so use it as an example to
explain how Python development works and DEPENDS on 3rd party developers
paying attention. Yes,I am sure you are concerned about speeding
adoption of Python3.latest-is-greatest, but that is not the world.

For example, RHEL8 is (coming) out. iirc, they way it comes out it what
they intend to support for 10 years - so changes are it will be Python
3.7 (at best) for several years. I have a system with Centos(-7) and
it's default python is python2

[root@t430 ~]# python3
bash: python3: command not found...
Similar command is: 'python'
[root@t430 ~]# python
Python 2.7.5 (default, Jun 20 2019, 20:27:34)
...

> I now think our cure is worse than the disease.  If code currently has a 
> non-raw string with '\latex', do we really need Python to yelp about it (for 
> 3.8) or reject it entirely (for 3.9)?   If someone can't remember exactly 
> which special characters need to be escaped, do we really need to stop them 
> in their tracks during a data analysis session?  Do we really need to reject 
> ASCII art in docstrings: ` \---> special case'?  
Simply put - yes, reject. You decided. There is a solution - perhaps
boring to implement - but as is mentioned - there are 'linters', so an
automated approach is likely possible. If not today, someone will write
a module.
> IIRC, the original problem to be solved was false positives rather than false 
> negatives:  filename = '..\training\new_memo.doc'.  The warnings and errors 
> don't do (and likely can't do) anything about this.
For "filenames" you could, perhaps, make an exception in the calls that
use them. e.g., when they are hard-coded in something such as
open("..\training\new_memo.doc"). iirc, Windows can (and does) use
forward-slash for file names for system calls like open. The "shell"
command.exe does not, because it uses "/" the way posix shells use "-"
(as in /h and -h for the "option" h).
>
> If Python 3.8 goes out as-is, we may be punching our users in the nose and 
> getting almost no gain from it.  ISTM this is a job best left for linters.  
> For a very long time, Python has been accepting the likes of 'more \latex 
> markup' and has been silently converting it to 'more \\latex markup'.  I now 
> think it should remain that way.  This issue in the 3.8 beta releases has 
> been an almost daily annoyance for me and my customers. Depending on how you 
> use Python, this may not affect you or it may arise multiple times per day.

IMHO - Python will not be punching anyone. Python will be delivering "a
promise", being decisive, being clear. Not following through only
creates insecurity - will they ever do it? Nah - no guts (these are
3rd-party developers chatting). Users are your friend. If they really
want Python3.8+ and they get lots of warning messages - THEY will
complain - and be heard - in ways CPython never will (or was).

Again - revisit is fine - and I hope my 2 cents helps you stay the course!

Michael





signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/54MBQXAKGCFNM2KSNSVS6HDAQQSERCMY/